JP4135197B2

JP4135197B2 - Image processing apparatus, image processing method, and camera

Info

Publication number: JP4135197B2
Application number: JP33838897A
Authority: JP
Inventors: 篤小林
Original assignee: Sony Corp
Current assignee: Sony Corp
Priority date: 1997-12-09
Filing date: 1997-12-09
Publication date: 2008-08-20
Anticipated expiration: 2017-12-09
Also published as: JPH11177995A

Description

【０００１】
【発明の属する技術分野】
本発明は、画像処理装置および画像処理方法、並びにカメラに関し、特にカラーフィルタを受光面上に有する固体撮像素子の出力信号を処理する画像処理装置およびその処理方法、並びにこれらを用いたカメラに関する。
【０００２】
【従来の技術】
従来、固体撮像素子に画像情報が入る前に、折り返しノイズを除くことを目的として空間サンプリング周波数ｆｓの１／２の周波数を除去するために、光学的なＬＰＦ(Low Pass Filter) を用いて帯域制限を行っていた。このように、光学ＬＰＦを用いて帯域制限を行うことで、固体撮像素子の出力信号には、空間サンプリング周波数ｆｓの１／２付近の周波数成分が含まれなくなり、その結果、撮像した画像の解像感が損なわれることになる。これに対し、撮像した画像の解像感を損なわないようにするために、上記のような特性の光学ＬＰＦを用いるのではなく、空間サンプリング周波数ｆｓの１／２の周波数よりも高い点のレスポンスを０とすれば良い。
【０００３】
【発明が解決しようとする課題】
しかしながら、ｆｓ／２の周波数よりも高い点のレスポンスを０とすると、偽信号が発生するだけではなく、色配列が例えばＲ（赤），Ｇ（緑），Ｂ（青）の原色ベイヤ配列のカラーフィルタを有す固体撮像素子において、相関検出による信号処理を行う場合には、Ｇによる相関を検出する際に、ｆｓ／２の空間周波数があると縦の相関が強いか横の相関が強いか判定できなくなり、画質劣化の要因となる。
【０００４】
すなわち、図１６において、原色ＲＧＢベイヤ配列（ａ）に対応した固体撮像素子の画素配列に対して、水平ｆｓ／２の画像（ｂ）の信号、または垂直ｆｓ／２の画像（ｃ）の信号が入ってきた場合には、Ｇの画素にのみ着目した場合の出力信号（ｄ）は、水平ｆｓ／２（ｂ）でも垂直ｆｓ／２（ｃ）でも同じになる。このため、Ｇの出力信号を見ただけでは縦の相関が強い（水平ｆｓ／２）のか、横の相関が強い（垂直ｆｓ／２）のかが判別できないことになる。
【０００５】
なお、図１６（ａ）において、Ｇｒ／ＧｂはそれぞれＲ行のＧ画素／Ｂ行のＧ画素を示している。また、同図（ｄ）には、Ｒ／Ｇの画素を黒で表示してあり、灰色（散点）の画素と白色の画素に着目されたい。
【０００６】
本発明は、上記課題に鑑みてなされたものであり、その目的とするところは、縦の相関が強いのか、横の相関が強いのかを確実に判別し、破綻のない画像の再現を可能とした画像処理装置および画像処理方法を提供することにある。
【０００７】
【課題を解決するための手段】
本発明による画像処理装置は、ベイヤ配列のカラーフィルタを受光面上に有する固体撮像素子から出力される画像信号に対して、補間すべき画素とその周辺画素との相関を検出して色補正処理を行う際に、前記補間すべき画素に関して補間する方向にローパスフィルタを通すことによって得られる補間データに補間係数を乗算し、その乗算結果を加算することによって補間処理を行う画像処理装置であって、入力画像の空間サンプリング周波数に対して１／２倍した周波数成分の存在を検出する周波数検出回路と、Ｒ／Ｇ／Ｂの各信号を基に被写体の所定の領域が無彩色であるかを検出する色情報検出回路と、Ｒの画素情報に対して前記補間処理を行うＲ補間部と、Ｇの画素情報に対して前記補間処理を行うＧ補間部と、Ｂの画素情報に対して前記補間処理を行うＢ補間部と、前記周波数検出回路が前記周波数成分の存在を検出しないときは、前記Ｒ補間部、前記Ｇ補間部および前記Ｂ補間部の各補間結果を出力し、前記周波数検出回路が前記周波数成分の存在を検出しかつ前記情報検出回路が無彩色であることを検出しないときは、Ｒ／Ｇ／Ｂの各信号から前記周波数成分を除去して出力し、前記周波数検出回路が前記周波数成分の存在を検出しかつ前記情報検出回路が無彩色であることを検出したときは、Ｒ／Ｇ／Ｂの各信号の水平／垂直方向の色差レベルに基づく補間係数を用いて水平／垂直方向について前記補間処理を行う補正処理部とを備えている。
【０００８】
また、本発明による画像処理方法は、ベイヤ配列のカラーフィルタを受光面上に有する固体撮像素子から出力される画像信号に対して、補間すべき画素とその周辺画素との相関を検出して色補正処理を行う際に、前記補間すべき画素に関して補間する方向にローパスフィルタを通すことによって得られる補間データに補間係数を乗算し、その乗算結果を加算することによって補間処理を行う画像処理方法であって、Ｒ／Ｇ／Ｂの各画素情報に対して前記補間処理を行う一方、入力画像の空間サンプリング周波数に対して１／２倍した周波数成分の存在を検出するとともに、Ｒ／Ｇ／Ｂの各信号を基に被写体の所定の領域が無彩色であるかを検出し、前記周波数成分の存在を検出しないときは、前記Ｒ／Ｇ／Ｂの各画素情報に対する前記補間処理の各補間結果を出力し、前記周波数成分の存在を検出しかつ無彩色であることを検出しないときは、Ｒ／Ｇ／Ｂの各信号から前記周波数成分を除去して出力し、前記周波数成分の存在を検出しかつ無彩色であることを検出したときは、Ｒ／Ｇ／Ｂの各信号の水平／垂直方向の色差レベルに基づく補間係数を用いて水平／垂直方向について補間処理を行うようにしている。
【０００９】
上記構成の画像処理装置およびその処理方法において、例えば、相関検出による信号処理を行う場合には、入力画像の空間サンプリング周波数に対する１／２の周波数成分があると、縦の相関が強いか横の相関が強いか判定できないことから、空間サンプリング周波数の１／２の周波数成分の検出を行うとともに、色に関する情報の検出、具体的には無彩色（＝白）の検出を行う。ただし、この無彩色の検出は、空間サンプリング周波数の１／２の周波数成分を検出した場合にのみ意味を持つ。これは、無彩色であるならば、空間サンプリング周波数の１／２の領域でも、水平相関が強いか、垂直相関が強いかを判定できるためである。
【００１０】
そして、空間サンプリング周波数の１／２の周波数成分を検出した場合において、無彩色と判定したならば、無彩色検出で算出した数値を持って水平補間または垂直補間を行う。これにより、無彩色時には、空間サンプリング周波数の１／２に対応する解像度まで特性を向上できる。一方、無彩色でない、つまり有彩色であると判定したならば、空間サンプリング周波数の１／２の周波数成分を含まないＲ／Ｇ／Ｂの各信号を出力する。これにより、空間サンプリング周波数の１／２付近の相関検出の誤検出を防止できる。
【００１１】
【発明の実施の形態】
以下、本発明の実施の形態について図面を参照して詳細に説明する。図１は、本発明による画像処理装置の基本構成を示すブロック図である。ここで、本画像処理装置の処理対象となるカラー固体撮像素子は、色配列として例えば図２に示すＲ（赤）Ｇ（緑）Ｂ（青）の原色ベイヤ配列のカラーフィルタ１１を受光面上に有する単板式固体撮像素子１２である。
【００１２】
なお、色配列は原色ベイヤ配列に限定されるものではなく、さらにカラーフィルタもＲＧＢの原色の色配列に限られるものではなく、他の原色の色配列の場合でも、補色を使用した色配列（例えば、Ｙｅ／Ｃｙ／Ｍｇ／Ｇ）の場合でも同様に適応可能である。また、固体撮像素子１２としては、全画素の信号電荷を独立に読み出すいわゆる全画素読み出し方式のＣＣＤ(Charge Coupled Device) 固体撮像素子（以下、ＣＣＤエリアセンサと称する）を用いるものとするが、全画素読み出し方式ではないＣＣＤ固体撮像素子にも適応可能である。
【００１３】
ＣＣＤエリアセンサ１２から出力されるＲＧＢ点順次データは、信号処理部１３において黒レベルクランプやホワイトバランスなどの信号処理が行われた後、検出部１４および補間部１５に供給される。検出部１４は、入力されるＲＧＢ点順次データから最適な補間方法を検出し、その補間情報を補間部１５へ送る。補間部１５は、検出部１４から入力される補間情報を基にＲＧＢ点順次データに対して補間処理を行って出力する。
【００１４】
検出部１４は、図３に示すように、空間サンプリング周波数ｆｓの１／２の周波数成分を検出するｆｓ／２検出部１６と、補間すべき画素（以下、補間画素と称す）に関して上下および左右の互いに９０°の整数倍の角度をなす４方向、即ち垂直（Ｖ）方向の相反する２方向および水平（Ｈ）方向の相反する２方向の計４方向の相関の程度を検出するＶＨ相関検出部１７と、補間画素に関して右上、左上、左下、右下の斜め方向、即ち上記４方向に対してそれぞれ４５°の角度をなす４方向の相関の程度を検出する斜め相関検出部１８とを有する構成となっている。
【００１５】
なお、本例では、上下左右４方向に加え、斜め４方向の計８方向の相関の程度を検出する構成を例に採っているが、上下左右の４方向だけの相関の程度を検出する構成であっても良い。ただし、以下の説明では、８方向の場合を例に採って説明するものとする。
【００１６】
一方、補間部１５は、図４に示すように、検出部１４から与えられる補間情報に基づいて、Ｒの画素情報に対して補間処理を行うＲ補間部１９と、Ｇの画素情報に対して補間処理を行うＧ補間部２０と、Ｂの画素情報に対して補間処理を行うＢ補間部２１と、検出部１４の検出情報に基づいてｆｓ／２でのハッチ状のノイズが発生するのを抑止するための補正処理を行うｆｓ／２補正処理部２２とを有する構成となっている。
【００１７】
以下に、検出部１４におけるｆｓ／２検出部１６、ＶＨ相関検出部１７および斜め相関検出部１８の各構成例について説明する。
【００１８】
図５は、検出部１４におけるｆｓ／２検出部１６の具体的な構成の一例を示すブロック図である。なお、図６に、ｆｓ／２検出部１６における通過フィルタの空間周波数‐レスポンスの特性を示す。
【００１９】
このｆｓ／２検出部１６は、通過する周波数のレスポンスの最大値が入力画像の空間サンプリング周波数ｆｓの１／２の空間周波数であるフィルタ特性、即ち図６の特性Ａを持つｆｓ／２通過フィルタ２３と、通過する周波数のレスポンスの最大値が入力画像の空間サンプリング周波数ｆｓの１／４の空間周波数であるフィルタ特性、即ち図６の特性Ｂを持つｆｓ／４通過フィルタ２４と、これら通過フィルタ２３，２４の各通過周波数成分に基づいてｆｓ／２の周波数成分を検出するｆｓ／２検出回路２５とを有する構成となっている。
【００２０】
ｆｓ／２検出部１６はさらに、輝度Ｙ０のレベルを所定の補正係数を掛けることによって補正する輝度補正回路２６を有している。そして、この輝度補正回路２６で補正された輝度レベルは、ｆｓ／２検出回路２５に供給される。ｆｓ／２検出回路２５は、ｆｓ／２，ｆｓ／４通過フィルタ２３，２４の各通過周波数成分を比較し、その比較結果からｆｓ／２の周波数成分の存在の有無を検出する構成となっている。
【００２１】
具体的には、ｆｓ／２，ｆｓ／４通過フィルタ２３，２４の各通過周波数成分の差分を求め、その差分から輝度補正回路２６で補正された輝度レベルを引いた結果が正ならば、ｆｓ／２の周波数成分が存在するものとする。ここで、ｆｓ／２，ｆｓ／４通過フィルタ２３，２４の各通過周波数成分の差分から輝度レベルを引くのは、輝度によってスレッシュホールドが変動するのを抑制するためである。以上により、ｆｓ／２検出系２７が構成されている。図７は、輝度Ｙ０の検出の概念図である。
【００２２】
ｆｓ／２検出部１６にはさらに、色に関する情報、具体的には被写体が例えば５×５画素の領域で無彩色（＝白）であるか否かを検出する無彩色検出系２８が設けられている。この無彩色検出系２８は、Ｒ／Ｇ／Ｂ信号の水平方向の色差レベルを算出する水平色差レベル算出回路２９と、垂直方向の色差レベルを算出する垂直色差レベル算出回路３０と、これら色差レベル算出回路２９，３０の各出力に基づいてｆｓ／２相関検出を行い、水平／垂直方向の補間係数ＷｈＨＧａｉｎ／ＷｈＶＧａｉｎを出力するｆｓ／２相関検出回路３１と、色差レベル算出回路２９，３０の各出力に基づいて無彩色の検出を行い、その無彩色レベルに応じた補間係数ＷｈＧａｉｎを出力する無彩色検出回路３２とを有する構成となっている。
【００２３】
図８は、ｆｓ／２検出回路２５におけるｆｓ／２検出（Ａ）および無彩色検出（Ｂ）の概念図である。先ず、図８（Ａ）のｆｓ／２検出系２７において、ｆｓ／２通過フィルタ２３を通過した水平ｆｓ／２成分および垂直ｆｓ／２成分は、絶対値化（ＡＢＳ;absolute)回路３３，３４で絶対値化された後、加算器３５で加算される。同様にして、ｆｓ／４通過フィルタ２４を通過した水平ｆｓ／４成分および垂直ｆｓ／４成分は、絶対値化回路３６，３７で絶対値化された後、加算器３８で加算される。なお、輝度補正回路２６においては、乗算器３９で輝度Ｙ０のレベルに所定の補正係数αを掛ける処理が行われる。
【００２４】
加算器３５の加算出力は、乗算器４０で所定の補正係数βが掛けられた後、減算器４１の一方の入力Ａとなる。また、加算器３８の加算出力は、加算器４２で輝度Ｙ０のレベルに所定の補正係数αを掛けたものが加算され、さらに加算器４３で所定のオフセット値γが付与された後、減算器４１の他方の入力Ｂとなる。ここで、補正係数βおよびオフセット値γは、スレッシュホールドを決めるパラメータとなる。
【００２５】
減算器４１は一方の入力Ａから他方の入力Ｂを減算する。すなわち、減算器４１において、ｆｓ／２成分（Ａ）とｆｓ／４成分（Ｂ）とを比較し（Ａ−Ｂ）、その差分（Ａ−Ｂ）が正、即ちｆｓ／２成分＞ｆｓ／４成分であれば、ｆｓ／２の周波数成分が存在する旨の判定結果を出す。そして、この差分（Ａ−Ｂ）がルックアップテーブル（ＬＵＴ）４４を通して補間係数Ｇ２Ｇａｉｎとして出力される。すなわち、検出されたｆｓ／２の周波数成分が多ければ多い程、ルックアップテーブル４４からは大きな値の補間係数Ｇ２Ｇａｉｎが出力される。
【００２６】
次に、図８（Ｂ）の無彩色検出系２８において、Ｒ／Ｇ／Ｂの信号は、垂直方向、水平方向のバンドパスフィルタ４５，４６を通過し、絶対値化回路４７，４８で絶対値化された後、バンドパスフィルタ４５，４６とは直交した方向にローパスフィルタ４９，５０を通過する。これにより、垂直方向の色差レベルＷｈＶおよび水平方向の色差レベルＷｈＨが算出される。そして、垂直、水平方向の色差レベルＷｈＶ，レベルＷｈＨは、減算器５１にその２入力Ａ，Ｂとして供給されるとともに、最小値算出回路５２にも供給される。
【００２７】
減算器５１は、一方の入力Ａから他方の入力Ｂを減算することで、垂直方向の色差レベルＷｈＶ（Ａ）と水平方向の色差レベルＷｈＨ（Ｂ）とを比較する。この比較結果から、垂直方向の相関が強いか、水平方向の相関が強いを判定することができる。すなわち、ＷｈＨ＞ＷｈＶのときは垂直方向の相関が強い、即ち縦縞であると判定し、ＷｈＶ＞ＷｈＨのときは垂直方向の相関が強い、即ち縦縞であると判定する。そして、減算器５１の減算出力（Ａ−Ｂ）は、ルックアップテーブル５３を通して、垂直方向／水平方向の相関の度合いに応じた補間係数ＷｈＶＧａｉｎ／ＷｈＨＧａｉｎとして出力される。なお、垂直方向の補間係数ＷｈＶＧａｉｎと水平方向の補間係数ＷｈＨＧａｉｎとは相補的な関係にあり、ＷｈＶＧａｉｎ＋ＷｈＨＧａｉｎは常に一定である。
【００２８】
また、最小値算出回路５２は、垂直方向の色差レベルＷｈＶおよび水平方向の色差レベルＷｈＨのうち、小さい方の数値を減算器５４にその一方の入力Ａとして供給する。減算器５４の他方の入力Ｂとしては、輝度Ｙ０に乗算器５５で所定の補正係数δを掛け、さらに減算器５６で所定のオフセット値εを減算して得られる参照値が供給される。減算器５４は、色差レベルＷｈＶ，ＷｈＨの小さい方の数値Ａと参照値Ｂとを比較し（Ａ−Ｂ）、その比較結果が負、即ち数値Ａが参照値Ｂよりも小さければ無彩色（白）と判定する。そして、減算器５４の減算出力（Ａ−Ｂ）は、ルックアップテーブル５７を通して、無彩色レベルに応じた補間係数ＷｈＧａｉｎとして出力される。
【００２９】
図９は、ＶＨ相関検出部１７および斜め相関検出部１８の具体的な構成の一例を示すブロック図である。
【００３０】
ＶＨ相関検出部１７は、補間画素の右側の画素の画素情報に基づいて相関の程度を示す相関値を算出する右側相関値算出回路６１と、補間画素の左側の画素の画素情報に基づいて相関値を算出する左側相関値算出回路６２と、補間画素の上側の画素の画素情報に基づいて相関値を算出する上側相関値算出回路６３と、補間画素の下側の画素の画素情報に基づいて相関値を算出する下側相関値算出回路６４と、これら相関値算出回路６１〜６４で算出した各相関値を補間ゲインに変換して出力する相関値→補間ゲイン変換回路６５とから構成されている。
【００３１】
上記構成のＶＨ相関検出部１７において、相関値→補間ゲイン変換回路６５からは、水平垂直補間用ゲインＲＧａｉｎ，ＬＧａｉｎ，ＴＧａｉｎ，ＢＧａｉｎおよび水平垂直ＲＢ補間用ゲインＲＧａｉｎＤ，ＬＧａｉｎＤ，ＴＧａｉｎＤ，ＢＧａｉｎＤが補間係数として出力される。
【００３２】
斜め相関検出部１８は、補間画素の右上側の画素の画素情報に基づいて相関値を算出する右上側相関値算出回路６６と、補間画素の左上側の画素の画素情報に基づいて相関値を算出する左上側相関値算出回路６７と、補間画素の左下側の画素の画素情報に基づいて相関値を算出する左下側相関値算出回路６８と、補間画素の右下側の画素の画素情報に基づいて相関値を算出する右下側相関値算出回路６９と、これら相関値算出回路６６〜６９で算出した各相関値を補間ゲインに変換し、補間係数として斜め補間用ゲインＤ１Ｇａｉｎ〜Ｄ４Ｇａｉｎを出力する相関値→補間ゲイン変換回路７０とから構成されている。
【００３３】
さらに、水平垂直４方向と斜め４方向は互いに直交していないため、ＶＨ‐斜め比較回路６０において、水平垂直の相関値と斜めの相関値を比較することにより、斜め補間補正用ゲインＶＨＧａｉｎ，ＤＧａｉｎを補間係数として算出するようにしている。
【００３４】
続いて、補間部１５におけるＲ／Ｇ／Ｂの各補間部およびｆｓ補正処理部２２の各構成例について説明する。
【００３５】
図１０は、Ｇ補間部２０の具体的な構成の一例を示すブロック図である。図１０において、色分離後のＧのデータは水平垂直４方向、即ち右側、左側、上側、下側の各相関用処理回路７１，７２，７３，７４にそれぞれ供給され、これら相関用処理回路７１，７２，７３，７４において４方向の補間データＧｒ，Ｇｌ，Ｇｔ，Ｇｂが生成される。これら補間データＧｒ，Ｇｌ，Ｇｔ，Ｇｂは、補間する方向にＬＰＦを通すことで生成される。
【００３６】
補間データＧｒ，Ｇｌ，Ｇｔ，Ｇｂは各々、乗算器７５，７６，７７，７８において、図９に示すＶＨ相関検出部１７で決定された補間係数、即ち水平垂直補間用ゲインＲＧａｉｎ，ＬＧａｉｎ，ＴＧａｉｎ，ＢＧａｉｎがそれぞれ掛けられる。そして、加算器７９〜８１で加算されることにより、Ｇの補間処理が行われる。補間処理後のＧの画像データは、（１）式で表される。

【００３７】
斜め４方向についても同様に、右上側、左上側、左下側、右下側の各相関用処理回路８２，８３，８４，８５において、Ｄ１／Ｄ２／Ｄ３／Ｄ４の各方向の補間データが生成され、乗算器８６，８７，８８，８９において、図９に示す斜め相関検出部１８で決定された斜め補間用ゲインＤ１Ｇａｉｎ，Ｄ２Ｇａｉｎ，Ｄ３Ｇａｉｎ，Ｄ４Ｇａｉｎがそれぞれ掛けられた後、加算器９０〜９２で足し合わされることによって補間処理が行われる。
【００３８】
また、水平垂直４方向相関検出／補間によるＧの画像データＧｖｈと、斜め４方向相関検出／補間によるＧの画像データＧｄを、状況に合わせて（２）式に示すように混合比を変化させて加算する。混合比の調整は、図９に示すＶＨ‐斜め比較回路６０で算出される斜め補間補正用ゲインＶＨＧａｉｎ，ＤＧａｉｎによって行われる。
Ｇ＝Ｇｖｈ×ＶＨＧａｉｎ＋Ｇｄ×ＤＧａｉｎ ……（２）
【００３９】
図１１は、Ｒ，Ｂ補間部１９，２１の具体的な構成の一例を示すブロック図である。このＲ，Ｂ補間部１９，２１の構成については、図１０に示すＧ補間部２０の構成と基本的に同じである。したがって、図１１中、図１０と同等部分には同一符号を付して示してある。ただし、右側、左側、上側、下側の各相関用処理回路７１′，７２′，７３′，７４′における補間データの生成法については、Ｇの場合に比べて複雑である。その理由として２つあり、ＲＢ補間用相関検出を行うことと、Ｇ／２成分をＲＢに加えるためである。
【００４０】
そのため、ＲＧ補間用相関検出は、水平（右、左）の補間データ用と垂直方向（上、下）の補間データ用では別々に行う。具体的には、Ｒ画素およびＢ画素の信号からＲ信号／Ｂ信号を補間する際に、先述した水平垂直ＲＧ補間用ゲインＲＧａｉｎＤ，ＬＧａｉｎＤ，ＴＧａｉｎＤ，ＢＧａｉｎＤを、Ｒ／Ｂ専用の補間係数として算出して用いる。これについては、本発明の要旨ではないので、ここではその詳細な説明は省略する。
【００４１】
図１２は、ｆｓ／２補正処理部２２の具体的な構成の一例を示すブロック図である。
【００４２】
図１２において、ＲＧＢ信号は、無彩色用処理回路１０１および有彩色用処理回路１０２に供給される。無彩色用処理回路１０１は、入力画像に対して水平方向の補間処理を行う水平補間回路１０３と、入力画像に対して垂直方向の補間処理を行う垂直補間回路１０４と、これら補間回路１０３，１０４の各出力に対してｆｓ／２検出部１６から与えられる補間係数ＷｈＨＧａｉｎ，ＷｈＶＧａｉｎを掛ける乗算器１０５，１０６と、これら乗算器１０５，１０６の各乗算出力を加算する加算器１０７とを有する構成となっている。
【００４３】
この無彩色用処理回路１０１を経た信号は、乗算器１０８でｆｓ／２検出部１６から与えられる補間係数ＷｈＧａｉｎが掛けられた後、加算器１０９の一方の入力となる。加算器１０９の他方の入力としては、有彩色用処理回路１０２を経た信号に、乗算器１１０でｆｓ／２検出部１６から与えられる補間係数ＷｈＧａｉｎから算出される補間係数ＷｈＧａｉｎ′を掛けたものが与えられる。補間係数ＷｈＧａｉｎ′は、例えば、定数より補間係数ＷｈＧａｉｎを引くことにより算出される。なお、有彩色用処理回路１０２は、図１３の特性図に示すように、入力画像の空間サンプリング周波数ｆｓの１／２の周波数成分のレスポンスが０であるフィルタ特性を持つｆｓ／２トラップ回路によって構成されている。
【００４４】
加算器１０９の加算出力は、乗算器１１１でｆｓ／２検出部１６から与えられる補間係数Ｇ２Ｇａｉｎが掛けられた後、加算器１１２の一方の入力となる。また、Ｒ／Ｇ／Ｂの各補間部１９，２０，２１から供給される各補間出力ＲＩＰ／ＧＩＰ／ＢＩＰは、乗算器１１３でｆｓ／２検出部１６から与えられる補間係数Ｇ２Ｇａｉｎから算出される補間係数Ｇ２Ｇａｉｎ′が掛けられた後、加算器１１２の他方の入力となる。なお、補間係数Ｇ２Ｇａｉｎ′は、例えば、定数より補間係数Ｇ２Ｇａｉｎを引くことにより算出される。図１４は、ｆｓ／２補正処理部２２における補正処理の概念図である。この概念図から明らかなように、補正処理はＲ／Ｇ／Ｂの各信号ごとに行われる。
【００４５】
上記構成の画像処理装置において、ｆｓ／２検出部１６では先ず、図５に示すｆｓ／２検出系２７によってｆｓ／２の周波数成分の存在の有無が検出される。すなわち、色分離後のＧ信号をｆｓ／２通過フィルタ２３およびｆｓ／４通過フィルタ２４に通し、ｆｓ／２検出回路２５において、それらの差分から輝度Ｙ０のレベルに所定の補正係数αを掛けたものを減算し、その減算結果が正ならば、ｆｓ／２の周波数成分が存在するものとする。
【００４６】
また、無彩色検出系２８では、被写体が例えば５×５画素の領域で無彩色であるか否かの検出が行われる。すなわち、図８（Ｂ）から明らかなように、垂直方向、水平方向それぞれに対してバンドパスフィルタ４５，４６を通した後、絶対値化回路４７，４８で絶対値をとり、それをバンドパスフィルタ４５，４６とは直交した方向にローパスフィルタ４９，５０を通した値のうち、小さい方の数値（ＷｈＶ／ＷｈＨ）が、輝度Ｙ０に所定の補正係数δを掛けた値よりも小さければ、無彩色（白）と判定する。
【００４７】
そして、ｆｓ／２補正処理部２２においては、ｆｓ／２検出部１６のｆｓ／２検出系２７から与えられる補間係数Ｇ２Ｇａｉｎ，Ｇ２Ｇａｉｎ′、並びに無彩色検出系２８から与えられる補間係数ＷｈＨＧａｉｎ，ＷｈＶＧａｉｎおよび補間係数ＷｈＧａｉｎ，ＷｈＧａｉｎ′に基づいて、ｆｓ／２の周波数成分の存在を検出しないとき、また検出した場合であって無彩色のとき又は有彩色のときにそれぞれ最適な補正処理が実行される。これら各補正処理について、以下に説明する。
【００４８】
先ず、ｆｓ／２の周波数成分の存在を検出しない場合には、ｆｓ／２付近の相関検出の誤検出の問題は生じないことから、通常のＲＧＢ補間処理が行われる。すなわち、ｆｓ／２の検出成分が少ないことから、ｆｓ検出部１６のｆｓ／２検出系２７から与えられる補間係数Ｇ２Ｇａｉｎが小さく、かつ補間係数Ｇ２Ｇａｉｎ′が大きいことから、加算器１１２での混合の比率は、Ｒ／Ｇ／Ｂの各補間部１９，２０，２１から供給される各補間出力ＲＩＰ／ＧＩＰ／ＢＩＰ側が大となり、補間出力ＲＩＰ／ＧＩＰ／ＢＩＰが主として出力される。
【００４９】
次に、ｆｓ／２の周波数成分の存在を検出した場合には、無彩色検出系２８の判定結果に基づいて補正処理を行う。先ず、無彩色でない、つまり有彩色であると判定された場合には、無彩色検出系２８から与えられる補間係数ＷｈＧａｉｎが小さく、かつ補間係数ＷｈＧａｉｎ′が大きいことから、加算器１０９での混合の比率は、有彩色用処理回路１０２側が大となり、有彩色用処理回路１０２でｆｓ／２の周波数成分が除去されたＲ／Ｇ／Ｂの各信号が主として出力される。このとき、加算器１１２での混合の比率は、ｆｓ／２の周波数成分の検出時であるから、有彩色用処理回路１０２側が大となっている。
【００５０】
一方、ｆｓ／２の周波数成分の存在を検出した場合において、無彩色であると判定された場合には、ｆｓ／２の領域であっても、水平相関が強いか、垂直相関が強いかを判定できるため、図１２に示す無彩色処理回路１０１において、無彩色検出で算出された数値、即ち無彩色検出系２８から与えられる補間係数ＷｈＨＧａｉｎ，ＷｈＶＧａｉｎを用いて水平補間回路１０３を経た信号と垂直補間回路１０４を経た信号との混合比を変えることによって、水平補間または垂直補間を行う。この場合、無彩色が前提であるため、図１４の概念図から明らかなように、ＲＢはＧで代用する。
【００５１】
上述したように、ｆｓ／２の周波数成分の存在を検出した場合において、有彩色と判定した場合には、Ｒ／Ｇ／Ｂの各補間部１９，２０，２１で補間処理された補間出力ＲＩＰ／ＧＩＰ／ＢＩＰを使用せずに、ｆｓ／２の周波数成分が除去されたＲ／Ｇ／Ｂの信号を使用するようにしたことで、相関検出による信号処理を行う際に、ｆｓ／２の空間周波数があっても、ｆｓ／２付近の相関検出の誤検出を防止できるため、破綻のない画像の再現が可能となる。
【００５２】
しかも、ｆｓ／２の周波数成分を検出した場合において、無彩色と判定した場合には、ｆｓ／２の周波数成分が除去されたＲ／Ｇ／Ｂの信号を使用せずに、無彩色検出で算出した数値を持って水平補間または垂直補間を行うようにしたことで、無彩色を撮像した場合における解像度をｆｓ／２相当まで上げることが可能となる。一例として、ＶＧＡ(Video Graphics Array)（６４０×４８０）画素の場合において、無彩色の被写体では解像度がｆｓ／２相当である４８０本まで改善できることになる。
【００５３】
図１５は、本発明に係るカメラの一例を示す概略構成図である。図１５において、被写体からの入射光は、レンズ１２１等を含む光学系によってＣＣＤエリアセンサ１２２の受光面（撮像面）上に結像される。ＣＣＤエリアセンサ１２２の受光面上には、色配列が例えば原色ベイヤ配列のカラーフィルタ１２３が設けられている。ＣＣＤエリアセンサ１２２は、ＣＣＤ駆動回路１２４によって露光、信号電荷の読み出しおよび転送などの駆動制御が行われる。
【００５４】
ＣＣＤエリアセンサ１２２の出力信号は画像処理装置１２５に供給され、種々の信号処理が行われる。この画像処理装置１２５として、水平垂直および斜めの８方向の相関を検出し、その検出結果に基づいて適応型補間処理を行うとともに、ｆｓ／２の周波数成分の検出時において、無彩色の場合と有彩色の場合で補間処理を変える構成の上記実施形態に係る画像処理装置が用いられる。
【００５５】
このように、例えば原色ベイヤ配列のカラーフィルタ１２３を持つＣＣＤエリアセンサ１２２を撮像デバイスとして用いたカメラにおいて、相関検出による信号処理を行う際に、ｆｓ／２の空間周波数成分が存在しても、ｆｓ／２付近の相関検出の誤検出を防止でき、破綻のない画像の再現が可能になるとともに、光学ＬＰＦを用いてｆｓ／２付近のレスポンスを０とする必要がなくなり、レスポンスが０となる空間周波数を高くすることが可能となるため、再現画像の解像度／解像感を向上できる。
【００５６】
【発明の効果】
以上説明したように、本発明によれば、ベイヤ配列のカラーフィルタを受光面上に有する固体撮像素子から出力される画像信号に対して、補間すべき画素とその周辺画素との相関を検出して色補正処理を行うに当たり、前記補間すべき画素に関して補間する方向にローパスフィルタを通すことによって得られる補間データに補間係数を乗算し、その乗算結果を加算することによって補間処理を行う際に、ｆｓ／２の周波数成分の存在を検出し、その周波数成分の存在を検出した場合において、有彩色と判定したときには、ｆｓ／２の周波数成分が除去されたＲ／Ｇ／Ｂの各信号を使用するようにしたことにより、ｆｓ／２付近の相関検出の誤検出を防止できるため、破綻のない画像が再現可能となり、また無彩色と判定したときには、無彩色検出で算出した数値を持って水平補間または垂直補間を行うようにしたことにより、無彩色を撮像した場合における解像度をｆｓ／２相当まで上げることが可能となる。
【図面の簡単な説明】
【図１】本発明による画像処理装置の基本構成を示すブロック図である。
【図２】カラーフィルタの原色ベイヤ配列図である。
【図３】検出部の構成の一例を示すブロック図である。
【図４】補間部の構成の一例を示すブロック図である。
【図５】ｆｓ／２検出部の具体的な構成の一例を示すブロック図である。
【図６】ｆｓ／２検出部における通過フィルタの空間周波数‐レスポンスの特性図である。
【図７】輝度Ｙ０の検出の概念図である。
【図８】ｆｓ／２検出の概念図であり、（Ａ）はｆｓ／２検出系を、（Ｂ）は無彩色検出系をそれぞれ示している。
【図９】ＶＨ相関検出部および斜め相関検出部の具体的な構成の一例を示すブロック図である。
【図１０】Ｇ補間部の具体的な構成の一例を示すブロック図である。
【図１１】Ｒ，Ｂ補間部の具体的な構成の一例を示すブロック図である。
【図１２】ｆｓ／２補正処理部の具体的な構成の一例を示すブロック図である。
【図１３】ｆｓ／２トラップ回路の特性図である。
【図１４】ｆｓ／２補正処理の概念図である。
【図１５】本発明に係るカメラの一例を示す概略構成図である。
【図１６】課題を説明するための図である。
【符号の説明】
１１，１２３…カラーフィルタ、１２，１２２…ＣＣＤエリアセンサ、１４…検出部、１５…補間部、１６…ｆｓ／２検出部、１７…ＶＨ相関検出部、１８…斜め相関検出部、１９…Ｒ補間部、２０…Ｇ補間部、２１…Ｂ補間部、２２…ｆｓ／２補正処理部、２３…ｆｓ／２通過フィルタ、２４…ｆｓ／４通過フィルタ、２５…ｆｓ／２検出回路、２６…輝度補正回路、２７…ｆｓ／２検出系、２８…無彩色検出系、２９…水平色差レベル算出回路、３０…垂直色差レベル算出回路、３１…ｆｓ／２相関検出回路、３２…無彩色検出回路[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing device, an image processing method, and a camera, and more particularly, to an image processing device that processes an output signal of a solid-state imaging device having a color filter on a light receiving surface, a processing method thereof, and a camera using these.
[0002]
[Prior art]
Conventionally, an optical LPF (Low Pass Filter) is used to remove a half of the spatial sampling frequency fs for the purpose of removing aliasing noise before image information enters the solid-state imaging device. There were restrictions. As described above, by performing band limitation using the optical LPF, the output signal of the solid-state imaging device does not include a frequency component in the vicinity of 1/2 of the spatial sampling frequency fs. The image will be lost. On the other hand, in order not to impair the sense of resolution of the captured image, instead of using the optical LPF having the characteristics as described above, the response at a point higher than half the frequency of the spatial sampling frequency fs. Should be 0.
[0003]
[Problems to be solved by the invention]
However, if the response at a point higher than the frequency of fs / 2 is set to 0, not only a false signal is generated, but also the color arrangement of the primary color Bayer arrangement of, for example, R (red), G (green), and B (blue). When performing signal processing by correlation detection in a solid-state imaging device having a color filter, when detecting a correlation by G, if there is a spatial frequency of fs / 2, the vertical correlation is strong or the horizontal correlation is strong. It becomes impossible to determine whether or not the image quality deteriorates.
[0004]
That is, in FIG. 16, a horizontal fs / 2 image (b) signal or a vertical fs / 2 image (c) signal with respect to the pixel array of the solid-state imaging device corresponding to the primary color RGB Bayer array (a). When the signal comes in, the output signal (d) when focusing only on the G pixel is the same for both horizontal fs / 2 (b) and vertical fs / 2 (c). Therefore, it is impossible to determine whether the vertical correlation is strong (horizontal fs / 2) or the horizontal correlation is strong (vertical fs / 2) only by looking at the G output signal.
[0005]
In FIG. 16A, Gr / Gb indicates G pixels in the R row / G pixels in the B row, respectively. In FIG. 4D, R / G pixels are displayed in black, and attention should be paid to gray (scattered) pixels and white pixels.
[0006]
The present invention has been made in view of the above problems, and the object of the present invention is to reliably determine whether the vertical correlation is strong or the horizontal correlation is strong, and it is possible to reproduce an image without failure. Another object is to provide an image processing apparatus and an image processing method.
[0007]
[Means for Solving the Problems]
An image processing apparatus according to the present invention includes: Bayer When color correction processing is performed by detecting the correlation between a pixel to be interpolated and its surrounding pixels, an image signal output from a solid-state imaging device having an array of color filters on the light receiving surface should be interpolated. An image processing apparatus for performing interpolation processing by multiplying interpolation data obtained by passing a low-pass filter in a direction to be interpolated with respect to a pixel and adding the multiplication result, and for a spatial sampling frequency of an input image A frequency detection circuit that detects the presence of a frequency component that is ½ times the color, a color information detection circuit that detects whether a predetermined area of the subject is an achromatic color based on each signal of R / G / B, and R An R interpolation unit that performs the interpolation process on the pixel information of G, a G interpolation unit that performs the interpolation process on the G pixel information, a B interpolation unit that performs the interpolation process on the B pixel information, in front When the frequency detection circuit does not detect the presence of the frequency component, it outputs the interpolation results of the R interpolation unit, the G interpolation unit, and the B interpolation unit, and the frequency detection circuit detects the presence of the frequency component. When the information detection circuit does not detect that the color is achromatic, the frequency component is removed from each R / G / B signal and output, and the frequency detection circuit detects the presence of the frequency component and When the information detection circuit detects an achromatic color, a correction is performed to perform the interpolation processing in the horizontal / vertical direction using an interpolation coefficient based on the color difference level in the horizontal / vertical direction of each R / G / B signal. And a processing unit.
[0008]
An image processing method according to the present invention includes: Bayer When color correction processing is performed by detecting the correlation between a pixel to be interpolated and its surrounding pixels, an image signal output from a solid-state imaging device having an array of color filters on the light receiving surface should be interpolated. An image processing method for performing interpolation processing by multiplying interpolation data obtained by passing a low-pass filter in a direction to be interpolated with respect to a pixel and adding the multiplication result, and each pixel of R / G / B While performing the interpolation processing on the information, the presence of a frequency component that is ½ times the spatial sampling frequency of the input image is detected, and a predetermined area of the subject is determined based on each R / G / B signal. Is detected, and if the presence of the frequency component is not detected, each interpolation result of the interpolation processing for each pixel information of the R / G / B is output, and the frequency component When detecting the presence and not detecting the achromatic color, the frequency component is removed from each R / G / B signal and output, and the presence of the frequency component is detected and the achromatic color is detected. When detected, interpolation processing is performed in the horizontal / vertical direction using an interpolation coefficient based on the color difference level in the horizontal / vertical direction of each R / G / B signal.
[0009]
In the image processing apparatus and the processing method with the above configuration, for example, when performing signal processing by correlation detection, if there is a half frequency component with respect to the spatial sampling frequency of the input image, the vertical correlation is strong or horizontal Since it is not possible to determine whether the correlation is strong, the frequency component that is ½ of the spatial sampling frequency is detected, and information about the color, specifically, achromatic color (= white) is detected. However, the detection of the achromatic color is meaningful only when a frequency component that is ½ of the spatial sampling frequency is detected. This is because if it is an achromatic color, it can be determined whether the horizontal correlation is strong or the vertical correlation is strong even in a half of the spatial sampling frequency.
[0010]
When a half frequency component of the spatial sampling frequency is detected, if it is determined that the color is achromatic, horizontal interpolation or vertical interpolation is performed using the numerical value calculated by the achromatic color detection. Thereby, at the time of an achromatic color, a characteristic can be improved to the resolution corresponding to 1/2 of the spatial sampling frequency. On the other hand, if it is determined that the color is not an achromatic color, that is, a chromatic color, R / G / B signals not including a frequency component that is 1/2 of the spatial sampling frequency are output. This can prevent erroneous detection of correlation detection in the vicinity of ½ of the spatial sampling frequency.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a basic configuration of an image processing apparatus according to the present invention. Here, the color solid-state imaging device to be processed by the image processing apparatus uses, for example, the color filter 11 of the primary color Bayer array of R (red) G (green) B (blue) shown in FIG. This is a single plate type solid-state imaging device 12.
[0012]
Note that the color arrangement is not limited to the primary color Bayer arrangement, and the color filter is not limited to the RGB primary color arrangement, and even in the case of other primary color arrangements, a color arrangement using complementary colors ( For example, the case of Ye / Cy / Mg / G) can be similarly applied. Further, as the solid-state imaging device 12, a so-called all-pixel readout type CCD (Charge Coupled Device) solid-state imaging device (hereinafter referred to as a CCD area sensor) that reads out signal charges of all the pixels independently is used. The present invention is also applicable to a CCD solid-state imaging device that is not a pixel readout method.
[0013]
The RGB point sequential data output from the CCD area sensor 12 is subjected to signal processing such as black level clamping and white balance in the signal processing unit 13 and then supplied to the detection unit 14 and the interpolation unit 15. The detection unit 14 detects an optimal interpolation method from the input RGB point sequential data and sends the interpolation information to the interpolation unit 15. The interpolation unit 15 performs an interpolation process on the RGB point sequential data based on the interpolation information input from the detection unit 14 and outputs the result.
[0014]
As shown in FIG. 3, the detection unit 14 detects the frequency component ½ of the spatial sampling frequency fs, and the vertical and horizontal sides of the pixel to be interpolated (hereinafter referred to as an interpolation pixel). VH correlation detection that detects the degree of correlation in a total of four directions, ie, four directions that form an integer multiple of 90 ° of each other, that is, two opposite directions in the vertical (V) direction and two opposite directions in the horizontal (H) direction And an oblique correlation detection unit 18 that detects the degree of correlation in the four directions that form an angle of 45 ° with respect to the four directions with respect to the interpolation pixels, that is, the upper right, upper left, lower left, and lower right diagonal directions. It has a configuration.
[0015]
In this example, a configuration that detects the degree of correlation in a total of eight directions in four oblique directions in addition to the four directions in the upper, lower, left, and right directions is used as an example. It may be. However, in the following description, the case of 8 directions will be described as an example.
[0016]
On the other hand, as illustrated in FIG. 4, the interpolation unit 15 performs an interpolation process on the R pixel information based on the interpolation information provided from the detection unit 14, and the G pixel information. Based on the detection information of the G interpolation unit 20 that performs the interpolation processing, the B interpolation unit 21 that performs the interpolation processing on the B pixel information, and the detection information of the detection unit 14 at fs / 2 Hatch The fs / 2 correction processing unit 22 performs a correction process for suppressing the generation of noise.
[0017]
Below, each structural example of the fs / 2 detection part 16, the VH correlation detection part 17, and the diagonal correlation detection part 18 in the detection part 14 is demonstrated.
[0018]
FIG. 5 is a block diagram illustrating an example of a specific configuration of the fs / 2 detection unit 16 in the detection unit 14. FIG. 6 shows the spatial frequency-response characteristics of the pass filter in the fs / 2 detector 16.
[0019]
This fs / 2 detector 16 has a filter characteristic in which the maximum value of the response of the passing frequency is a spatial frequency that is ½ of the spatial sampling frequency fs of the input image, that is, an fs / 2 pass filter having the characteristic A in FIG. 23, a filter characteristic in which the maximum value of the response of the passing frequency is ¼ the spatial frequency of the spatial sampling frequency fs of the input image, that is, the fs / 4 pass filter 24 having the characteristic B of FIG. 6, and these pass filters The fs / 2 detection circuit 25 detects the fs / 2 frequency component based on the

pass frequency components

23 and 24.
[0020]
The fs / 2 detector 16 further includes a luminance correction circuit 26 that corrects the level of the luminance Y0 by multiplying it by a predetermined correction coefficient. The luminance level corrected by the luminance correction circuit 26 is supplied to the fs / 2 detection circuit 25. The fs / 2 detection circuit 25 compares the pass frequency components of the fs / 2 and fs / 4 pass filters 23 and 24 and detects the presence / absence of the fs / 2 frequency component from the comparison result. Yes.
[0021]
Specifically, if the difference between the pass frequency components of the fs / 2 and fs / 4 pass filters 23 and 24 is obtained and the luminance level corrected by the luminance correction circuit 26 is subtracted from the difference, the result is fs. Assume that a frequency component of / 2 exists. Here, the reason why the luminance level is subtracted from the difference between the pass frequency components of the fs / 2 and fs / 4 pass filters 23 and 24 is to prevent the threshold from fluctuating due to the luminance. Thus, the fs / 2 detection system 27 is configured. FIG. 7 is a conceptual diagram of detection of the luminance Y0.
[0022]
The fs / 2 detector 16 is further provided with an achromatic color detection system 28 for detecting information on the color, specifically, whether or not the subject is an achromatic color (= white) in an area of 5 × 5 pixels, for example. ing. The achromatic color detection system 28 includes a horizontal color difference level calculation circuit 29 that calculates the color difference level in the horizontal direction of the R / G / B signal, a vertical color difference level calculation circuit 30 that calculates the color difference level in the vertical direction, and these color difference levels. The fs / 2 correlation detection circuit 31 that performs fs / 2 correlation detection based on the outputs of the

calculation circuits

29 and 30 and outputs horizontal / vertical interpolation coefficients WhHGain / WhVGain, and the color difference

level calculation circuits

29 and 30. The achromatic color detection circuit 32 detects an achromatic color based on the output and outputs an interpolation coefficient WhGain corresponding to the achromatic color level.
[0023]
FIG. 8 is a conceptual diagram of fs / 2 detection (A) and achromatic color detection (B) in the fs / 2 detection circuit 25. First, in the fs / 2 detection system 27 of FIG. 8A, the horizontal fs / 2 component and the vertical fs / 2 component that have passed through the fs / 2 pass filter 23 are absolute value (ABS)

circuits

33, 34. Is added to the absolute value by the adder 35. Similarly, the horizontal fs / 4 component and the vertical fs / 4 component that have passed through the fs / 4 pass filter 24 are converted into absolute values by the absolute

value conversion circuits

36 and 37 and then added by the adder 38. In the luminance correction circuit 26, the multiplier 39 multiplies the luminance Y0 level by a predetermined correction coefficient α.
[0024]
The added output of the adder 35 is multiplied by a predetermined correction coefficient β by the multiplier 40 and then becomes one input A of the subtractor 41. Further, the addition output of the adder 38 is obtained by adding the level of the luminance Y0 to the level of the luminance Y0 by the adder 42 and adding a predetermined offset value γ by the adder 43, and then adding the subtracter. 41, the other input B. Here, the correction coefficient β and the offset value γ are parameters that determine the threshold.
[0025]
The subtracter 41 subtracts the other input B from one input A. That is, the subtracter 41 compares the fs / 2 component (A) and the fs / 4 component (B) (A−B), and the difference (A−B) is positive, that is, fs / 2 component> fs / If there are four components, a determination result that a frequency component of fs / 2 exists is output. The difference (A−B) is output as an interpolation coefficient G2Gain through a lookup table (LUT) 44. That is, the greater the frequency component of the detected fs / 2, the larger the interpolation coefficient G2Gain that is output from the lookup table 44.
[0026]
Next, in the achromatic color detection system 28 of FIG. 8B, the R / G / B signal passes through the band-pass filters 45 and 46 in the vertical and horizontal directions, and is absolute by the absolute value conversion circuits 47 and 48. After being digitized, the low-

pass filters

49 and 50 pass in a direction orthogonal to the band-pass filters 45 and 46. Thereby, the color difference level WhV in the vertical direction and the color difference level WhH in the horizontal direction are calculated. The color difference levels WhV and WhH in the vertical and horizontal directions are supplied to the subtracter 51 as the two inputs A and B, and are also supplied to the minimum value calculation circuit 52.
[0027]
The subtractor 51 compares the color difference level WhV (A) in the vertical direction with the color difference level WhH (B) in the horizontal direction by subtracting the other input B from one input A. From this comparison result, it can be determined whether the correlation in the vertical direction is strong or the correlation in the horizontal direction is strong. That is, when WhH> WhV, it is determined that the correlation in the vertical direction is strong, that is, vertical stripes, and when WhV> WhH, it is determined that the correlation in the vertical direction is strong, that is, vertical stripes. Then, the subtraction output (A−B) of the subtractor 51 is output as an interpolation coefficient WhVGain / WhHGain according to the degree of correlation in the vertical / horizontal direction through the lookup table 53. Note that the vertical interpolation coefficient WhVGain and the horizontal interpolation coefficient WhHGain are in a complementary relationship, and WhVGain + WhHGain is always constant.
[0028]
Further, the minimum value calculation circuit 52 supplies the subtractor 54 with the smaller numerical value of the color difference level WhV in the vertical direction and the color difference level WhH in the horizontal direction as one input A thereof. As the other input B of the subtracter 54, a reference value obtained by multiplying the luminance Y0 by a predetermined correction coefficient δ by the multiplier 55 and further subtracting a predetermined offset value ε by the subtractor 56 is supplied. The subtractor 54 compares the numerical value A having the smaller color difference levels WhV and WhH with the reference value B (AB), and if the comparison result is negative, that is, the numerical value A is smaller than the reference value B, an achromatic color ( White). Then, the subtraction output (A−B) of the subtracter 54 is output as an interpolation coefficient WhGain corresponding to the achromatic color level through the lookup table 57.
[0029]
FIG. 9 is a block diagram illustrating an example of a specific configuration of the VH correlation detection unit 17 and the diagonal correlation detection unit 18.
[0030]
The VH correlation detection unit 17 calculates the correlation value indicating the degree of correlation based on the pixel information of the pixel on the right side of the interpolation pixel, and correlates based on the pixel information of the pixel on the left side of the interpolation pixel. A left correlation value calculation circuit 62 that calculates a value, an upper correlation value calculation circuit 63 that calculates a correlation value based on pixel information of an upper pixel of the interpolation pixel, and a pixel information of a lower pixel of the interpolation pixel A lower correlation value calculation circuit 64 that calculates correlation values, and a correlation value → interpolation gain conversion circuit 65 that converts the correlation values calculated by the correlation value calculation circuits 61 to 64 into interpolation gains and outputs them. Yes.
[0031]
In the VH correlation detection unit 17 configured as described above, the correlation value → interpolation gain conversion circuit 65 obtains the horizontal and vertical interpolation gains RGain, LGain, TGain, BGain and the horizontal / vertical RB interpolation gains RGainD, LGainD, TGainD, and BGainD as interpolation coefficients. Is output as
[0032]
The diagonal correlation detection unit 18 calculates the correlation value based on the pixel information of the upper right side pixel of the interpolation pixel and the upper right side correlation value calculation circuit 66 that calculates the correlation value based on the pixel information of the upper right side pixel of the interpolation pixel. The upper left correlation value calculation circuit 67 to calculate, the lower left correlation value calculation circuit 68 to calculate the correlation value based on the pixel information of the lower left pixel of the interpolation pixel, and the pixel information of the lower right pixel of the interpolation pixel Lower right side correlation value calculation circuit 69 for calculating a correlation value based on this, and converts each correlation value calculated by these correlation value calculation circuits 66 to 69 into an interpolation gain, and outputs diagonal interpolation gains D1 Gain to D4 Gain as interpolation coefficients. Correlation value → interpolation gain conversion circuit 70.
[0033]
Further, since the four horizontal and vertical directions and the four oblique directions are not orthogonal to each other, the VH-diagonal comparison circuit 60 compares the horizontal and vertical correlation values with the diagonal correlation values to obtain the diagonal interpolation correction gains VHGain and DGain. Is calculated as an interpolation coefficient.
[0034]
Next, configuration examples of the R / G / B interpolation units and the fs correction processing unit 22 in the interpolation unit 15 will be described.
[0035]
FIG. 10 is a block diagram illustrating an example of a specific configuration of the G interpolation unit 20. In FIG. 10, the G data after color separation is supplied to the respective

correlation processing circuits

71, 72, 73, 74 in four horizontal and vertical directions, that is, the right side, left side, upper side, and lower side. , 72, 73 and 74, four-way interpolation data Gr, Gl, Gt and Gb are generated. These interpolation data Gr, Gl, Gt, and Gb are generated by passing the LPF in the interpolation direction.
[0036]
The interpolation data Gr, Gl, Gt, and Gb are respectively multiplied by the interpolation coefficients determined by the VH correlation detection unit 17 shown in FIG. 9 in the

multipliers

75, 76, 77, and 78, that is, horizontal and vertical interpolation gains RGain, LGain, and TGain. , BGain, respectively. Then, addition is performed by the adders 79 to 81 to perform G interpolation processing. The G image data after the interpolation processing is expressed by equation (1).

[0037]
Similarly, in the four oblique directions, interpolation data in each direction of D1 / D2 / D3 / D4 is generated in the

correlation processing circuits

82, 83, 84, 85 on the upper right side, upper left side, lower left side, and lower right side. In the

multipliers

86, 87, 88, and 89, the diagonal interpolation gains D1Gain, D2Gain, D3Gain, and D4Gain determined by the diagonal correlation detector 18 shown in FIG. An interpolation process is performed by adding them.
[0038]
Further, the G image data Gvh by horizontal / vertical four-way correlation detection / interpolation and the G image data Gd by diagonal four-way correlation detection / interpolation are changed according to the situation as shown in equation (2). And add. The adjustment of the mixing ratio is performed by the diagonal interpolation correction gains VHGain and DGain calculated by the VH-oblique comparison circuit 60 shown in FIG.
G = Gvh × VHGain + Gd × DGain (2)
[0039]
FIG. 11 is a block diagram illustrating an example of a specific configuration of the R and

B interpolation units

19 and 21. The configurations of the R and

B interpolation units

19 and 21 are basically the same as the configuration of the G interpolation unit 20 shown in FIG. Therefore, in FIG. 11, the same parts as those in FIG. 10 are denoted by the same reference numerals. However, the method of generating interpolation data in each of the correlation processing circuits 71 ′, 72 ′, 73 ′, 74 ′ on the right side, the left side, the upper side, and the lower side is more complicated than the case of G. There are two reasons for this, to perform correlation detection for RB interpolation and to add a G / 2 component to RB.
[0040]
Therefore, the correlation detection for RG interpolation is performed separately for horizontal (right, left) interpolation data and vertical (up, down) interpolation data. Specifically, when the R signal / B signal is interpolated from the R pixel and B pixel signals, the above-described horizontal / vertical RG interpolation gains RGainD, LGainD, TGainD, and BGainD are calculated as interpolation coefficients dedicated to R / B. And use. Since this is not the gist of the present invention, detailed description thereof is omitted here.
[0041]
FIG. 12 is a block diagram illustrating an example of a specific configuration of the fs / 2 correction processing unit 22.
[0042]
In FIG. 12, the RGB signals are supplied to the achromatic color processing circuit 101 and the chromatic color processing circuit 102. The achromatic processing circuit 101 includes a horizontal interpolation circuit 103 that performs horizontal interpolation processing on the input image, a vertical interpolation circuit 104 that performs vertical interpolation processing on the input image, and these

interpolation circuits

103 and 104. And the

multipliers

105 and 106 that multiply the interpolation coefficients WhHGain and WhVGain given from the fs / 2 detector 16 to the outputs of the fs / 2, and the adder 107 that adds the multiplication outputs of the

multipliers

105 and 106, and It has become.
[0043]
The signal that has passed through the achromatic color processing circuit 101 is multiplied by the interpolation coefficient WhGain provided from the fs / 2 detector 16 by the multiplier 108 and then becomes one input of the adder 109. The other input of the adder 109 is obtained by multiplying the signal that has passed through the chromatic color processing circuit 102 by the interpolation coefficient WhGain ′ calculated from the interpolation coefficient WhGain supplied from the fs / 2 detector 16 by the multiplier 110. Given. The interpolation coefficient WhGain ′ is calculated by subtracting the interpolation coefficient WhGain from a constant, for example. As shown in the characteristic diagram of FIG. 13, the chromatic color processing circuit 102 is an fs / 2 trap circuit having a filter characteristic in which the response of a frequency component ½ of the spatial sampling frequency fs of the input image is zero. It is configured.
[0044]
The addition output of the adder 109 becomes one input of the adder 112 after being multiplied by the interpolation coefficient G2Gain given from the fs / 2 detector 16 by the multiplier 111. Further, the interpolation outputs RIP / GIP / BIP supplied from the R / G /

B interpolation units

19, 20, and 21 are calculated from the interpolation coefficient G2Gain provided from the fs / 2 detection unit 16 by the multiplier 113. After being multiplied by the interpolation coefficient G2Gain ', it becomes the other input of the adder 112. The interpolation coefficient G2Gain ′ is calculated by subtracting the interpolation coefficient G2Gain from a constant, for example. FIG. 14 is a conceptual diagram of correction processing in the fs / 2 correction processing unit 22. As is apparent from this conceptual diagram, the correction process is performed for each R / G / B signal.
[0045]
In the image processing apparatus configured as described above, the fs / 2 detector 16 first detects the presence or absence of the fs / 2 frequency component by the fs / 2 detection system 27 shown in FIG. That is, the color-separated G signal is passed through the fs / 2 pass filter 23 and the fs / 4 pass filter 24, and the fs / 2 detection circuit 25 multiplies the level of luminance Y0 by a predetermined correction coefficient α from the difference between them. If the result is subtracted and the result of the subtraction is positive, it is assumed that a frequency component of fs / 2 exists.
[0046]
The achromatic color detection system 28 detects whether or not the subject is an achromatic color in an area of 5 × 5 pixels, for example. That is, as apparent from FIG. 8B, after passing through the band-pass filters 45 and 46 in the vertical direction and the horizontal direction, the absolute values are taken by the absolute value conversion circuits 47 and 48, and the band-pass filters are obtained. If the smaller numerical value (WhV / WhH) among the values obtained by passing the low-

pass filters

49 and 50 in the direction orthogonal to the filters 45 and 46 is smaller than the value obtained by multiplying the luminance Y0 by a predetermined correction coefficient δ, Determined to be achromatic (white).
[0047]
In the fs / 2 correction processing unit 22, the interpolation coefficients G2Gain and G2Gain ′ given from the fs / 2 detection system 27 of the fs / 2 detection unit 16, and the interpolation coefficients WhHGin and WhVGain given from the achromatic color detection system 28, and Based on the interpolation coefficients WhGain and WhGain ′, optimal correction processing is executed when the presence of the frequency component of fs / 2 is not detected, or when it is detected and is achromatic or chromatic. Each of these correction processes will be described below.
[0048]
First, when the presence of the frequency component of fs / 2 is not detected, there is no problem of erroneous detection of correlation detection in the vicinity of fs / 2, and therefore normal RGB interpolation processing is performed. That is, since the detection component of fs / 2 is small, the interpolation coefficient G2Gain given from the fs / 2 detection system 27 of the fs detection unit 16 is small and the interpolation coefficient G2Gain 'is large. The ratio is large on the side of each interpolation output RIP / GIP / BIP supplied from each

interpolation unit

19, 20, 21 of R / G / B, and the interpolation output RIP / GIP / BIP is mainly output.
[0049]
Next, when the presence of the frequency component of fs / 2 is detected, correction processing is performed based on the determination result of the achromatic color detection system 28. First, when it is determined that the color is not an achromatic color, that is, a chromatic color, the interpolation coefficient WhGain given from the achromatic color detection system 28 is small and the interpolation coefficient WhGain 'is large. The ratio is large on the chromatic color processing circuit 102 side, and R / G / B signals from which the fs / 2 frequency component has been removed by the chromatic color processing circuit 102 are mainly output. At this time, since the mixing ratio in the adder 112 is when the frequency component of fs / 2 is detected, the chromatic color processing circuit 102 side is large.
[0050]
On the other hand, when the presence of the frequency component of fs / 2 is detected and it is determined that the color is achromatic, whether the horizontal correlation is strong or the vertical correlation is strong even in the fs / 2 region. 12 can be determined. In the achromatic color processing circuit 101 shown in FIG. 12, the numerical value calculated by the achromatic color detection, that is, the interpolation coefficient WhHGain and WhVGain given from the achromatic color detection system 28 is used and the signal passed through the horizontal interpolation circuit 103 is vertical. Horizontal or vertical interpolation is performed by changing the mixing ratio with the signal that has passed through the interpolation circuit 104. In this case, since an achromatic color is assumed, RB is substituted for G as apparent from the conceptual diagram of FIG.
[0051]
As described above, when the presence of the frequency component of fs / 2 is detected, if it is determined that the color component is a chromatic color, the interpolation output RIP subjected to interpolation processing by the R / G /

B interpolation units

19, 20, and 21. By using the R / G / B signal from which the frequency component of fs / 2 has been removed without using / GIP / BIP, when performing signal processing by correlation detection, Even if there is a spatial frequency, it is possible to prevent erroneous detection of correlation detection in the vicinity of fs / 2, so that an image without failure can be reproduced.
[0052]
In addition, when the frequency component of fs / 2 is detected and it is determined that the color is achromatic, it is possible to detect achromatic color without using the R / G / B signal from which the frequency component of fs / 2 is removed. By performing the horizontal interpolation or the vertical interpolation with the calculated numerical value, it becomes possible to increase the resolution when an achromatic color is imaged to the equivalent of fs / 2. As an example, in the case of VGA (Video Graphics Array) (640 × 480) pixels, the resolution of an achromatic subject can be improved to 480 lines corresponding to fs / 2.
[0053]
FIG. 15 is a schematic configuration diagram illustrating an example of a camera according to the present invention. In FIG. 15, incident light from a subject is imaged on a light receiving surface (imaging surface) of a CCD area sensor 122 by an optical system including a lens 121 and the like. On the light receiving surface of the CCD area sensor 122, a color filter 123 having a primary color Bayer array, for example, is provided. The CCD area sensor 122 is controlled by the CCD driving circuit 124 such as exposure, signal charge reading and transfer.
[0054]
The output signal of the CCD area sensor 122 is supplied to the image processing device 125, and various signal processing is performed. As the image processing device 125, the horizontal / vertical and diagonal eight-direction correlations are detected, adaptive interpolation processing is performed based on the detection results, and when the frequency component of fs / 2 is detected, an achromatic color case is used. The image processing apparatus according to the above-described embodiment configured to change the interpolation process in the case of a chromatic color is used.
[0055]
Thus, for example, in a camera using a CCD area sensor 122 having a color filter 123 of a primary color Bayer array as an imaging device, when performing signal processing by correlation detection, even if a spatial frequency component of fs / 2 is present, It is possible to prevent erroneous detection of correlation detection in the vicinity of fs / 2 and to reproduce an image without failure, and it is not necessary to set the response in the vicinity of fs / 2 to 0 using an optical LPF, and the response becomes 0. Since the spatial frequency can be increased, the resolution / resolution of the reproduced image can be improved.
[0056]
【The invention's effect】
As explained above, according to the present invention, Bayer array When detecting the correlation between a pixel to be interpolated and its surrounding pixels and performing color correction processing on an image signal output from a solid-state imaging device having a color filter on the light receiving surface, interpolation is performed on the pixel to be interpolated. When interpolation processing is performed by multiplying the interpolation data obtained by passing the low-pass filter in the direction to be performed and adding the multiplication results, the presence of the frequency component of fs / 2 is detected, and the frequency component is detected. In the case of detecting the presence of a chromatic color, the R / G / B signal from which the frequency component of fs / 2 is removed is used to detect correlation near fs / 2. Since false detection can be prevented, an image without failure can be reproduced, and when it is determined that the color is achromatic, horizontal or vertical interpolation is performed using the value calculated by achromatic color detection. By the the performed, the resolution in the case where the achromatic captured it is possible to raise up to fs / 2 equivalent.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a basic configuration of an image processing apparatus according to the present invention.
FIG. 2 is a primary color Bayer arrangement diagram of a color filter.
FIG. 3 is a block diagram illustrating an example of a configuration of a detection unit.
FIG. 4 is a block diagram illustrating an example of a configuration of an interpolation unit.
FIG. 5 is a block diagram illustrating an example of a specific configuration of an fs / 2 detection unit.
FIG. 6 is a characteristic diagram of the spatial frequency-response of the pass filter in the fs / 2 detector.
FIG. 7 is a conceptual diagram of detection of luminance Y0.
8A and 8B are conceptual diagrams of fs / 2 detection, where FIG. 8A shows an fs / 2 detection system and FIG. 8B shows an achromatic color detection system.
FIG. 9 is a block diagram illustrating an example of a specific configuration of a VH correlation detection unit and a diagonal correlation detection unit.
FIG. 10 is a block diagram illustrating an example of a specific configuration of a G interpolation unit.
FIG. 11 is a block diagram illustrating an example of a specific configuration of an R and B interpolation unit.
FIG. 12 is a block diagram illustrating an example of a specific configuration of an fs / 2 correction processing unit.
FIG. 13 is a characteristic diagram of the fs / 2 trap circuit.
FIG. 14 is a conceptual diagram of fs / 2 correction processing.
FIG. 15 is a schematic configuration diagram showing an example of a camera according to the present invention.
FIG. 16 is a diagram for explaining a problem;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 11,123 ... Color filter, 12,122 ... CCD area sensor, 14 ... Detection part, 15 ... Interpolation part, 16 ... fs / 2 detection part, 17 ... VH correlation detection part, 18 ... Oblique correlation detection part, 19 ... R Interpolation unit, 20 ... G interpolation unit, 21 ... B interpolation unit, 22 ... fs / 2 correction processing unit, 23 ... fs / 2 pass filter, 24 ... fs / 4 pass filter, 25 ... fs / 2 detection circuit, 26 ... Luminance correction circuit 27 ... fs / 2 detection system 28 ... achromatic color detection system 29 ... horizontal color difference level calculation circuit 30 ... vertical color difference level calculation circuit 31 ... fs / 2 correlation detection circuit 32 ... achromatic color detection circuit

Claims

When the color correction processing is performed by detecting the correlation between the pixel to be interpolated and the surrounding pixels for the image signal output from the solid-state imaging device having the Bayer array color filter on the light receiving surface, the interpolation is performed. An image processing apparatus that performs interpolation processing by multiplying interpolation data obtained by passing a low-pass filter in a direction to be interpolated with respect to a power pixel, and adding the multiplication result,
A frequency detection circuit that detects the presence of a frequency component that is ½ times the spatial sampling frequency of the input image;
An information detection circuit for detecting whether a predetermined area of the subject is an achromatic color based on each signal of R (red) / G (green) / B (blue);
An R interpolation unit that performs the interpolation processing on R pixel information;
A G interpolation unit that performs the interpolation processing on G pixel information;
A B interpolation unit for performing the interpolation processing on the B pixel information;
When the frequency detection circuit does not detect the presence of the frequency component, the interpolation results of the R interpolation unit, the G interpolation unit, and the B interpolation unit are output, and the frequency detection circuit detects the presence of the frequency component. When the information detection circuit does not detect that the color is achromatic, the frequency component is removed from each R / G / B signal and output, and the frequency detection circuit detects the presence of the frequency component. When the information detection circuit detects an achromatic color, the interpolation processing is performed in the horizontal / vertical direction using an interpolation coefficient based on the color difference level in the horizontal / vertical direction of each signal of R / G / B. An image processing apparatus comprising: a correction processing unit.

The frequency detection circuit includes a first filter in which a maximum response value of a passing frequency is a half of a spatial sampling frequency of an input image, and a maximum value of a response of a passing frequency is a spatial sampling of the input image. A detection unit that detects a frequency component that is ½ of the spatial sampling frequency by comparing each of the passing frequency components of the second filter that is ¼ of the frequency and the first and second filters. The image processing apparatus according to claim 1, further comprising:

The image processing apparatus according to claim 2, wherein the frequency detection circuit includes means for subtracting an average signal level of pixels in the vicinity of the pixel to be interpolated from a pass frequency component of the first filter.

The color information detection circuit detects a change amount in the horizontal direction and the vertical direction of the input image signal around the pixel to be interpolated, and the horizontal direction and the vertical direction detected by the change amount detection circuit. The image processing apparatus according to claim 1, further comprising: an achromatic color detection circuit that detects an achromatic color level based on each direction change amount.

The achromatic color detection circuit detects an achromatic color level by comparing a predetermined reference value with a minimum value of each change amount in the horizontal direction and the vertical direction detected by the change amount detection circuit. The image processing apparatus according to claim 4.

The correction processing unit includes an achromatic color processing circuit that performs the interpolation processing in the horizontal direction or the vertical direction with respect to the input image, and a chromatic color that removes a frequency component that is ½ of the spatial sampling frequency of the input image. Processing circuit, and a first mixing circuit that mixes a signal that has passed through the achromatic color processing circuit and a signal that has passed through the chromatic color processing circuit at a first mixing ratio according to the detection output of the color information detection circuit The image processing apparatus according to claim 1, further comprising:

The color information detection circuit includes an achromatic color detection circuit that detects an achromatic color level based on each change amount of the input image signal in the horizontal direction and the vertical direction around the pixel to be interpolated, and the first mixing circuit The first mixing ratio is changed so that a ratio of a signal which has passed through the processing circuit for achromatic color becomes large when an achromatic color level detected by the achromatic color detection circuit is large. 6. The image processing apparatus according to 6.

The color information detection circuit includes a mixing ratio setting circuit that compares the horizontal and vertical changes detected by the change detection circuit and sets a second mixing ratio according to the comparison result. The achromatic color processing circuit includes a first signal processing system that performs horizontal interpolation on the input image, a second signal processing system that performs vertical interpolation on the input image, and the first signal processing. The image processing apparatus according to claim 6, further comprising: a second mixing circuit that mixes a signal that has passed through the system and a signal that has passed through the second signal processing system at the second mixing ratio.

In the mixing ratio setting circuit, when the amount of change in the vertical direction is large compared to the amount of change in the horizontal direction, the ratio of the signal that has passed through the first signal processing system is large, and the horizontal ratio is larger than the amount of change in the vertical direction. The image processing apparatus according to claim 8, wherein when the amount of change in direction is large, the second mixing ratio is changed so that a ratio of signals that have passed through the second signal processing system is increased.

When the color correction processing is performed by detecting the correlation between the pixel to be interpolated and the surrounding pixels for the image signal output from the solid-state imaging device having the Bayer array color filter on the light receiving surface, the interpolation is performed. An image processing method for performing interpolation processing by multiplying interpolation data obtained by passing a low-pass filter in a direction to be interpolated with respect to a power pixel, and adding the multiplication result,
While performing the interpolation processing on each pixel information of R (red) / G (green) / B (blue),
Detects the presence of a frequency component that is ½ times the spatial sampling frequency of the input image, detects whether a predetermined area of the subject is achromatic based on each R / G / B signal,
When the presence of the frequency component is not detected, each interpolation result of the interpolation processing for each pixel information of the R / G / B is output, and the presence of the frequency component is detected and the achromatic color is not detected When the frequency components are removed from the R / G / B signals and output, and the presence of the frequency components is detected and the achromatic color is detected, the R / G / B signals are output. An image processing method comprising: performing correction processing for performing the interpolation processing in the horizontal / vertical direction using an interpolation coefficient based on a color difference level in the horizontal / vertical direction.

In the detection of the information on the color, each change amount in the horizontal direction and the vertical direction of the input image signal is detected around the pixel to be interpolated, and an achromatic color level is detected based on each change amount. The image processing method according to claim 10 .

The image processing method according to claim 11 , wherein the achromatic color level is detected by comparing a minimum value of each change amount in the horizontal direction and the vertical direction with a predetermined reference value.

In the correction process, a signal obtained by performing the interpolation process in the horizontal direction or the vertical direction on the input image, and a signal obtained by removing a frequency component of 1/2 of the spatial sampling frequency from the input image, The image processing method according to claim 11 , wherein mixing is performed at a mixing ratio corresponding to an achromatic color level.

For a solid-state image sensor having a Bayer array color filter on a light-receiving surface, an optical system for imaging incident light from a subject on the light-receiving surface of the solid-state image sensor, and an image signal output from the solid-state image sensor When the color correction processing is performed by detecting the correlation between the pixel to be interpolated and its surrounding pixels, the interpolation data obtained by passing the low-pass filter in the direction of interpolation for the pixel to be interpolated is multiplied by the interpolation coefficient. And an image processing apparatus that performs an interpolation process by adding the multiplication results,
The image processing apparatus includes:
A frequency detection circuit that detects the presence of a frequency component that is ½ times the spatial sampling frequency of the input image;
An information detection circuit for detecting whether a predetermined area of the subject is an achromatic color based on each signal of R (red) / G (green) / B (blue);
An R interpolation unit that performs the interpolation processing on R pixel information;
A G interpolation unit that performs the interpolation processing on G pixel information;
A B interpolation unit for performing the interpolation processing on the B pixel information;
When the frequency detection circuit does not detect the presence of the frequency component, the interpolation results of the R interpolation unit, the G interpolation unit, and the B interpolation unit are output, and the frequency detection circuit detects the presence of the frequency component. When the information detection circuit does not detect that the color is achromatic, the frequency component is removed from each R / G / B signal and output, and the frequency detection circuit detects the presence of the frequency component. When the information detection circuit detects an achromatic color, the interpolation processing is performed in the horizontal / vertical direction using an interpolation coefficient based on the color difference level in the horizontal / vertical direction of each signal of R / G / B. And a correction processing unit.

The color information detection circuit includes a change amount detection circuit that detects each change amount in the horizontal direction and the vertical direction of the input image signal around the pixel to be interpolated, and the horizontal direction detected by the change amount detection circuit and The camera according to claim 14, further comprising: an achromatic color detection circuit that detects an achromatic color level based on each amount of change in the vertical direction.

The correction processing unit includes an achromatic color processing circuit that performs the interpolation processing in the horizontal direction or the vertical direction with respect to the input image, and a chromatic color that removes a frequency component that is ½ of the spatial sampling frequency of the input image. And a mixing circuit that mixes the signal that has passed through the achromatic color processing circuit and the signal that has passed through the chromatic color processing circuit at a mixing ratio according to the detection output of the color information detection circuit. The camera according to claim 14 .