JP3926019B2 - Imaging device - Google Patents

Description

また、画素位置（２ｉ＋１、２ｊ）ではＢ信号の水平および垂直方向ローパスフィルタ１７および２２を介した値を例えば次式により得る。

【００３２】
なお、上記式（３）〜（８）は各水平方向ローパスフィルタおよび垂直方向ローパスフィルタ出力の算出例であり、フィルタのタップ数および係数は上記式（３）〜（８）に限るものではなく、他のタップ数および係数でもよい。
【００３３】
そして、上記水平方向ローパスフィルタ１６での出力Ｒｈｌｐｆは演算手段１８へと送られ、演算手段１８へはＲ信号と水平方向ローパスフィルタ１５の出力信号Ｇｈｌｐｆも入力されており、画素位置（２ｉ、２ｊ＋１）での水平方向の相関をもつＧ信号ｇｈ１（２ｉ、２ｊ＋１）を次式
ｇｈ１（２ｉ、２ｊ＋１）＝Ｒ（２ｉ、２ｊ＋１）×（Ｇｈｌｐｆ／Ｒｈｌｐｆ） （９）
により算出し、切り換え手段２５へと送る。また、垂直方向ローパスフィルタ２１での出力Ｒｖｌｐｆは演算手段２３へと送られ、演算手段２３へはＲ信号と垂直方向ローパスフィルタ２０の出力信号Ｇｖｌｐｆも入力されており、画素位置（２ｉ、２ｊ＋１）での垂直方向に相関をもつＧ信号ｇｖ１（２ｉ、２ｊ＋１）を次式
ｇｖ１（２ｉ、２ｊ＋１）＝Ｒ（２ｉ、２ｊ＋１）×（Ｇｖｌｐｆ／Ｒｖｌｐｆ） （１０）
により算出し、切り換え手段２５へと送る。
【００３４】
同様に、上記水平方向ローパスフィルタ１７での出力Ｂｈｌｐｆは演算手段１９へと送られ、演算手段１９へはＢ信号と水平方向ローパスフィルタ１５の出力信号Ｇｈｌｐｆも入力されており、画素位置（２ｉ＋１、２ｊ）での水平方向の相関をもつＧ信号ｇｈ２（２ｉ＋１、２ｊ）を次式
ｇｈ２（２ｉ＋１、２ｊ）＝Ｂ（２ｉ＋１、２ｊ）×（Ｇｈｌｐｆ／Ｂｈｌｐｆ） （１１）
により算出し、切り換え手段２５へと送る。また、垂直方向ローパスフィルタ２２での出力Ｂｖｌｐｆは演算手段２４へと送られ、演算手段２４へはＢ信号と垂直方向ローパスフィルタ２０の出力信号Ｇｖｌｐｆも入力されており、画素位置（２ｉ＋１、２ｊ）での垂直方向に相関をもつＧ信号ｇｖ２（２ｉ＋１、２ｊ）を次式
ｇｖ２（２ｉ＋１、２ｊ）＝Ｂ（２ｉ＋１、２ｊ）×（Ｇｖｌｐｆ／Ｂｖｌｐｆ） （１２）
により算出し、切り換え手段２５へと送る。
【００３５】
上記式（９）〜（１２）による算出方式は、局所的領域での色の変化が少ないことを前提としており、つまり、局所的な領域での各色信号の比はほぼ等しいことにより、水平方向または垂直方向における局所的な領域での各色信号の比はＲ、Ｇ、Ｂの水平方向または垂直方向のローパスフィルタを介した値の比で与えられる。
【００３６】
切り換え手段２５では、第１のエッジ判定手段５における判定手段１３からの周辺画素の信号レベル変化を判定した結果ｅｄ１と画素位置に基づき、Ｇの画素信号を上記平均値算出手段１４、演算手段１８、１９および演算手段２３、２４、分離手段４からの信号より選択し切り換える。つまり、判定手段１３による出力信号ｅｄ１が‘２’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ＞ΔＶで垂直方向に相関が高いと判定される場合は、各画素位置での垂直方向に相関を持つＧを出力する演算手段の信号を選択する。判定手段１３による出力信号ｅｄ１が‘３’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ≦ΔＶで水平方向に相関が高いと判定される場合には、各画素位置での水平方向に相関を持つＧを出力する演算手段の信号を選択する。さらに、判定手段１３による出力信号ｅｄ１が‘１’を示す場合には、ΔＨとΔＶの両方が予め定めた値ｔｈ１以下であり、周辺画素での信号レベルの変化がないと判定されるため、周波数の変化を考慮する必要がなく、平均値算出手段１４によるＧ信号ｇａを選択する。
【００３７】
すなわち、切り換え手段２５において、画素位置（２ｉ、２ｊ＋１）のＧ信号ｇ（２ｉ、２ｊ＋１）の場合は、判定手段１３からの出力ｅｄ１＝１では平均値算出手段１４からのＧ信号ｇａを、ｅｄ＝２の場合は演算手段２３からのＧ信号ｇｖ１を、ｅｄ＝３の場合は演算手段１８からのＧ信号ｇｈ１を選択し、一方、画素位置（２ｉ＋１、２ｊ）のＧ信号ｇ（２ｉ＋１、２ｊ）の場合は、判定手段１３からの出力ｅｄ１＝１では平均値算出手段１４からのＧ信号ｇａを、ｅｄ＝２の場合は演算手段２４からのＧ信号ｇｖ２を、ｅｄ＝３の場合は演算手段１９からのＧ信号ｇｈ２を選択する。なお、Ｇが得られている画素位置（２ｉ、２ｊ）、（２ｉ＋１、２ｊ＋１）においては、判定手段１３からは例えばｅｄ１＝０が出力されており、このときは分離手段４からのＧ信号をそのまま出力すればよい。
【００３８】
したがって、Ｇ成分復元手段６からは画素位置（２ｉ、２ｊ）、（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）、（２ｉ＋１、２ｊ＋１）それぞれの画素でのＧ信号が出力され、つまり、撮像素子の画素数分の解像度のＧ信号を得られる。このＧ成分復元手段６からの出力Ｇ０は次に、第１のＲＢ成分復元手段７、および第２のエッジ判定手段８と第２のＲＢ成分復元手段９へと送られる。
【００３９】
次に、第１のＲＢ成分復元手段７では、分離手段４からのＲ、Ｂ信号（図６ （ａ）および（ｂ））と上記Ｇ成分復元手段６からの全画素の信号をもつＧ信号Ｇ０により、Ｒ、Ｂ信号それぞれにおける画素位置（２ｉ、２ｊ）および（２ｉ＋１，２ｊ＋１）での画素を復元生成する。図７（ａ）、（ｂ）は第１のＲＢ成分復元手段７におけるＲ、Ｂ信号の算出を説明するための各画素のＲ、Ｂを示す図であり、図中ｒおよびｂは第１のＲＢ成分復元手段７において復元生成されるＲ、Ｂ画素信号を示している。Ｒ信号での画素位置（２ｉ、２ｊ）においては、水平方向に隣接するの２ｉラインの画素が撮像素子１より得られており、また、Ｇ信号はＧ成分復元手段６により全画素復元されている（図５）。よって、画素位置（２ｉ、２ｊ）において、Ｇ、Ｒ信号に対して水平方向のローパスフィルタを介した値Ｇ１ｈｌｐｆ、Ｒ１ｈｌｐｆを例えば、

により算出し、このＧ１ｈｌｐｆとＲ１ｈｌｐｆの比と画素Ｇ（２ｉ、２ｊ）により、画素位置（２ｉ、２ｊ）でのＲ信号ｒ（２ｉ、２ｊ）を次式により算出する。
ｒ（２ｉ、２ｊ）＝ Ｇ（２ｉ、２ｊ）×（Ｒ１ｈｌｐｆ／Ｇ１ｈｌｐｆ）
＝Ｇ（２ｉ、２ｊ）×（ Ｒ（２ｉ、２ｊ−１）＋Ｒ（２ｉ、２ｊ＋１））
／（ｇ（２ｉ、２ｊ−１）＋ｇ（２ｉ、２ｊ＋１）） （１５）
【００４０】
また、画素位置（２ｉ＋１、２ｊ＋１）においては、垂直方向に隣接する２ｊ＋１列の画素が撮像素子１より得られており、また、Ｇ信号はＧ成分復元手段６により全画素復元されているので、画素位置（２ｉ＋１、２ｊ＋１）において、Ｇ、Ｒ信号に対して垂直方向のローパスフィルタを介した値Ｇ１ｖｌｐｆ、Ｒ１ｖｌｐｆを例えば、

により算出し、このＧ１ｖｌｐｆとＲ１ｖｌｐｆの比と画素Ｇ（２ｉ＋１、２ｊ＋１）により、画素位置（２ｉ＋１、２ｊ＋１）でのＲ信号ｒ（２ｉ＋１、２ｊ＋１）を次式により算出する。
ｒ（２ｉ＋１、２ｊ＋１）＝Ｇ（２ｉ＋１、２ｊ＋１）×（Ｒ１ｖｌｐｆ／Ｇ１ｖｌｐｆ）
＝Ｇ（２ｉ＋１、２ｊ＋１）×（Ｒ（２ｉ、２ｊ＋１）＋Ｒ（２ｉ＋２、２ｊ＋１））
／（ｇ（２ｉ、２ｊ＋１）＋ｇ（２ｉ＋２、２ｊ＋１）） （１８）
【００４１】
同様に、Ｂ信号については、画素位置（２ｉ、２ｊ）においては、垂直方向に隣接する２ｊ列の画素が撮像素子１より得られており、また、Ｇ信号はＧ成分復元手段６により全画素復元されている（図５）。よって、画素位置（２ｉ、２ｊ）において、Ｇ、Ｂ信号に対して垂直方向のローパスフィルタを介した値Ｇ１ｖｌｐｆ、Ｂ１ｖｌｐｆを例えば、

により算出し、このＧ１ｖｌｐｆとＢ１ｖｌｐｆの比と画素Ｇ（２ｉ、２ｊ）により、画素位置（２ｉ、２ｊ）でのＢ信号ｂ（２ｉ、２ｊ）を次式により算出する。
ｂ（２ｉ、２ｊ）＝ Ｇ（２ｉ、２ｊ）×（Ｂ１ｖｌｐｆ／Ｇ１ｖｌｐｆ）
＝Ｇ（２ｉ、２ｊ）×（ Ｂ（２ｉ−１、２ｊ）＋Ｂ（２ｉ＋１、２ｊ））
／（ｇ（２ｉ−１、２ｊ）＋ｇ（２ｉ＋１、２ｊ）） （２１）
【００４２】
また、画素位置（２ｉ＋１、２ｊ＋１）においては、水平方向に隣接するの２ｉ＋１ラインの画素が撮像素子１より得られており、また、Ｇ信号はＧ成分復元手段６により全画素復元されている。よって、画素位置（２ｉ＋１、２ｊ＋１）において、Ｇ、Ｂ信号に対して水平方向のローパスフィルタを介した値Ｇ１ｈｌｐｆ、Ｂ１ｈｌｐｆを例えば、

により算出し、このＧ１ｈｌｐｆとＢ１ｈｌｐｆの比と画素Ｇ（２ｉ＋１、２ｊ＋１）により、画素位置（２ｉ＋１、２ｊ＋１）でのＢ信号ｂ（２ｉ＋１、２ｊ＋１）を次式により算出する。
ｂ（２ｉ＋１、２ｊ＋１）＝Ｇ（２ｉ＋１、２ｊ＋１）×（Ｂ１ｈｌｐｆ／Ｇ１ｈｌｐｆ）
＝Ｇ（２ｉ＋１、２ｊ＋１）×（ Ｂ（２ｉ＋１、２ｊ）＋Ｂ（２ｉ＋１、２ｊ＋２））
／（ｇ（２ｉ＋１、２ｊ）＋ｇ（２ｉ＋１、２ｊ＋２）） （２４）
【００４３】
なお、上記式（１５）、（１８）、（２１）、（２４）は上記Ｇでの復元方法と同様局所的領域での色信号の変化が少ないことを前提としており、つまり、各信号の比は局所的な領域ではほぼ等しいということによる。また、式（１３）〜（２４）におけるＧ１ｈｌｐｆ、Ｇ１ｖｌｐｆ、Ｒ１ｈｌｐｆ、Ｒ１ｖｌｐｆ、Ｂ１ｈｌｐｆ、Ｂ１ｖｌｐｆの算出式は、水平および垂直方向のローパスフィルタ出力の算出例であり、フィルタのタップ数および係数は上記に限るものではなく、他のタップ数および係数であってもよい。そして、第１のＲＢ成分復元手段７からは図７に示すＲ、ｒおよびＢ、ｂの画素信号Ｒ１およびＢ１が出力されて、第２のＲＢ成分復元手段９へと送られる。
【００４４】
次に、第２のＲＢ成分復元手段９および第２のエッジ判定手段８の動作について図４により説明する。第２のエッジ判定手段８において、Ｇ成分復元手段６から出力されたＧ信号Ｇ０は水平方向エッジ検出手段３０および垂直方向エッジ判定手段３１へと入力され、画素位置（２ｉ、２ｊ＋１）および（２ｉ＋１、２ｊ）の画素での左右および上下の画素の差分つまりエッジ成分を検出する。水平方向エッジ検出手段３０では、上記画素位置での左右の画素の差の絶対値ΔＨを得て、これを判定手段３２へと出力し、垂直方向エッジ検出手段３１では、上下の画素の差の絶対値ΔＶを得て、判定手段３２へと出力する。例えば、上記画素位置（２ｉ、２ｊ＋１）においては、水平方向エッジ検出手段３０で、
ΔＨ＝｜Ｇ（２ｉ、２ｊ）−Ｇ（２ｉ、２ｊ＋２）｜
を算出し、垂直方向エッジ検出手段３１で、
ΔＶ＝｜ Ｇ（２ｉ−１、２ｊ＋１）−Ｇ（２ｉ＋１、２ｊ＋１）｜
を算出し、上記ΔＨ、ΔＶを判定手段３２へと送る。以下、この画素の差の絶対値をエッジ成分と呼ぶ。
【００４５】
上記判定手段３２では、水平方向でのエッジ成分ΔＨおよび垂直方向エッジ成分ΔＶにより、水平垂直方向での周辺画素での信号レベルの変化を判定し、その判定結果を示す信号を出力するのであるが、ここで、上記第１のＲＢ成分復元手段７により、Ｒ信号Ｒ１では画素位置（２ｉ＋１、２ｊ）以外の画素信号が得られており（図７（ａ））、Ｂ信号Ｂ１では画素位置（２ｉ、２ｊ＋１）以外の画素信号が得られている（図７（ｂ））。したがって、画素位置（２ｉ＋１、２ｊ）でのエッジ成分を判定した結果ｅｄｒをＲ信号の画素位置（２ｉ＋１，２ｊ）の復元に対して出力し、一方、画素位置（２ｉ、２ｊ＋１）でのエッジ成分を判定した結果ｅｄｂをＢ信号の画素位置（２ｉ、２ｊ＋１）の復元に対して出力する。すなわち、画素位置（２ｉ＋１、２ｊ）における判定結果ｅｄｒと画素位置（２ｉ、２ｊ＋１）における判定結果ｅｄｂにおいて、ΔＨとΔＶの両方が予め定めた値ｔｈ２以下の場合は、周辺画素での信号レベルの変化がないと判定し、例えばｅｄｒ＝１およびｅｄｂ＝１として出力する。一方、ΔＨまたはΔＶが予め定めた値ｔｈ２より大きい場合は、その画素においてエッジ成分があると判定し、さらに、ΔＨ＞ΔＶの場合は垂直方向に相関が高いと判定して、例えばｅｄｒ＝２およびｅｄｂ＝２を出力し、ΔＨ≦ΔＶの場合は水平方向に相関が高いと判定して例えばｅｄｒ＝３およびｅｄｂ＝３を出力する。なお、Ｒ、Ｂが得られている画素位置においてはエッジ成分を判定する必要はなく、判定手段３２からは例えばｅｄｒ＝０およびｅｄｂ＝０を出力するものとする。そして、判定手段３２の出力ｅｄｒは第２のＲＢ成分復元手段９におけるＲ切り換え手段３６へ、また、出力ｅｄｂは第２のＲＢ成分復元手段９におけるＢ切り換え手段４０へと送られる。
【００４６】
次に、第２のＲＢ成分復元手段９において、第１のＲＢ成分復元手段７からのＲ信号出力Ｒ１はＲ平均値算出手段３３、Ｒ水平方向演算手段３４およびＲ垂直方向演算手段３５、Ｒ切り換え手段３６へと入力される。図８（ａ）は第２のＲＢ成分復元手段９において画素位置（２ｉ＋１、２ｊ）のＲ信号の復元を説明するための図であり、図中Ｒおよびｒで示された画素は撮像素子で得られた信号と上記第１のＲＢ復元手段７において復元された画素信号であり、よって、斜線で示されたｒ′画素の信号を求めることとなる。まず、Ｒ平均値算出手段３３では、Ｒ信号Ｒ１において、画素位置（２ｉ＋１、２ｊ）での斜め方向に隣接する４画素Ｒ（２ｉ、２ｊ−１）、Ｒ（２ｉ、２ｊ＋１）、Ｒ（２ｉ＋２、２ｊ−１）、Ｒ（２ｉ＋２、２ｊ＋１）の平均値ｒａを次式により算出する。
ｒａ（２ｉ＋１、２ｊ）＝｛ Ｒ（２ｉ、２ｊ−１）＋Ｒ（２ｉ、２ｊ＋１）
＋ Ｒ（２ｉ＋２、２ｊ−１）＋Ｒ（２ｉ＋２、２ｊ＋１）｝／４
【００４７】
Ｒ水平方向演算手段３４ではＧ成分復元手段６からのＧ信号Ｇ０（図５）も入力されており、ＲおよびＧ信号の画素位置（２ｉ＋１，２ｊ）において、水平方向のローパスフィルタを介した値Ｒ２ｈｌｐｆ、Ｇ２ｈｌｐｆを例えば、

により算出し、このＲ２ｈｌｐｆとＧ２ｈｌｐｆの比と画素ｇ（２ｉ＋１、２ｊ）により、次式の演算で（２ｉ＋１，２ｊ）の水平方向に相関を持つ画素値ｒｈを算出し出力する。
ｒｈ（２ｉ＋１、２ｊ）＝ｇ（２ｉ＋１、２ｊ）×（Ｒ２ｈｌｐｆ／Ｇ２ｈｌｐｆ）
＝ｇ（２ｉ＋１、２ｊ）
×（ ｒ（２ｉ＋１、２ｊ−１）＋ｒ（２ｉ＋１、２ｊ＋１））
／（Ｇ（２ｉ＋１、２ｊ−１）＋Ｇ（２ｉ＋１、２ｊ＋１）） （２７）
【００４８】
Ｒ垂直方向演算手段３５ではＧ成分復元手段６からのＧ信号Ｇ０も入力されており、ＲおよびＧ信号における画素位置（２ｉ＋１，２ｊ）において、垂直方向のローパスフィルタを介した値Ｒ２ｖｌｐｆ、Ｇ２ｖｌｐｆを例えば、
Ｒ２ｖｌｐｆ＝（ｒ（２ｉ、２ｊ）＋ｒ（２ｉ＋２、２ｊ））／２ （２８）
Ｇ２ｖｌｐｆ＝（Ｇ（２ｉ、２ｊ）＋Ｇ（２ｉ＋２、２ｊ））／２ （２９）
により算出し、このＲ２ｖｌｐｆとＧ２ｖｌｐｆの比と画素ｇ（２ｉ＋１、２ｊ）により、次式の演算で（２ｉ＋１，２ｊ）の垂直方向に相関を持つ画素値ｒｖを算出し出力する。
ｒｖ（２ｉ＋１、２ｊ）＝ｇ（２ｉ＋１、２ｊ）×（Ｒ２ｖｌｐｆ／Ｇ２ｖｌｐｆ）
＝ｇ（２ｉ＋１、２ｊ）×（ｒ（２ｉ、２ｊ）＋ｒ（２ｉ＋２、２ｊ））
／（Ｇ（２ｉ、２ｊ）＋Ｇ（２ｉ＋２、２ｊ）） （３０）
【００４９】
なお、上記式（２７）、（３０）は上述したように、局所的領域での色信号の変化が少ないことを前提としており、つまり、各信号の比は局所的な領域ではほぼ等しいということにより、また、式（２５）〜（３０）におけるＧ２ｈｌｐｆ、Ｇ２ｖｌｐｆ、Ｒ２ｈｌｐｆ、Ｒ２ｖｌｐｆの算出式は、水平および垂直方向のローパスフィルタ出力の算出例であり、フィルタのタップ数および係数は上記に限るものではなく、他のタップ数および係数であってもよい。そして、上記出力ｒｈおよびｒｖはともにＲ切り換え手段３６へと送られる。
【００５０】
Ｒ切り換え手段３６では、第２のエッジ判定手段８における判定手段３２からの周辺画素の信号レベル変化を判定した結果ｅｄｒに基づき、画素位置（２ｉ＋１、２ｊ）の信号を上記Ｒ平均値算出手段３３、Ｒ水平方向演算手段３４およびＲ垂直方向演算手段３５、第１のＲＢ成分復元手段７からの信号より選択し切り換える。つまり、判定手段３２による出力信号ｅｄｒが‘２’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ＞ΔＶで垂直方向に相関が高いと判定される場合は、垂直方向に相関を持つＲを出力するＲ垂直方向演算手段３５の信号ｒｖを選択する。判定手段３２による出力信号ｅｄｒが‘３’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ≦ΔＶで水平方向に相関が高いと判定される場合には、水平方向に相関を持つＲを出力するＲ水平方向演算手段３４の信号を選択する。さらに、判定手段３２による出力信号ｅｄｒが‘１’を示す場合には、ΔＨとΔＶの両方が予め定めた値ｔｈ２以下であり、周辺画素での信号レベルの変化がないと判定されるため、周波数の変化を考慮する必要がなく、Ｒ平均値算出手段３３によるＲ信号ｒａを選択する。なお、Ｒが得られている画素位置（２ｉ、２ｊ）、（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ＋１）においては、判定手段３２からは例えばｅｄｒ＝０が出力されており、このときは第１のＲＢ成分復元手段７からのＲ信号をそのまま出力すればよい。
【００５１】
Ｂ信号についても上記Ｒ信号での処理と同様であり、第２のＲＢ成分復元手段９において、第１のＲＢ成分復元手段７からのＢ信号出力Ｂ１はＢ平均値算出手段３７、Ｂ水平方向演算手段３８およびＢ垂直方向演算手段３９、Ｂ切り換え手段４０へと入力される。図８（ｂ）は第２のＲＢ成分復元手段９において画素位置（２ｉ、２ｊ＋１）のＢ信号の復元を説明するための図であり、図中Ｂおよびｂで示された画素は撮像素子で得られた信号と上記第１のＲＢ復元手段７において復元された画素信号であり、よって、斜線で示されたｂ′画素の信号を求めることとなる。まず、Ｂ平均値算出手段３７では、Ｂ信号Ｂ１において、画素位置（２ｉ、２ｊ＋１）での斜め方向に隣接する４画素の平均値ｂａを次式により算出する。
ｂａ（２ｉ、２ｊ＋１）＝｛Ｂ（２ｉ−１、２ｊ）＋Ｂ（２ｉ−１、２ｊ＋２）
＋Ｂ（２ｉ＋１、２ｊ）＋Ｂ（２ｉ＋１、２ｊ＋２）｝／４
【００５２】
Ｂ水平方向演算手段３８ではＧ成分復元手段６からのＧ信号Ｇ０（図５）も入力されており、ＢおよびＧ信号における画素位置（２ｉ，２ｊ＋１）において、水平方向のローパスフィルタを介した値Ｂ２ｈｌｐｆ、Ｇ３ｈｌｐｆを例えば、Ｂ２ｈｌｐｆ＝（ｂ（２ｉ、２ｊ）＋ｂ（２ｉ、２ｊ＋２））／２ （３１）
Ｇ３ｈｌｐｆ＝（Ｇ（２ｉ、２ｊ）＋Ｇ（２ｉ、２ｊ＋２））／２ （３２）
により算出し、このＢ２ｈｌｐｆとＧ３ｈｌｐｆの比と画素ｇ（２ｉ、２ｊ＋１）により、次式の演算で（２ｉ，２ｊ＋１）の水平方向に相関を持つ画素値ｂｈを算出し出力する。
ｂｈ（２ｉ、２ｊ＋１）＝ｇ（２ｉ、２ｊ＋１）×（Ｂ２ｈｌｐｆ／Ｇ３ｈｌｐｆ）
＝ｇ（２ｉ、２ｊ＋１）×（ｂ（２ｉ、２ｊ）＋ｂ（２ｉ、２ｊ＋２））
／（Ｇ（２ｉ、２ｊ）＋Ｇ（２ｉ、２ｊ＋２）） （３３）
【００５３】
Ｂ垂直方向演算手段３９ではＧ成分復元手段６からのＧ信号Ｇ０も入力されており、ＢおよびＧ信号における画素位置（２ｉ，２ｊ＋１）において、垂直方向のローパスフィルタを介した値Ｂ２ｖｌｐｆ、Ｇ３ｖｌｐｆを例えば、

により算出し、このＢ２ｖｌｐｆとＧ３ｖｌｐｆの比と画素ｇ（２ｉ、２ｊ＋１）により、次式の演算で（２ｉ，２ｊ＋１）の垂直方向に相関を持つ画素値ｂｖを算出し出力する。
ｂｖ（２ｉ、２ｊ＋１）＝ｇ（２ｉ、２ｊ＋１）×（Ｂ２ｖｌｐｆ／Ｇ３ｖｌｐｆ）
＝ｇ（２ｉ、２ｊ＋１）×（ｂ（２ｉ−１、２ｊ＋１）＋ｂ（２ｉ＋１、２ｊ＋１））
／（Ｇ（２ｉ−１、２ｊ＋１）＋Ｇ（２ｉ＋１、２ｊ＋１））（３６）
【００５４】
なお、上記式（３３）、（３６）は上述したように、局所的領域での色信号の変化が少ないことを前提としており、つまり、各信号の比は局所的な領域ではほぼ等しいということにより、また、式（３１）〜（３６）におけるＧ３ｈｌｐｆ、Ｇ３ｖｌｐｆ、Ｂ２ｈｌｐｆ、Ｂ２ｖｌｐｆの算出式は、水平および垂直方向のローパスフィルタ出力の算出例であり、フィルタのタップ数および係数は上記に限るものではなく、他のタップ数および係数であってもよい。そして、上記出力ｂｈおよびｂｖはともにＢ切り換え手段４０へと送られる。
【００５５】
Ｂ切り換え手段４０では、第２のエッジ判定手段８における判定手段３２からの周辺画素の信号レベル変化を判定した結果ｅｄｂに基づき、画素位置（２ｉ、２ｊ＋１）の信号を上記Ｂ平均値算出手段３７、Ｂ水平方向演算手段３８およびＢ垂直方向演算手段３９、第１のＲＢ成分復元手段７からの信号より選択し切り換える。つまり、判定手段３２による出力信号ｅｄｂが‘２’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ＞ΔＶで垂直方向に相関が高いと判定される場合は、垂直方向に相関を持つＲを出力するＲ垂直方向演算手段３９の信号ｂｖを選択する。判定手段３２による出力信号ｅｄｂが‘３’を示す場合、すなわち、その画素においてエッジ成分があると判定されるが、ΔＨ≦ΔＶで水平方向に相関が高いと判定される場合には、水平方向に相関を持つＢを出力するＢ水平方向演算手段３８の信号を選択する。さらに、判定手段３２による出力信号ｅｄｂが‘１’を示す場合には、ΔＨとΔＶの両方が予め定めた値ｔｈ２以下であり、周辺画素での信号レベルの変化がないと判定されるため、周波数の変化を考慮する必要がなく、Ｂ平均値算出手段３７によるＢ信号ｂａを選択する。なお、Ｂが得られている画素位置（２ｉ、２ｊ）、（２ｉ＋１、２ｊ）、（２ｉ＋１、２ｊ＋１）においては、判定手段３２からは例えばｅｄｂ＝０が出力されており、このときは第１のＲＢ成分復元手段７からのＢ信号をそのまま出力すればよい。
【００５６】
以上より、第２のＲＢ成分復元手段９からは画素位置（２ｉ、２ｊ）、（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）、（２ｉ＋１、２ｊ＋１）それぞれの画素でのＲおよびＢ信号が出力され、つまり、撮像素子の画素数分の解像度のＲ、Ｂ信号を得られるとになる。Ｇ信号は上記Ｇ成分復元手段６においてすでに撮像素子の画素数分の解像度の信号を得ており、したがって、Ｇ成分を用い水平垂直方向のエッジ成分を判定して局所的な領域での空間周波数の変化を判定し、その判定結果に基づき各色信号の生成を切り換えるともに、エッジ成分が所定値を越える場合の信号の算出において、局所的な領域での色信号の比により算出を行い、エッジ成分が所定値以下では周辺画素の平均値で求めるので、そのため、偽色、偽輪郭が軽減された画像を得ることになる。
【００５７】
図９は、ゾーンプレートを画像シミュレーションにより処理した場合の水平方向において発生する偽色レベル（color-alias level 、縦軸）と解像度（横軸）との関係を示すものであり、上記図２０による従来技術による処理後の偽色レベルを波線で示し、図２による実施の形態１による処理後の偽色レベルを実線で示している。従来の技術による処理では多くの偽色が発生しているが、実施の形態１による処理では偽色の発生が抑圧されている。各処理でこの水平方向の偽色信号のレベルを積分して比較すると、約１８．４ｄＢの抑圧効果がある。
【００５８】
また、図１０は、垂直方向での発生する偽色レベル（color-alias level 、縦軸）と解像度（横軸）との関係を示すものであり、上記と同様に、上記図２０による従来技術による処理後の偽色レベルを波線で示し、図２による実施の形態１による処理後の偽色レベルを実線で示している。従来の技術による処理では多くの偽色が発生しているが、実施の形態１による処理では偽色の発生が抑圧されており、垂直方向の偽色信号のレベルを積分して比較すると、約１８．５ｄＢの抑圧効果がある。
【００５９】
さらに、図１１および図１２は、ゾーンプレートを画像シミュレーションにより処理した場合の水平方向における輝度信号のレベルをＲＧＢ信号より算出して縦軸に、解像度を横軸に示した図であり、図２による実施の形態１による処理後の場合を図１１に、上記図２０による従来技術による処理後の場合を図１２に示す。また、垂直方向における輝度信号のレベルと解像度との関係を、図１３に図２による実施の形態１による処理後の場合を、図１４に上記図２０による従来技術による処理後の場合示す。水平および垂直方向どちらに対しても、従来の技術による処理に比較して実施の形態１による処理ではレベルの低下が少なく、水平垂直方向の解像度が向上している。
【００６０】
なお、上記実施の形態１では、撮像素子１の色フィルタの配列を図１に示す原色の色フィルタであり、各光電変換素子を独立に呼び出す方式の撮像素子とし、画素位置（２ｉ，２ｊ）と画素位置（２ｉ＋１、２ｊ＋１）（ｉ＝０、１、２、…、ｊ＝０、１、２、…）にＧ信号を通過させる分光特性を持った第１の色フィルタを、画素位置（２ｉ、２ｊ＋１）にＲ信号を通過させる分光特性を持った第２の色フィルタを、画素位置（２ｉ＋１、２ｊ）にＢ信号を通過させる分光特性を持った第３の色フィルタを配列した場合として説明したが、撮像素子におけるは第１、第２および第３の色フィルタの分光特性はＲ、Ｇ、Ｂに限るものではなく、例えば画素混合方式の撮像素子であってもよく、図１５に示すように、画素位置（２ｉ，２ｊ）と画素位置（２ｉ＋１、２ｊ＋１）（ｉ＝０、１、２、…、ｊ＝０、１、２、…）に第１の信号Ａを通過させる分光特性を持った第１の色フィルタを、画素位置（２ｉ、２ｊ＋１）に第２の信号Ｂを通過させる分光特性を持った第２の色フィルタを、画素位置（２ｉ＋１、２ｊ）に第３の信号Ｃを通過させる分光特性を持った第３の色フィルタを配列し、上記実施の形態１の処理により撮像素子の画素数分の各Ａ、Ｂ、Ｃの信号を復元した後ＲＧＢの色信号を再生できればよく、上記と同様の効果を奏する。
【００６１】
また、上記実施の形態１では、図１および図１５の撮像素子の色フィルタの配列を画素位置（２ｉ，２ｊ）と画素位置（２ｉ＋１、２ｊ＋１）（ｉ＝０、１、２、…、ｊ＝０、１、２、…）に第１の色フィルタを、画素位置（２ｉ、２ｊ＋１）に第２の色フィルタを、画素位置（２ｉ＋１、２ｊ）に第３の色フィルタを配列した場合（図１、１５中の斜線部分）として説明したが、図１６に示されるように、画素位置（２ｉ，２ｊ＋１）と画素位置（２ｉ＋１、２ｊ）（ｉ＝０、１、２、…、ｊ＝０、１、２、…）に第１の色フィルタを、画素位置（２ｉ、２ｊ）に第２の色フィルタを、画素位置（２ｉ＋１、２ｊ＋１）に第３の色フィルタを配列した場合（図１６中の斜線部分）でも同様の効果を奏し、上下４画素毎にｎラインには第１の色フィルタ、第２の色フィルタが配列され、ｎ＋１ラインでは第３の色フィルタと第１の色フィルタが配列されて、上下のラインで第１のフィルタが斜め方向の画素に配列されていればよい。
【００６２】
また、上記実施の形態１において、第１のエッジ判定手段５および第２のエッジ判定手段８においての判定結果を示す信号を、ΔＨとΔＶの両方が予め定めた値以下の場合は‘１’とし、エッジ成分があると判定しΔＨ＞ΔＶの場合は‘２’とし、ΔＨ≦ΔＶの場合は‘３’とし、画素信号が得られている画素位置においては‘０’として出力する場合について説明しているが、これに限るものではなく、それぞれの判定結果を区別できる信号を出力するのであれば、他の値であってもよい。
【００６３】
なお、上記実施の形態１では、ハードウェアにより図２の構成の処理を行う場合について説明しているが、撮像装置におけるソフトウェアにより同様の処理を行うことができることは言うまでもなく、上記実施の形態１と同様の効果を奏する。
【００６４】
実施の形態２．
実施の形態１では第１のエッジ判定手段５において、Ｇ信号における画素位置（２ｉ、２ｊ＋１）および（２ｉ＋１、２ｊ）での上下左右でのエッジ成分を判定し、第２のエッジ判定手段８においては、Ｒ信号の復元に対してＧ信号における画素位置（２ｉ＋１、２ｊ）でのエッジ成分を、Ｂ信号の復元に対しＧ信号における画素位置（２ｉ、２ｊ＋１）でのエッジ成分を判定するよう構成したが、どちらも画素位置（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）での左右および上下の画素のエッジ成分を検出しており、これらの画素位置での上下左右の画素信号は撮像素子より得られた信号であるため、同一のエッジ成分を求めることとなる。よって、図１７に示すように、１つのＧ成分エッジ判定手段によりＧ信号における画素位置（２ｉ、２ｊ＋１）および（２ｉ＋１、２ｊ）でのエッジ成分を判定するような構成とすることもできる。
【００６５】
図１７において、１〜４、６〜７、および９は上記実施の形態１での撮像装置と同一のものであり、４１はＧ成分エッジ判定手段であり、水平方向エッジ検出手段４２、垂直方向エッジ検出手段４３、判定手段４４により構成される。
【００６６】
次に動作を説明する。撮像素子１から各画素信号Ｒ、Ｇ、Ｂを読み出し、Ａ／Ｄコンバータ２によりＡ／Ｄ変換されてフレームメモリ３に入力され、分離手段４により各Ｒ、Ｇ、Ｂ信号を分離し、Ｇ成分復元手段６、第１のＲＢ復元手段７および第２のＲＢ復元手段９で撮像素子の画素数の各信号を生成し復元する動作は上記実施の形態１と同一であるのでその詳細な説明は省略する。
【００６７】
Ｇ成分エッジ判定手段４１には分離手段４におけるＧ信号（図１８）が入力され、Ｇ成分エッジ判定手段４１内の水平方向エッジ検出手段４２、垂直方向エッジ検出手段４３へと与えられる。水平方向エッジ検出手段４２および垂直方向エッジ検出手段４３においては画素位置（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）（図１８中斜線で示す画素）での左右および上下の画素の差分つまりエッジ成分を検出する。つまり、水平方向エッジ検出手段４２では、上記画素位置での左右の画素の差の絶対値ΔＨｇを得て、これを判定手段４４へと出力し、垂直方向エッジ検出手段４３では、上下の画素の差の絶対値ΔＶｇを得て、判定手段５５へと出力する。例えば、画素位置（２ｉ、２ｊ＋１）においては、水平方向エッジ検出手段４２で、
ΔＨｇ＝｜Ｇ（２ｉ、２ｊ）−Ｇ（２ｉ、２ｊ＋２）｜
を算出し、垂直方向エッジ検出手段４３で
ΔＶｇ＝｜ Ｇ（２ｉ−１、２ｊ＋１）−Ｇ（２ｉ＋１、２ｊ＋１）｜
を算出し、上記ΔＨｇ、ΔＶｇを判定手段４４へと送る。
【００６８】
上記判定手段４４では、水平方向でのエッジ成分ΔＨｇおよび垂直方向エッジ成分ΔＶｇにより、水平垂直方向での周辺画素での信号レベルの変化を判定し、入力されるＧ信号の各画素位置に応じて、その判定結果を示す信号ｅｄ１をＧ成分復元手段６へ、ｅｄｒ、ｅｄｂを第２のＲＢ成分復元手段９へと出力する。各判定信号は、ΔＨｇとΔＶｇの両方が予め定めた値以下の場合は、周辺画素での信号レベルの変化がないと判定し、例えば‘１’とする。一方、ΔＨｇまたはΔＶｇが予め定めた値より大きい場合は、その画素においてエッジ成分があると判定し、さらに、ΔＨｇ＞ΔＶｇの場合は垂直方向に相関が高いと判定して例えば‘２’とし、ΔＨｇ≦ΔＶｇの場合は水平方向に相関が高いと判定して例えば、‘３’とする。ここで、Ｇ成分復元手段６においては、画素位置（２ｉ、２ｊ＋１）と（２ｉ＋１、２ｊ）での画素を復元するため、両画素位置において上記エッジ判定結果をｅｄ１として出力し、Ｇが得られている画素位置（２ｉ、２ｊ）、（２ｉ＋１、２ｊ＋１）においては例えばｅｄ１＝０を出力する。そして、判定手段４４の出力ｅｄ１はＧ成分復元手段６における切り換え手段２５へと送られる。
【００６９】
一方、第２のＲＢ成分復元手段９においては、Ｒ信号の画素位置（２ｉ＋１，２ｊ）の復元を行い、Ｂ信号の画素位置（２ｉ、２ｊ＋１）の復元を行う。よって、画素位置（２ｉ＋１、２ｊ）において上記エッジ成分を判定した結果をｅｄｒとして、第２のＲＢ成分復元手段９での処理する画素位置にタイミングを合わせて出力し、他の画素位置（２ｉ、２ｊ）、（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ＋１）においては例えばｅｄｒ＝０を出力する。そして、判定手段４４の出力ｅｄｒは第２のＲＢ成分復元手段９におけるＲ切り換え手段３６へと送られる。一方、画素位置（２ｉ、２ｊ＋１）において上記エッジ成分を判定した結果をｅｄｂとして、第２のＲＢ成分復元手段９での処理する画素位置にタイミングを合わせて出力し、他の画素位置（２ｉ、２ｊ）、（２ｉ＋１、２ｊ）、（２ｉ＋１、２ｊ＋１）においては例えばｅｄｂ＝０を出力する。そして、判定手段４４の出力ｅｄｂは第２のＲＢ成分復元手段９におけるＢ切り換え手段４０へと送られる。
【００７０】
よって、Ｇ成分エッジ判定手段４１において、上記Ｇ成分復元手段６および第２のＲＢ成分復元手段９におけるＧ、Ｒ、Ｂ信号を復元する画素位置に対応するＧ信号でのエッジ成分の判定結果を出力されることとなる。
【００７１】
なお、上記実施の形態２では、画素位置（２ｉ、２ｊ）、（２ｉ＋１、２ｊ＋１）にＧ信号が、画素位置（２ｉ、２ｊ＋１）にＲ信号が、画素位置（２ｉ＋１、２ｊ＋１）にＢ信号が撮像素子より得られる場合について説明したが、上記実施の形態１同様、画素位置（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）にＧ信号が、画素位置（２ｉ、２ｊ）にＲ信号が、画素位置（２ｉ＋１、２ｊ＋１）にＢ信号が配列されていてもよく、また、色信号はＲＧＢに限ることはない。
【００７２】
また、上記実施の形態１と同様、実施の形態２においても、図１７の構成の処理を撮像装置におけるソフトウェアにより同様の処理を行うことができることは言うまでもなく、上記実施の形態２と同様の効果を奏する。
【００７３】
【発明の効果】
この発明は、以上説明したように構成されているので、以下に示すような効果を奏する。
【００７４】
この発明に係る撮像装置によれば、画素位置（２ｉ、２ｊ）（ｉ＝０、１、２、…およびｊ＝０、１、２、…）と（２ｉ＋１、２ｊ＋１）に第１の色信号を通過させる分光特性を持った第１の色フィルタを、画素位置（２ｉ、２ｊ＋１）に第２の色信号を通過させる分光特性を持った第２の色フィルタ、画素位置（２ｉ＋１、２ｊ）に第３の色信号を通過させる分光特性を持った第３の色フィルタを配列した上下４画素が垂直水平方向に繰り返し配列された撮像素子において、第１の色フィルタの位置における第１の色信号の画素位置（２ｉ、２ｊ＋１）、（２ｉ＋１、２ｊ）において、周辺画素信号に基づきエッジ成分を判定し、この判定結果に基づき、第１、第２および第３の色信号により第１の色信号の上記画素位置での信号を算出して撮像素子の画素数の第１の色信号を復元し、上記エッジ判定結果に基づき、復元された第１の色信号と各色フィルタからの色信号により第２および第３の色信号を算出し撮像素子の画素数の第２、第３の色信号を得ることにより、偽色、偽輪郭の軽減された画像を得ることができる。
【００７５】
また、この発明に係る撮像装置によれば、エッジ判定手段が、第１の色信号の所定画素における左右の隣接画素の差の絶対値を算出して水平方向のエッジ成分ΔＨを検出し、上下の画素の差の絶対値を算出して垂直方向のエッジ成分ΔＶを検出して、水平方向のエッジ成分ΔＨと垂直方向のエッジ成分ΔＶに基づき、前記所定画素における水平または垂直方向のエッジ成分を判定する判定することにより、局所的な領域での水平、垂直方向の空間周波数の変化を判定することができ、偽色、偽輪郭の軽減された画像を得ることができる。
【００７６】
また、この発明に係る撮像装置によれば、前記エッジ判定手段における判定手段が、上記水平方向エッジ検出手段からの出力ΔＨまたは上記垂直方向エッジ検出手段からの出力ΔＶが予め定めた値より大きい場合は、所定画素の周辺画素にエッジ成分を検出したとし、さらに、ΔＨ＞ΔＶであれば垂直方向により相関があり、ΔＨ≦ΔＶであれば水平方向により相関があると判定するとともに、上記ΔＨとΔＶがともに予め定めた値より小さい場合にはエッジ成分を検出しないと判定することにより、局所的な領域での水平、垂直方向の空間周波数の変化を判定することができ、偽色、偽輪郭の軽減された画像を得ることができる。
【００７７】
また、この発明に係る撮像装置によれば、第１の色信号を算出して復元する手段が、第２の色信号Ｂのある所定画素ｌ行ｍ列Ｂ（ｌ、ｍ）の位置（ｌ＝２ｉ、ｍ＝２ｊ＋１またはｌ＝２ｉ＋１、ｍ＝２ｊ）において、第１の色信号Ａ、第２の色信号Ｂのそれぞれに対して水平方向のローパスフィルタを介した値Ａｈｌｐｆ（ｌ、ｍ）、Ｂｈｌｐｆ（ｌ、ｍ）を算出し、前記水平方向ローパスフィルタからの出力信号であるＡｈｌｐｆ（ｌ、ｍ）とＢｈｌｐｆ（ｌ、ｍ）との比と上記画素位置の画素値Ｂ（ｌ、ｍ）により、ｌ行ｍ列の第１の色信号Ａにおける画素値Ａ（ｌ、ｍ）を、Ａ（ｌ、ｍ）＝Ｂ（ｌ、ｍ）×｛Ａｈｌｐｆ（ｌ、ｍ）／Ｂｈｌｐｆ（ｌ、ｍ）｝により算出し、第３の色信号Ｃのある他の画素位置においても同様に第１の色信号Ａにおける画素値を算出する水平方向信号算出手段と、上記所定画素ｌ行ｍ列Ｂ（ｌ、ｍ）の位置において、第１の色信号Ａ、第２の色信号Ｂのそれぞれに対して垂直方向のローパスフィルタを介した値Ａｖｌｐｆ（ｌ、ｍ）、Ｂｖｌｐｆ（ｌ、ｍ）を算出し、前記垂直方向ローパスフィルタからの出力信号であるＡｖｌｐｆ（ｌ、ｍ）とＢｖｌｐｆ（ｌ、ｍ）との比と上記画素位置の画素値Ｂ（ｌ、ｍ）により、ｌ行ｍ列の第１の色信号Ａである画素値Ａ（ｌ、ｍ）を、Ａ（ｌ、ｍ）＝Ｂ（ｌ、ｍ）×｛Ａｖｌｐｆ（ｌ、ｍ）／Ｂｖｌｐｆ（ｌ、ｍ）｝により算出し、第３の色信号Ｃのある他の画素位置においても同様に第１の色信号Ａにおける画素値を算出する垂直方向信号算出手段と、第１の色信号Ａにおける上記所定画素ｌ行ｍ列の位置での上下左右の隣接画素の平均値よりｌ行ｍ列の第１の色信号Ａにおける画素値Ａ（ｌ、ｍ）を算出する平均値算出手段とを備え、上記エッジ判定手段の出力に基づき、上記水平方向信号算出手段の出力または垂直方向信号算出手段出力、または、平均値算出手段からの出力より選択し、上記所定画素ｌ行ｍ列における第１の色信号Ａの画素値Ａ（ｌ、ｍ）を得て、撮像素子における画素数の第１の色信号を得ることにより、偽色、偽輪郭の軽減された画像を得ることができる。
【００７８】
また、この発明に係る撮像装置によれば、前記第１の色信号を算出する手段が、上記エッジ判定手段の出力が所定画素ｌ行ｍ列位置においてエッジ成分を検出しないと判定した場合は上記平均値算出手段の出力を選択し、垂直方向に相関があると判定した場合は上記垂直方向信号算出手段の出力を選択し、水平方向に相関があると判定した場合は上記水平方向信号算出手段の出力を選択し、撮像素子における画素数の第１の色信号を得ることにより、偽色、偽輪郭の軽減された画像を得ることができる。
【００７９】
また、この発明に係る撮像装置によれば、第２、第３の色信号を算出して復元する手段が、所定画素ｌ行ｍ列の位置（ｌ＝２ｉ、ｍ＝２ｊまたはｌ＝２ｉ＋１、ｍ＝２ｊ＋１）において、上記第１の色信号を算出する手段からの出力Ａに対し水平方向のローパスフィルタを介した値Ａ１ｈｌｐｆ（ｌ、ｍ）と垂直方向ローパスフィルタを介した値Ａ１ｖｌｐｆ（ｌ、ｍ）を算出し、第２の色信号Ｂに対して水平方向のローパスフィルタを介した値Ｂ１ｈｌｐｆ（ｌ、ｍ）と第３の色信号Ｃに対して垂直方向のローパスフィルタを介した値Ｃ１ｖｌｐｆ（ｌ、ｍ）（または、第２の色信号Ｂに対して垂直方向のローパスフィルタを介した値Ｂ１ｖｌｐｆ（ｌ、ｍ）と第３の色信号Ｃに対して水平方向のローパスフィルタを介した値Ｃ１ｈｌｐｆ（ｌ、ｍ））とを算出し、Ａ１ｈｌｐｆ（ｌ、ｍ）とＢ１ｈｌｐｆ（ｌ、ｍ）との比（またはＣ１ｈｌｐｆ（ｌ、ｍ）との比）と、Ａ１ｖｌｐｆ（ｌ、ｍ）とＣ１ｖｌｐｆ（ｌ、ｍ）との比（または、Ｂ１ｖｌｐｆ（ｌ、ｍ）との比）と、第１の色信号を算出する手段からの出力Ａにおける所定画素ｌ行ｍ列での画素値Ａ（ｌ、ｍ）から、ｌ行ｍ列の第２の色信号Ｂと第３の色信号Ｃにおける画素値Ｂ（ｌ、ｍ）とＣ（ｌ、ｍ）を、Ｂ（ｌ、ｍ）＝Ａ（ｌ、ｍ）×｛Ｂ１ｈｌｐｆ（ｌ、ｍ）／Ａ１ｈｌｐｆ（ｌ、ｍ）｝、Ｃ（ｌ、ｍ）＝Ａ（ｌ、ｍ）×｛Ｃ１ｖｌｐｆ（ｌ、ｍ）／Ａ１ｖｌｐｆ（ｌ、ｍ）｝、（または、Ｂ（ｌ、ｍ）＝Ａ（ｌ、ｍ）×｛Ｂ１ｖｌｐｆ（ｌ、ｍ）／Ａ１ｖｌｐｆ（ｌ、ｍ）｝、Ｃ（ｌ、ｍ）＝Ａ（ｌ、ｍ）×｛Ｃ１ｈｌｐｆ （ｌ、ｍ）／Ａ１ｈｌｐｆ（ｌ、ｍ）｝）により算出する信号算出手段を備えるとともに、上記ｌ行ｍ列の位置とは異なる所定画素ｘ行ｙ列の位置（ｘ＝２ｉ＋１、ｙ＝２ｊまたはｘ＝２ｉ、ｙ＝２ｊ＋１）において、上記第１の色信号を算出する手段からの出力Ａに対し水平方向のローパスフィルタを介した値Ａ２ｈｌｐｆ（ｘ、ｙ）、前記信号算出手段からの出力での第２の色信号Ｂに対し水平方向のローパスフィルタを介した値Ｂ２ｈｌｐｆ（ｘ、ｙ）を算出し、Ａ２ｈｌｐｆ（ｘ、ｙ）とＢ２ｈｌｐｆ（ｘ、ｙ）の比と上記第１の算出手段からの出力Ａでの画素ｘ行ｙ列での画素値Ａ（ｘ、ｙ）により、ｘ行ｙ列の位置における第２の色信号Ｂ（ｘ、ｙ）を、Ｂ（ｘ、ｙ）＝Ａ（ｘ、ｙ）×｛Ｂ２ｈｌｐｆ（ｘ、ｙ）／Ａ２ｈｌｐｆ（ｘ、ｙ）｝により算出し、第３の色信号Ｃにおいても同様にＣ信号を算出する水平方向信号算出手段と、上記第１の色信号を算出する手段からの出力Ａに対し垂直方向のローパスフィルタを介した値Ａ２ｖｌｐｆ（ｘ、ｙ）、前記信号算出手段からの出力での第２の色信号Ｂに対して垂直方向のローパスフィルタを介した値Ｂ２ｖｌｐｆ（ｘ、ｙ）を算出し、Ａ２ｖｌｐｆ（ｘ、ｙ）とＢ２ｖｌｐｆ（ｘ、ｙ）の比と上記第１の算出手段からの出力Ａでの画素ｘ行ｙ列での画素値Ａ（ｘ、ｙ）により、ｘ行ｙ列の位置における第２の色信号Ｂ（ｘ、ｙ）を、Ｂ（ｘ、ｙ）＝Ａ（ｘ、ｙ）×｛Ｂ２ｖｌｐｆ（ｘ、ｙ）／Ａ２ｖｌｐｆ（ｘ、ｙ）｝により算出し、第３の色信号Ｃにおいても同様にＣ信号を算出する垂直方向信号算出手段と、上記信号算出手段からの出力における第２、第３の色信号での所定画素ｘ行ｙ列の位置において、斜めに隣接する画素の平均値を算出する平均値算出手段とを備え、上記エッジ判定手段の出力に基づき、前記水平方向信号算出手段、垂直方向信号算出手段、平均値算出手段からのそれぞれの出力から選択して、上記所定画素ｘ行ｙ列での第２、第３の色信号を得て、撮像素子における画素数の第２、第３の色信号を得ることにより、偽色、偽輪郭の軽減された画像を得ることができる。
【００８０】
さらに、この発明に係る撮像装置によれば、前記第２、第３の色信号を算出する手段が、上記エッジ判定手段の出力が所定画素ｘ行ｙ列の位置においてエッジ成分を検出しないと判定した場合は上記平均値算出手段の出力を選択し、垂直方向に相関があると判定した場合は上記垂直方向信号算出手段の出力を選択し、水平方向に相関があると判定した場合は上記水平方向信号算出手段の出力を選択し、撮像素子における画素数の第２、第３の色信号を得ることにより、偽色、偽輪郭の軽減された画像を得ることができる。
【図面の簡単な説明】
【図１】 この発明の実施の形態１における撮像素子の色フィルタの配列の一例を示す図である。
【図２】 この発明の実施の形態１による撮像装置の構成の一例を示すブロック図である。
【図３】 この発明の実施の形態１による撮像装置における第１のエッジ判定手段５とＧ成分復元手段６の構成の一例を示すブロック図である。
【図４】 この発明の実施の形態１による撮像装置における第２のエッジ判定手段８と第２のＲＢ成分復元手段９の構成の一例を示すブロック図である。
【図５】 この発明の実施の形態１による撮像装置におけるＧ信号の復元の動作を説明するためのＧ信号の画素を示す図である。
【図６】 この発明の実施の形態１による撮像装置におけるＧ信号の復元の動作を説明するためのＲ信号およびＢ信号の画素を示す図である。
【図７】 この発明の実施の形態１による撮像装置における第１のＲＢ成分復元装置７の動作を説明するためのＲ、Ｂ信号の画素を示す図である。
【図８】 この発明の実施の形態１による撮像装置における第２のエッジ判定手段８と第２のＲＢ成分復元手段９の動作を説明するためのＲ、Ｂ信号の画素を示す図である。
【図９】 この発明の実施の形態１による撮像装置と従来の装置により処理したシミュレーション画像での水平方向の解像度での偽色レベルを示す図である。
【図１０】 この発明の実施の形態１による撮像装置と従来の装置により処理したシミュレーション画像での垂直方向の解像度での偽色レベルを示す図である。
【図１１】 この発明の実施の形態１による撮像装置により処理したシミュレーション画像での水平方向の解像度での輝度信号のレベルを示す図である。
【図１２】 従来の装置により処理したシミュレーション画像での水平方向の解像度での輝度信号のレベルを示す図である。
【図１３】 この発明の実施の形態１による撮像装置により処理したシミュレーション画像での垂直方向の解像度での輝度信号のレベルを示す図である。
【図１４】 従来の装置により処理したシミュレーション画像での垂直方向の解像度での輝度信号のレベルを示す図である。
【図１５】 この発明の実施の形態１による撮像素子の他の色フィルタ配列の例を示す図である。
【図１６】 この発明の実施の形態１による撮像素子の他の色フィルタ配列の例を示す図である。
【図１７】 この発明の実施の形態２による撮像装置の構成の一例を示すブロック図である。
【図１８】 この発明の実施の形態２による撮像装置におけるＧ成分エッジ判定手段４１の動作を説明するためのＧ信号の画素を示す図である。
【図１９】 従来の撮像装置における撮像素子の色フィルタ配列の一例を示す図である。
【図２０】 従来の撮像装置の構成の一例を示すブロック図である。
【図２１】 従来の撮像装置の動作を説明するための各信号の画素を示す図である。
【図２２】 従来の撮像装置の他の構成の例を示すブロック図である。
【符号の説明】
１ 撮像素子、２ Ａ／Ｄコンバータ、３ フレームメモリ、４ 分離手段、５ 第１のエッジ判定手段、６ Ｇ成分復元手段、７ 第１のＲＢ成分復元手段、８ 第２のエッジ判定手段、９ 第２のＲＢ成分復元手段、１１，３０，４２水平方向エッジ検出手段、１２，３１，４３ 垂直方向エッジ検出手段、１３判定手段、１４ 平均値算出手段、１５〜１７ 水平方向ローパスフィルタ、１８，１９ 演算手段、２０〜２２ 垂直方向ローパスフィルタ、２３、２４ 演算手段、２５ 切り換え手段、３２ 判定手段、３３ Ｒ平均値算出手段、３４ Ｒ水平方向演算手段、３５ Ｒ垂直方向演算手段、３６ Ｒ切り換え手段、３７ Ｂ平均値算出手段、３８ Ｂ水平方向演算手段、３９ Ｂ垂直方向演算手段、４０ Ｂ切り換え手段、４１ Ｇ成分エッジ判定手段、４４判定手段。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to high image quality and downsizing in an imaging apparatus such as an electronic still camera.
[0002]
[Prior art]
Conventionally, in an imaging apparatus such as an electronic still camera, one image sensor (hereinafter referred to as a single-plate image sensor) is used for miniaturization. FIG. 19 is a diagram illustrating an example of a typical color filter array of a conventional image sensor. In the figure, R is an image sensor having a color filter having spectral characteristics that allows R light to pass through. Similarly, B and G are image sensors having respective color filters. As shown in FIG. 19, G and R are arranged every two pixels on the n line, and B and G are arranged every two pixels on the n + 1 line. Therefore, the R and B signals are obtained for every upper and lower four pixels (shaded portions in the figure), and the G signal is obtained for every two pixels. In order to improve the resolution by obtaining each signal of the number of pixels of the image sensor. Then, arithmetic processing is performed from the obtained pixel signals, and R, G, and B signals corresponding to the number of imaging elements are generated.
[0003]
FIG. 20 is a block diagram showing an example of the configuration of a conventional imaging device that generates R, G, and B signals from signals from the imaging device in the imaging device using the primary color filter shown in FIG. . In FIG. 20, 101 is an image sensor, 102 is a frame memory, 103 is a separating unit that separates signals in the frame memory 102 into R, G, and B signals, and 104 is an interpolating unit. As shown in FIG. 19, the pixel is composed of pixels R, G, and B.
[0004]
Next, the operation will be described. The pixel signals R, G, and B are read from the image sensor 101, and the pixel signals are taken into the frame memory 102. Each signal is separated by the separation means 103 from the signal taken into the frame memory 102 and sent to the interpolation means 104. Interpolation means 104 interpolates and generates pixel signals not obtained in each R, G, and B signal from adjacent pixel signals, and calculates and outputs RGB signals for the number of pixels of all image sensors.
[0005]
Here, an example of the interpolation method in the interpolation means 104 will be described with reference to FIG. The R, G, and B signals separated by the separation unit 103 are as shown in FIGS. 21A, 21B, and 21C, and the pixels indicated by G, R, and B in the drawing are obtained from the image sensor 102. On the other hand, it is a pixel signal from which no blank pixel is obtained. For the G signal (FIG. 21 (a)), in order to interpolate the signal g (hereinafter referred to as pixel position (n, m)) at the vertical n line and the horizontal mth pixel position, adjacent pixels in the vertical direction. Difference (| G (n−1, m) −G (n + 1, m) |) and the difference (| G (n, m−1) −G (n, m + 1) |) between the adjacent pixels in the left-right direction. Interpolation is performed using pixel signals in a direction in which the difference is small. For example, when the difference between adjacent pixels in the left-right direction is small, it is calculated as g (n, m) = (G (n, m−1) + G (n, m + 1)) / 2, and the difference between adjacent pixels in the vertical direction is calculated. Is small, it is calculated as g (n, m) = (G (n-1, m) + G (n + 1, m)) / 2.
[0006]
For the R and B signals (FIGS. 21B and 21C), first, horizontal pixel interpolation is performed, and then vertical interpolation is performed. For example, for R in FIG. 21B, interpolation is performed on vertical n−1 lines and n + 1 lines, and signals at pixel positions (n−1, m) (n + 1, m) are
r (n-1, m) = (R (n-1, m-1) + R (n-1, m + 1)) / 2
r (n + 1, m) = (R (n + 1, m-1) + R (n + 1, m + 1)) / 2
Next, the signals at the horizontal pixel positions m−1, m, and m + 1 are obtained by interpolating the pixels of the n lines up and down (n−1 and n + 1 lines). B can be obtained in the same manner.
[0007]
With the above interpolation method, R, G, and B signals of the number of pixels of all the image sensors can be calculated at the output of the interpolation unit 104.
[0008]
FIG. 22 shows an example of a conventional image pickup apparatus using a pixel mixing type image pickup device that reads out the upper and lower two pixels mixed and disclosed in Japanese Patent Application Laid-Open No. Hei 6-178307. It is a block diagram in the case where it is configured to interpolate and generate signals from elements from three horizontal scanning lines. In the figure, 105 is an image sensor, 106 is a frame memory, 107 is a signal selection circuit, 108 is a color interpolation circuit, and 109 is an RGB matrix. As shown in FIG. 22, the image sensor 105 is composed of four pixels A, B, C, and D (hereinafter, the number assigned to each pixel signal indicates a pixel position). A and B pixels are alternately arranged for each line so that they can be generated.
[0009]
Next, the operation will be described. Each pixel signal is read as it is without reading out the pixel mixture from the image sensor 105, and each pixel signal is taken into the frame memory 106. From the signals fetched into the frame memory 106, the signal selection circuit 107 selects adjacent three lines of signals and sends them to the color interpolation circuit 108. In the color interpolation circuit 108, the color signals A, B, C and D are generated by interpolation from the signals of the three vertical lines, and then output as RGB signals by the RGB matrix circuit 109.
[0010]
Here, the color interpolation circuit 108 generates and interpolates each color signal. This interpolation method will be described. For example, in the color signal interpolation generation of the n2 line, the signal of the vertical three lines n1, n2, and n3 is selected by the signal selection circuit 107 and sent to the color interpolation circuit 108, and the color signal obtained in the n2 line Are C and D pixels, and there are no A and B pixels. Therefore, the A and B pixels are interpolated from the n1 and n3 line signals in the vertical direction. However, since the A1 and B pixel positions are different in the n1 and n3 lines, the horizontal interpolation coefficient is changed. Become. Now, assuming that the color signals A ′, B ′, C ′, and D ′ are generated by interpolation from the five horizontal pixels before interpolation for the third pixel (n2, 3) of the n2 line of the color signal after interpolation, for example, , C ′, and D ′ are generated by interpolation with a weight only in the center in the horizontal direction,
C _{twenty three} ′ = (C _{twenty one} / 2 + C _{twenty three} + C _{twenty five} / 2) / 2
D _{twenty three} ′ = (D _{twenty two} + D _{twenty four} ) / 2
And On the other hand, for A 'and B'
A _{twenty three} ′ = (A ₁₁ / 4 + A ₁₃ / 2 + A ₁₅ / 4) / 2 + (A ₃₁ / 4 + A ₃₃ / 2 + A ₃₅ / 4) / 2
B _{twenty three} ′ = (B ₁₂ / 2 + B ₁₄ / 2) / 2 + (B ₃₂ / 2 + B ₃₄ / 2) / 2
From this equation, the horizontal pixels can be weighted and interpolated.
[0011]
Next, in the interpolation generation of the n3 line color signal, the A and B pixels are generated by interpolation from the n3 line, and the C and D pixels are generated by interpolation from the n2 and n4 lines. That is, at the pixel position (3, 3), for example,
A ₃₃ ′ = (A ₃₂ + A ₃₄ ) / 2
B ₃₃ ′ = (B ₃₁ / 2 + B ₃₃ + B ₃₅ / 2) / 2
C ₃₃ ′ = (C _{twenty one} / 4 + C _{twenty three} / 2 + C _{twenty five} / 4) / 2 + (C ₄₁ / 4 + C ₄₃ / 2 + C ₄₅ / 4) / 2
D ₃₃ ′ = (D _{twenty two} / 2 + D _{twenty four} / 2) / 2 + (D ₄₂ / 2 + D ₄₄ / 2) / 2
It becomes.
[0012]
Hereinafter, considering that the pixel arrangement of A and B is changed for each line, in the n4 line, for the pixel position (4, 3),
A ₄₃ ′ = (A ₃₁ / 4 + A ₃₃ / 2 + A ₃₅ / 4) / 2 + (A ₅₁ / 4 + A ₅₃ / 2 + A ₅₅ / 4) / 2
B ₄₃ ′ = (B ₃₂ / 2 + B ₃₄ / 2) / 2 + (B ₅₂ / 2 + B ₅₄ / 2) / 2
C ₄₃ ′ = (C ₄₁ / 2 + C ₄₃ + C ₄₅ / 2) / 2
D ₄₃ ′ = (D ₄₂ + D ₄₄ ) / 2
In the n5 line, for the pixel position (5, 3),
A ₅₃ ′ = (A ₅₂ + A ₅₄ ) / 2
B ₅₃ ′ = (B ₅₂ + B ₅₄ ) / 2
C ₅₃ ′ = (C ₄₁ / 4 + C ₄₃ / 2 + C ₄₅ / 4) / 2 + (C ₆₁ / 4 + C ₆₃ / 2 + C ₆₅ / 4) / 2
D ₅₃ ′ = (D ₄₂ / 2 + D ₄₄ / 2) / 2 + (D ₆₂ / 2 + D ₆₄ / 2) / 2
It becomes. Thereafter, the color signals A ′, B ′, C ′, and D ′ are generated by sequentially repeating the interpolation method for the n2, n3, n4, and n5 lines.
[0013]
Accordingly, the color signal is interpolated and generated by the three vertical lines, so that it is possible to obtain a signal of the number of pixels of the image sensor with little deterioration in vertical resolution of the color signal. In the above description, the four color signals are described as A, B, C, and D. For example, the four color signals are Mg (magenta), G (green), Cy (cyan), and Ye (yellow). A color can be considered, and the interpolation coefficient in the color interpolation circuit 108 may be any coefficient that can interpolate and generate a color signal.
[0014]
[Problems to be solved by the invention]
A conventional imaging device is configured to generate a color signal by interpolation using pixel signals in horizontal adjacent pixels and upper and lower lines, and a space in a local region such as an edge in an image from the obtained color signal. The change in frequency is not taken into account, and there is a problem that false colors and false contours are generated.
[0015]
The present invention has been made to solve the above-described problems. The first color filter and the second color filter having the spectral sensitivity characteristic with respect to the first color signal in the upper and lower four pixels of the vertical two rows and two horizontal columns. The second color filter having the spectral sensitivity characteristic for the color signal is arranged in the first vertical row, and the second vertical row has the same pixel position as the first color filter in the first vertical row. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged in a column, and a first color filter having a spectral sensitivity characteristic for the first color signal is arranged in the same column as the second color filter. In the imaging apparatus comprising an imaging device in which the color filters of the upper and lower four pixels are sequentially and repeatedly arranged in the vertical and horizontal directions, the first color filter At the position of Based on the edge component at the predetermined pixel position of the first color signal, the signal at the predetermined pixel position of the first color signal is calculated to restore the signal of the number of pixels of the image sensor, and the restored first color signal and Obtaining an imaging device that obtains an image with reduced false colors and false contours by calculating the second and third color signals based on the edge components and restoring signals corresponding to the number of pixels of the imaging device. Objective.
[0016]
[Means for Solving the Problems]
In the imaging device according to the present invention, the first color filter having the spectral sensitivity characteristic for the first color signal and the second spectral characteristic having the spectral sensitivity characteristic for the second color signal in the upper and lower four pixels in the vertical 2 rows and the horizontal 2 columns. Color filters are arranged in the first row, and in the second row, the spectral sensitivity characteristic for the third color signal in the same column as the pixel position in the first row where the first color filter is arranged And a first color filter having spectral sensitivity characteristics for the same first color signal as the first color filter is arranged in the same column as the second color filter. In the imaging apparatus provided with the imaging device in which the color filters of the upper and lower four pixels are sequentially and repeatedly arranged in the vertical and horizontal directions, the first color filter At the position of Predetermined pixel position of the first color signal of An edge determination unit that determines an edge component at a predetermined pixel position based on a peripheral pixel signal, and the first, second, and third color filters based on an output of the edge determination unit At the position of First calculation means for calculating a signal at the predetermined pixel position in the first color signal based on the first, second and third color signals, and based on the output of the edge determination means, the first calculation means Output and , Second calculating means for calculating signals in the second and third color signals from the second and third color signals, and the first, second, and third color signals of the number of pixels in the image sensor. To get.
[0017]
Further, in the imaging apparatus according to the present invention, the edge determination unit detects a horizontal edge component by calculating an absolute value of a difference between left and right adjacent pixels at a predetermined pixel position of the first color signal. Edge detection means; vertical edge detection means for detecting an edge component in the vertical direction by calculating an absolute value of a difference between upper and lower pixels at a predetermined pixel position of the first color signal; and And a determination unit that determines a horizontal or vertical edge component of the predetermined pixel based on an output from the direction edge detection unit.
[0018]
In the imaging apparatus according to the present invention, when the determination unit in the edge determination unit determines that the output from the horizontal edge detection unit or the output from the vertical edge detection unit is greater than a predetermined value, the predetermined unit If an edge component is detected in the peripheral pixels of the pixel, and if the output from the horizontal edge detection means is larger than the output from the vertical edge detection means, there is a correlation in the vertical direction, and the output from the horizontal edge detection means Is smaller than the output of the vertical edge detection means, it is determined that there is a correlation in the horizontal direction, and when the outputs from the horizontal edge detection means and the vertical edge detection means are both smaller than a predetermined value, It is determined that the edge component is not detected.
[0019]
Further, in the imaging apparatus according to the present invention, the first calculation means has the first color signal A, the first color signal A at the position of the predetermined pixel 1 row m column B (l, m) where the second color signal B exists. For each of the two color signals B, values Ahlpf (l, m) and Bhlpf (l, m) through a horizontal low-pass filter are calculated, and Ahlpf (l, m) which is an output signal from the horizontal low-pass filter is calculated. m) and Bhlpf (l, m) and the pixel value B (l, m) at the pixel position, the pixel value A (l, m) in the first color signal A of l rows and m columns is represented by A (L, m) = B (l, m) × {Ahlpf (l, m) / Bhlpf (l, m)}, and the same applies to the other pixel positions where the third color signal C is present. Horizontal direction signal calculation means for calculating a pixel value in the color signal A of the predetermined pixel 1 row m column B ( , M), values Avlpf (l, m) and Bvlpf (l, m) are calculated through a low-pass filter in the vertical direction for each of the first color signal A and the second color signal B, Based on the ratio of Avlpf (l, m) and Bvlpf (l, m), which are output signals from the vertical low-pass filter, and the pixel value B (l, m) at the pixel position, the first of 1 row and m column The pixel value A (l, m) in the color signal A is calculated by A (l, m) = B (l, m) × {Avlpf (l, m) / Bvlpf (l, m)} Similarly, at other pixel positions where the color signal C is present, the vertical direction signal calculation means for calculating the pixel value in the first color signal A, and the position of the predetermined pixel 1 row and m column in the first color signal A Pixels in the first color signal A of l rows and m columns from the average value of adjacent pixels in the upper, lower, left and right directions Average value calculating means for calculating A (l, m), and based on the output of the edge determining means, the output of the horizontal direction signal calculating means or the output of the vertical direction signal calculating means, or the average value calculating means A pixel value A (l, m) of the first color signal A in the predetermined pixel 1 row m column is obtained by selecting from the output, and a first color signal of the number of pixels in the image sensor is obtained.
[0020]
Further, in the imaging apparatus according to the present invention, when the first calculating unit determines that the output of the edge determining unit does not detect an edge component at the position of the predetermined pixel 1 row and m column, the average value calculating unit If it is determined that there is a correlation in the vertical direction, the output of the vertical direction signal calculation means is selected. If it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction signal calculation means is selected. Thus, the first color signal of the number of pixels in the image sensor is obtained.
[0021]
Further, in the imaging apparatus according to the present invention, the second calculation means is a value obtained by passing a low-pass filter in the horizontal direction with respect to the output A from the first calculation means at the position of the predetermined pixel 1 row and m column. A1hlpf (l, m) and the value A1vlpf (l, m) through the vertical low-pass filter are calculated, and the value B1hlpf (l, m) through the horizontal low-pass filter is calculated for the second color signal B. The value C1vlpf (l, m) through the low-pass filter in the vertical direction for the third color signal C (or the value B1vlpf (l, m) through the low-pass filter in the vertical direction for the second color signal B and the first A value C1hlpf (l, m)) through a low-pass filter in the horizontal direction is calculated for the color signal C of 3, and the ratio of A1hlpf (l, m) to B1hlpf (l, m) (or C1hlpf ( l, m)), the ratio of A1vlpf (l, m) to C1vlpf (l, m) (or the ratio of B1vlpf (l, m)), and the output from the first calculation means From the pixel value A (l, m) at a predetermined pixel l row and m column in A, the pixel value B (l, m) and C (in the second color signal B and the third color signal C in l row and m column C l, m), B (l, m) = A (l, m) × {B1hlpf (l, m) / A1hlpf (l, m)}, C (l, m) = A (l, m) × {C1vlpf (l, m) / A1vlpf (l, m)}, (or B (l, m) = A (l, m) × {B1vlpf (l, m) / A1vlpf (l, m)} C ( l, m) = A (l, m) × {C1hlpf (l, m) / A1hlpf (l, m)}) The value A2hlpf (x, y) via the low-pass filter in the horizontal direction with respect to the output A from the first calculation means at the position of the predetermined pixel x row y column different from the position, the output from the signal calculation means A value B2hlpf (x, y) through a horizontal low-pass filter is calculated for the second color signal B, and the ratio of A2hlpf (x, y) to B2hlpf (x, y) and the first calculation means are calculated. The second color signal B (x, y) at the position of the x row and the y column is represented by B (x, y) = by the pixel value A (x, y) at the pixel x row y column in the output A from Horizontal direction signal calculation means for calculating A (x, y) × {B2hlpf (x, y) / A2hlpf (x, y)} and similarly calculating the C signal for the third color signal C; 1 through the low pass filter in the vertical direction for the output A from the calculation means Value A2vlpf (x, y), a value B2vlpf (x, y) passing through a low-pass filter in the vertical direction with respect to the second color signal B output from the signal calculation means, and A2vlpf (x, y) ) And B2vlpf (x, y) and the pixel value A (x, y) at the pixel x row y column at the output A from the first calculation means, the second at the position of x row y column The color signal B (x, y) is calculated by B (x, y) = A (x, y) × {B2vlpf (x, y) / A2vlpf (x, y)}. Similarly, the vertical direction signal calculation means for calculating the C signal, and the average of pixels adjacent obliquely at the position of the predetermined pixel x row y column in the second and third color signals in the output from the signal calculation means Average value calculating means for calculating a value, based on the output of the edge determining means The second and third color signals in the predetermined pixel x row y column are obtained by selecting from the respective outputs from the horizontal direction signal calculation unit, vertical direction signal calculation unit, and average value calculation unit, and an image sensor To obtain the second and third color signals of the number of pixels.
[0022]
Furthermore, in the imaging apparatus according to the present invention, when the second calculation unit determines that the output of the edge determination unit does not detect an edge component at the position of the predetermined pixel x row y column, the average value calculation unit When the output is selected and it is determined that there is a correlation in the vertical direction, the output of the vertical direction signal calculation means is selected. When it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction signal calculation means is selected. The second and third color signals of the number of pixels in the image sensor are obtained.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof.
Embodiment 1 FIG.
FIG. 1 is a diagram showing an example of a color filter array of an image pickup device in an electronic still camera according to Embodiment 1 of the present invention, and shows an image pickup device using a primary color filter and calling each photoelectric conversion device independently. ing. In the figure, G is a pixel position in the vertical direction 2i (i = 0, 1, 2,...) And horizontal direction 2j (j = 0, 1, 2,...) (Hereinafter referred to as pixel position (2i, 2j)). The first color filter having a spectral characteristic that passes the G signal at the pixel position (2i + 1, 2j + 1), and R is the spectral characteristic that passes the R signal at the pixel position (2i, 2j + 1). B is a third color filter having a spectral characteristic that allows the B signal to pass through, at a pixel position (2i + 1, 2j). As shown in FIG. 1, the R and B signals are obtained every 4 pixels in the upper and lower directions (the shaded area in the figure), and the G signal is obtained every 2 pixels. The upper and lower 4 pixels are repeatedly arranged in the vertical and horizontal directions. ing.
[0024]
FIG. 2 has the image sensor of the color filter array shown in FIG. Embodiment 1 1 is a block diagram illustrating a configuration of an imaging device in an electronic still camera according to FIG. In the figure, 1 is an image sensor comprising the color filter array shown in FIG. 1, 2 is an A / D converter, 3 is a frame memory, 4 is a pixel signal for R, G and B, and outputs each signal. Separating means 5 is a first edge determining means for determining an edge at a predetermined pixel in the G signal, and 6 is an image based on each signal from the separating means 4 and an output from the first edge determining means 5. G component restoration means for restoring G so as to obtain a G signal of the total number of pixels in the element 1. 7 is a G signal after restoration having a signal of the total number of pixels, which is an output of the G component restoration means 6, and pixel positions (2i, 2j) in R and B from the R and B signals from the separation means 4 and ( 2i + 1, 2j + 1) is a first RB component restoring means for restoring the pixels, 8 is a second edge judging means for judging an edge at a predetermined pixel in the G signal from the G component restoring means 6, and 9 is Based on the R and B signals from the first RB component restoration means 7 and the output from the second edge determination means 8, the pixel position at the pixel position (2i + 1, 2j) at R and the pixel position at B ( 2i, 2j + 1) is a second RB component restoring means for restoring the pixel B at 2i, 2j + 1).
[0025]
FIG. 3 is a block diagram showing a configuration example of the first edge determination unit 5 and the G component restoration unit 6. In the figure, 11 is a horizontal edge detecting means for detecting a difference between left and right pixels in a predetermined pixel of the G signal, that is, an edge component, and 12 is a vertical edge for detecting an edge component that is a difference between upper and lower pixels in the predetermined pixel of the G signal. The detection means 13 is a determination means for determining a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the outputs from the horizontal and vertical edge detection means 11 and 12, and outputting the determination result. The edge detection unit 11, the vertical edge detection unit 12 and the determination unit 13 constitute a first edge determination unit 5. 14 is an average value calculating means for calculating the average value of the upper, lower, left and right pixels in a predetermined pixel of the G signal, 15 to 17 are horizontal low-pass filters, and 18 is an R signal and horizontal low-pass filter 15. At the position of Output signal Ghlpf and horizontal low-pass filter 16 At the position of Arithmetic means for calculating the output signal Rhlpf and outputting a G signal at a pixel position (2i, 2j + 1) having a correlation in the horizontal direction, 19 is a B signal and a horizontal low-pass filter 15 At the position of Output signal Ghlpf and horizontal low-pass filter 17 At the position of This is a calculation means for performing calculation based on the output signal Bhlpf and outputting a G signal at a pixel position (2i + 1, 2j) having a correlation in the horizontal direction. 20 to 22 are vertical low-pass filters, and 23 is an R signal and vertical low-pass filter 20. At the position of Output signal Gvlpf and vertical low-pass filter 21 At the position of An arithmetic means for performing an operation based on the output signal Rvlpf and outputting a G signal at a pixel position (2i, 2j + 1) having a correlation in the vertical direction, 24 is a B signal and a vertical low-pass filter 20 At the position of Output signal Gvlpf and vertical low-pass filter 22 At the position of This is a calculation means for performing calculation based on the output signal Bvlpf and outputting a G signal at a pixel position (2i + 1, 2j) having a correlation in the vertical direction. Reference numeral 25 denotes a switching unit. Based on the result of determining the signal level change of the peripheral pixels from the first edge determination unit 5 and the pixel position, the G pixel signal is converted into the average value calculation unit 14 and the calculation units 18 and 19. Further, switching is selected based on signals from the calculation means 23 and 24 and the separation means 4.
[0026]
FIG. 4 is a block diagram showing a configuration example of the second edge determination unit 8 and the second RB component restoration unit 9. In the figure, the second edge determining means 8 includes a horizontal edge detecting means 30 for detecting a difference between left and right pixels in a predetermined pixel of the G signal, a vertical edge detecting means 31 for detecting a difference between upper and lower pixels, and the horizontal Then, the output from the vertical edge detection means 30 and 31 is used to determine a change in signal level in the peripheral pixels in the horizontal and vertical directions, and the determination result for the pixel position (2i + 1, 2j) of the R signal and the pixel position of the B signal It is comprised by the determination means 32 which outputs the determination result with respect to (2i + 1, 2j). In the second RB component restoring means 9 for restoring the pixel R at the pixel position (2i + 1, 2j) in R and the pixel B at the pixel position (2i, 2j + 1) in B, 33 is the R signal An R average value calculating means 34 for calculating an average value of pixels adjacent in the oblique direction is provided by means of a G signal from the G component restoring means 6 and an R signal from the first RB restoring means 7 so that the pixel position (2i + 1, R horizontal direction calculating means for calculating the pixel signal in 2j) with the adjacent pixels in the horizontal direction, and 35 for calculating the pixel signal at the pixel position (2i + 1, 2j) with the adjacent pixels in the vertical direction in the same manner as described above. The vertical direction calculating means 36 is an R switching means for switching the R pixel in accordance with the pixel position based on the determination result from the second edge determining means 8, and 37 is an image adjacent to the diagonal direction in the B signal. B average value calculating means 38 for calculating the average value of the pixel signal at the pixel position (2i, 2j + 1) in B based on the G signal from the G component restoring means 6 and the B signal from the first RB restoring means 7. B horizontal direction calculating means for calculating the pixel signal at the pixel position (2i, 2j + 1) in the same manner as described above. 40 Is a B switching means for switching the B pixel in accordance with the pixel position based on the determination result from the second edge determination means 8.
[0027]
Next, the operation will be described. The pixel signals R, G, and B are read from the image sensor 1, and their outputs are A / D converted by the A / D converter 2 and input to the frame memory 3. The R, G, and B signals are separated from the signal input to the frame memory 3 by the separating means 4, and the G signal is sent to the first edge determining means 5 and the G component restoring means 6, where the R, B signals Is sent to the G component restoring means 6 and the first RB restoring means 7. The first edge determination unit 5 determines an edge component at a predetermined pixel position in the G signal and sends the determination result to the G component restoration unit 6, and the G component restoration unit 6 outputs the determination result from the first edge determination unit 5. Based on the determination result, the G component pixel signal is restored so as to obtain the G signal of the total number of pixels in the image sensor 1. This operation will be described with reference to FIG.
[0028]
In the first edge determination unit 5, the G signal is input to the horizontal edge detection unit 11 and the vertical edge determination unit 12. Now, as shown in FIG. 1, the G component in the image sensor 1 is obtained at pixel positions (2i, 2j) and (2i + 1, 2j + 1). In order to obtain the G signal of the number of pixels of the image sensor, the pixel position ( 2i, 2j + 1) and (2i + 1, 2j) pixels are obtained. FIG. 5 is a diagram showing the G signal at each pixel position from the separating means 4, where G indicates the G signal from the image sensor 1, and g indicates the pixel from which no signal is obtained. Therefore, in the horizontal edge detection means 11 and the vertical edge determination means 12, the difference between the left and right and upper and lower pixels at the pixel positions (2i, 2j + 1) and (2i + 1, 2j) (g indicated by hatching in FIG. 5), that is, the edge Detect ingredients. The horizontal edge detection means 11 obtains the absolute value ΔH of the difference between the left and right pixels at the pixel position, and outputs this to the determination means 13, and the vertical edge detection means 12 obtains the difference between the upper and lower pixels. An absolute value ΔV is obtained and output to the determination means 13. For example, at the pixel position (2i, 2j + 1), the horizontal edge detection means 11
ΔH = | G (2i, 2j) −G (2i, 2j + 2) | (1)
And the vertical edge detection means 12
ΔV = | G (2i−1, 2j + 1) −G (2i + 1, 2j + 1) | (2)
And ΔH and ΔV are sent to the determination means 13. Hereinafter, the absolute value of the pixel difference is referred to as an edge component.
[0029]
The determination means 13 determines a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the horizontal edge component ΔH and the vertical edge component ΔV, and outputs a signal ed1 indicating the determination result. That is, when both ΔH and ΔV are equal to or smaller than a predetermined value th1, it is determined that there is no change in the signal level in the surrounding pixels, and for example, ed1 = 1 is output. On the other hand, if ΔH or ΔV is larger than the predetermined value th1, it is determined that there is an edge component in the pixel. Further, if ΔH> ΔV, it is determined that the correlation is high in the vertical direction, and ed1 = 2 is set, for example. When ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction, and for example, ed1 = 3 is output. Note that it is not necessary to determine edge components at pixel positions (2i, 2j) and (2i + 1, 2j + 1) where G is obtained, and ed1 = 0 is output from the determination unit 13, for example. The output ed1 of the determination unit 13 is sent to the switching unit 25 in the G component restoration unit 6.
[0030]
Next, in the G component restoration unit 6, the G signal is input to the average value calculation unit 14, the horizontal low-pass filter 15, and the vertical low-pass filter 20. The average value calculating means 14 calculates an average value ga of the values of the four pixels in the vertical and horizontal directions at the pixel positions (2i, 2j + 1) and (2i + 1, 2j), and sends it to the switching means 25. The horizontal low-pass filter 15 outputs a value Ghlpf through a G horizontal low-pass filter, and the vertical low-pass filter 20 outputs a value Gvlpf through a G low-pass filter. For example, the pixel position (2i, 2j + 1) is calculated as follows:
Ghlpf = {G (2i, 2j) + G (2i, 2j-2)
+ G (2i, 2j + 2) + G (2i, 2j + 4)} / 4 (3)
Gvlpf = {G (2i-3,2j + 1) + G (2i-1,2j + 1)
+ G (2i + 1, 2j + 1) + G (2i + 3, 2j + 1)} / 4 (4)
The output Ghlpf of the horizontal low-pass filter 15 is sent to calculating means 18 and 19, and the output Gvlpf of the vertical low-pass filter 20 is sent to calculating means 23 and 24.
[0031]
On the other hand, the R signal is input to the horizontal low-pass filter 16 and the vertical low-pass filter 21 to output values Rhlpf and Rvlpf via the horizontal and vertical low-pass filters, and the horizontal low-pass filter 17 for the B signal. Input to the vertical low-pass filter 22 and output values Bhlpf and Bvlpf via the horizontal and vertical low-pass filters, respectively. Here, the R component in the image sensor 1 is obtained at the pixel position (2i, 2j + 1) as shown in FIG. 1, and the R signal at each pixel position from the separating means 4 is as shown in FIG. 6 (a). Further, the B component is obtained at the pixel position (2i + 1, 2j), and the B signal at each pixel position from the separating means 4 is as shown in FIG. A blank pixel is a pixel from which no signal is obtained by the image sensor. Therefore, at the pixel position (2i, 2j + 1), the value of the R signal through the horizontal and vertical low-pass filters 16 and 21 is obtained by the following equation, for example.

Further, at the pixel position (2i + 1, 2j), the value of the B signal through the horizontal and vertical low-pass filters 17 and 22 is obtained by the following equation, for example.

[0032]
The above equations (3) to (8) are calculation examples of the horizontal low-pass filter and the vertical low-pass filter output, and the number of taps and coefficients of the filter are not limited to the above equations (3) to (8). Other tap numbers and coefficients may be used.
[0033]
The output Rhlpf from the horizontal low-pass filter 16 is sent to the calculating means 18, and the R signal and the output signal Ghlpf from the horizontal low-pass filter 15 are also input to the calculating means 18, and the pixel position (2i, 2j + 1). ) For the G signal gh1 (2i, 2j + 1) having the horizontal correlation
gh1 (2i, 2j + 1) = R (2i, 2j + 1) × (Ghlpf / Rhlpf) (9)
And is sent to the switching means 25. The output Rvlpf from the vertical low-pass filter 21 is sent to the calculating means 23. The R signal and the output signal Gvlpf from the vertical low-pass filter 20 are also input to the calculating means 23, and the pixel position (2i, 2j + 1). G signal gv1 (2i, 2j + 1) having a correlation in the vertical direction at
gv1 (2i, 2j + 1) = R (2i, 2j + 1) × (Gvlpf / Rvlpf) (10)
And is sent to the switching means 25.
[0034]
Similarly, the output Bhlpf from the horizontal low-pass filter 17 is sent to the calculating means 19, and the B signal and the output signal Ghlpf from the horizontal low-pass filter 15 are also input to the calculating means 19, and the pixel position (2i + 1, Gj gh2 (2i + 1, 2j) having a horizontal correlation in 2j)
gh2 (2i + 1, 2j) = B (2i + 1, 2j) × (Ghlpf / Bhlpf) (11)
And is sent to the switching means 25. The output Bvlpf from the vertical low-pass filter 22 is sent to the calculating means 24, and the B signal and the output signal Gvlpf from the vertical low-pass filter 20 are also input to the calculating means 24, and the pixel position (2i + 1, 2j). G signal gv2 (2i + 1, 2j) correlated in the vertical direction at
gv2 (2i + 1, 2j) = B (2i + 1, 2j) × (Gvlpf / Bvlpf) (12)
And is sent to the switching means 25.
[0035]
The calculation methods according to the above formulas (9) to (12) are based on the premise that the color change in the local region is small, that is, the ratio of the color signals in the local region is substantially equal, Alternatively, the ratio of each color signal in a local region in the vertical direction is given by a ratio of values of R, G, B through a low-pass filter in the horizontal or vertical direction.
[0036]
In the switching means 25, based on the result ed1 and the pixel position of the peripheral pixel signal level change from the determination means 13 in the first edge determination means 5, the G pixel signal is converted into the average value calculation means 14 and the calculation means 18. , 19 and arithmetic means 23 and 24, and signals from the separating means 4 are selected and switched. That is, when the output signal ed1 by the determination unit 13 indicates “2”, that is, it is determined that there is an edge component in the pixel, but when ΔH> ΔV and it is determined that the correlation is high in the vertical direction, A signal of a computing means that outputs G having a correlation in the vertical direction at the pixel position is selected. When the output signal ed1 by the determination unit 13 indicates '3', that is, when it is determined that there is an edge component in the pixel, but when ΔH ≦ ΔV and it is determined that the correlation is high in the horizontal direction, each pixel The signal of the computing means that outputs G having a correlation in the horizontal direction at the position is selected. Further, when the output signal ed1 by the determination unit 13 indicates “1”, it is determined that both ΔH and ΔV are equal to or less than a predetermined value th1, and there is no change in the signal level in the peripheral pixels. The G signal ga by the average value calculation means 14 is selected without considering the change in frequency.
[0037]
That is, in the switching means 25, in the case of the G signal g (2i, 2j + 1) at the pixel position (2i, 2j + 1), when the output ed1 = 1 from the determination means 13, the G signal ga from the average value calculation means 14 is When G = 2, the G signal gv1 from the computing means 23 is selected. When ed = 3, the G signal gh1 from the computing means 18 is selected, while the G signal g (2i + 1, 2j) at the pixel position (2i + 1, 2j) is selected. ) In the case of the output ed1 = 1 from the determination means 13, the G signal ga from the average value calculation means 14, the G signal gv2 from the calculation means 24 in the case of ed = 2, and the calculation in the case of ed = 3. The G signal gh2 from the means 19 is selected. At the pixel positions (2i, 2j) and (2i + 1, 2j + 1) where G is obtained, for example, ed1 = 0 is output from the determination means 13, and at this time, the G signal from the separation means 4 is output. Just output as it is.
[0038]
Therefore, the G component restoration means 6 outputs G signals at the respective pixel positions (2i, 2j), (2i, 2j + 1), (2i + 1, 2j), (2i + 1, 2j + 1), that is, A G signal having a resolution corresponding to the number of pixels can be obtained. The output G0 from the G component restoring means 6 is then sent to the first RB component restoring means 7, the second edge determining means 8 and the second RB component restoring means 9.
[0039]
Next, in the first RB component restoration means 7, the G signal having the R and B signals (FIGS. 6A and 6B) from the separation means 4 and the signals of all the pixels from the G component restoration means 6. With G0, the pixels at the pixel positions (2i, 2j) and (2i + 1, 2j + 1) in the R and B signals are restored and generated. FIGS. 7A and 7B are diagrams showing R and B of each pixel for explaining the calculation of the R and B signals in the first RB component restoring means 7, in which r and b are the first The R and B pixel signals restored and generated by the RB component restoring means 7 are shown. At the pixel position (2i, 2j) in the R signal, pixels in the 2i line adjacent in the horizontal direction are obtained from the image sensor 1, and the G signal is restored by the G component restoring means 6 for all pixels. (FIG. 5). Therefore, at the pixel position (2i, 2j), values G1hlpf and R1hlpf through the low-pass filter in the horizontal direction with respect to the G and R signals are, for example,

Based on the ratio of G1hlpf and R1hlpf and the pixel G (2i, 2j), the R signal r (2i, 2j) at the pixel position (2i, 2j) is calculated by the following equation.
r (2i, 2j) = G (2i, 2j) × (R1hlpf / G1hlpf)
= G (2i, 2j) x (R (2i, 2j-1) + R (2i, 2j + 1))
/ (G (2i, 2j-1) + g (2i, 2j + 1)) (15)
[0040]
In addition, at the pixel position (2i + 1, 2j + 1), 2j + 1 columns of pixels adjacent in the vertical direction are obtained from the image sensor 1, and the G signal is restored by the G component restoring means 6 so that all pixels are restored. At the pixel position (2i + 1, 2j + 1), values G1vlpf and R1vlpf through a low-pass filter in the vertical direction with respect to the G and R signals are, for example,

Based on the ratio of G1vlpf and R1vlpf and the pixel G (2i + 1, 2j + 1), the R signal r (2i + 1, 2j + 1) at the pixel position (2i + 1, 2j + 1) is calculated by the following equation.
r (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (R1vlpf / G1vlpf)
= G (2i + 1,2j + 1) × (R (2i, 2j + 1) + R (2i + 2,2j + 1))
/ (G (2i, 2j + 1) + g (2i + 2, 2j + 1)) (18)
[0041]
Similarly, with respect to the B signal, 2j columns of pixels adjacent in the vertical direction are obtained from the image sensor 1 at the pixel position (2i, 2j). It has been restored (FIG. 5). Therefore, at the pixel position (2i, 2j), the values G1vlpf and B1vlpf through the low-pass filter in the vertical direction with respect to the G and B signals are, for example,

The B signal b (2i, 2j) at the pixel position (2i, 2j) is calculated from the ratio of G1vlpf and B1vlpf and the pixel G (2i, 2j) by the following equation.
b (2i, 2j) = G (2i, 2j) × (B1vlpf / G1vlpf)
= G (2i, 2j) x (B (2i-1, 2j) + B (2i + 1, 2j))
/ (G (2i-1,2j) + g (2i + 1,2j)) (21)
[0042]
Further, at the pixel position (2i + 1, 2j + 1), pixels in the 2i + 1 line that are adjacent in the horizontal direction are obtained from the image sensor 1, and the G signal is restored by the G component restoring means 6 in all pixels. Therefore, at the pixel position (2i + 1, 2j + 1), values G1hlpf and B1hlpf through the low-pass filter in the horizontal direction with respect to the G and B signals are, for example,

The B signal b (2i + 1, 2j + 1) at the pixel position (2i + 1, 2j + 1) is calculated from the ratio of G1hlpf and B1hlpf and the pixel G (2i + 1, 2j + 1) by the following equation.
b (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (B1hlpf / G1hlpf)
= G (2i + 1,2j + 1) × (B (2i + 1,2j) + B (2i + 1,2j + 2))
/ (G (2i + 1, 2j) + g (2i + 1, 2j + 2)) (24)
[0043]
Note that the above formulas (15), (18), (21), and (24) are based on the premise that there is little change in the color signal in the local region as in the restoration method in G. This is because the ratio is almost equal in the local region. The formulas for calculating G1hlpf, G1vlpf, R1hlpf, R1vlpf, B1hlpf, and B1vlpf in the equations (13) to (24) are examples of calculating the low-pass filter outputs in the horizontal and vertical directions. The number of taps and the coefficient may be other than the tap. Then, R, r, B, and b pixel signals R 1 and B 1 shown in FIG. 7 are output from the first RB component restoring means 7 and sent to the second RB component restoring means 9.
[0044]
Next, the operations of the second RB component restoration unit 9 and the second edge determination unit 8 will be described with reference to FIG. In the second edge determination means 8, the G signal G0 output from the G component restoration means 6 is input to the horizontal direction edge detection means 30 and the vertical direction edge determination means 31, and the pixel positions (2i, 2j + 1) and (2i + 1). 2j), the difference between the left and right and upper and lower pixels, that is, the edge component is detected. The horizontal edge detection means 30 obtains the absolute value ΔH of the difference between the left and right pixels at the pixel position, and outputs this to the determination means 32. The vertical edge detection means 31 obtains the difference between the upper and lower pixels. An absolute value ΔV is obtained and output to the determination means 32. For example, at the pixel position (2i, 2j + 1), the horizontal edge detection means 30
ΔH = | G (2i, 2j) −G (2i, 2j + 2) |
And the vertical edge detection means 31
ΔV = | G (2i−1,2j + 1) −G (2i + 1,2j + 1) |
And ΔH and ΔV are sent to the determination means 32. Hereinafter, the absolute value of the pixel difference is referred to as an edge component.
[0045]
The determination means 32 determines a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the horizontal edge component ΔH and the vertical edge component ΔV, and outputs a signal indicating the determination result. Here, pixel signals other than the pixel position (2i + 1, 2j) are obtained in the R signal R1 by the first RB component restoring means 7 (FIG. 7 (a)), and the pixel position ( Pixel signals other than 2i, 2j + 1) are obtained (FIG. 7B). Therefore, as a result of determining the edge component at the pixel position (2i + 1, 2j), edr is output for the restoration of the pixel position (2i + 1, 2j) of the R signal, while the edge component at the pixel position (2i, 2j + 1) is output. The result edb is output for restoration of the pixel position (2i, 2j + 1) of the B signal. That is, in the determination result edr at the pixel position (2i + 1, 2j) and the determination result edb at the pixel position (2i, 2j + 1), when both ΔH and ΔV are equal to or smaller than a predetermined value th2, the signal level of the surrounding pixels It is determined that there is no change, and for example, output is performed as edr = 1 and edb = 1. On the other hand, if ΔH or ΔV is greater than a predetermined value th2, it is determined that there is an edge component in the pixel. Further, if ΔH> ΔV, it is determined that the correlation is high in the vertical direction. For example, edr = 2 And edb = 2 are output, and when ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction, and for example, edr = 3 and edb = 3 are output. Note that it is not necessary to determine the edge component at the pixel position where R and B are obtained, and edr = 0 and edb = 0 are output from the determination unit 32, for example. The output edr of the determination means 32 is sent to the R switching means 36 in the second RB component restoration means 9, and the output edb is sent to the B switching means 40 in the second RB component restoration means 9.
[0046]
Next, in the second RB component restoring means 9, the R signal output R1 from the first RB component restoring means 7 is an R average value calculating means 33, an R horizontal direction calculating means 34, an R vertical direction calculating means 35, R Input to the switching means 36. FIG. 8A is a diagram for explaining the restoration of the R signal at the pixel position (2i + 1, 2j) in the second RB component restoration means 9, and the pixels indicated by R and r in the figure are imaging elements. The obtained signal and the pixel signal restored by the first RB restoration means 7, and therefore, the signal of the r ′ pixel indicated by the oblique line is obtained. First, in the R average value calculation means 33, four pixels R (2i, 2j-1), R (2i, 2j + 1), R (2i + 2) that are adjacent in the diagonal direction at the pixel position (2i + 1, 2j) in the R signal R1. 2j-1) and R (2i + 2, 2j + 1) are calculated by the following equation.
ra (2i + 1, 2j) = {R (2i, 2j-1) + R (2i, 2j + 1)
+ R (2i + 2,2j-1) + R (2i + 2,2j + 1)} / 4
[0047]
The R horizontal direction calculating means 34 also receives the G signal G0 (FIG. 5) from the G component restoring means 6, and the value of the R and G signal via the horizontal low-pass filter at the pixel position (2i + 1, 2j). R2hlpf, G2hlpf, for example,

Based on the ratio of R2hlpf and G2hlpf and the pixel g (2i + 1, 2j), the pixel value rh having a correlation in the horizontal direction of (2i + 1, 2j) is calculated and output by the following equation.
rh (2i + 1, 2j) = g (2i + 1, 2j) × (R2hlpf / G2hlpf)
= G (2i + 1, 2j)
X (r (2i + 1,2j-1) + r (2i + 1,2j + 1))
/ (G (2i + 1,2j-1) + G (2i + 1,2j + 1)) (27)
[0048]
The R vertical direction calculation means 35 also receives the G signal G0 from the G component restoration means 6, and at the pixel positions (2i + 1, 2j) in the R and G signals, the values R2vlppf and G2vlppf are passed through the low-pass filter in the vertical direction. For example,
R2vlpf = (r (2i, 2j) + r (2i + 2, 2j)) / 2 (28)
G2vlpf = (G (2i, 2j) + G (2i + 2, 2j)) / 2 (29)
Based on the ratio of R2vlpf and G2vlpf and the pixel g (2i + 1, 2j), a pixel value rv having a correlation in the vertical direction of (2i + 1, 2j) is calculated and output by the following equation.
rv (2i + 1, 2j) = g (2i + 1, 2j) × (R2vlpf / G2vlpf)
= G (2i + 1, 2j) x (r (2i, 2j) + r (2i + 2, 2j))
/ (G (2i, 2j) + G (2i + 2, 2j)) (30)
[0049]
Note that, as described above, the above equations (27) and (30) are based on the premise that there is little change in the color signal in the local region, that is, the ratio of each signal is almost equal in the local region. In addition, the calculation formulas of G2hlpf, G2vlpf, R2hlpf, and R2vlpf in the formulas (25) to (30) are examples of calculating the low-pass filter output in the horizontal and vertical directions, and the number of taps and coefficients of the filter are limited to the above Instead, other tap numbers and coefficients may be used. The outputs rh and rv are both sent to the R switching means 36.
[0050]
In the R switching unit 36, the signal at the pixel position (2i + 1, 2j) is converted to the R average value calculating unit 33 based on the result edr of determining the signal level change of the peripheral pixels from the determining unit 32 in the second edge determining unit 8. , R horizontal direction calculating means 34, R vertical direction calculating means 35, and signals from the first RB component restoring means 7 are selected and switched. That is, when the output signal edr by the determination unit 32 indicates “2”, that is, when it is determined that there is an edge component in the pixel, but ΔH> ΔV and it is determined that the correlation in the vertical direction is high, the vertical The signal rv of the R vertical direction calculation means 35 that outputs R having a correlation in the direction is selected. When the output signal edr by the determination unit 32 indicates '3', that is, when it is determined that there is an edge component in the pixel, but when ΔH ≦ ΔV and it is determined that the correlation in the horizontal direction is high, the horizontal direction The signal of the R horizontal direction calculation means 34 that outputs R having a correlation to is selected. Furthermore, when the output signal edr by the determination unit 32 indicates “1”, both ΔH and ΔV are equal to or less than a predetermined value th2, and it is determined that there is no change in the signal level in the surrounding pixels. The R signal ra by the R average value calculation means 33 is selected without considering the change in frequency. It should be noted that at the pixel positions (2i, 2j), (2i, 2j + 1), (2i + 1, 2j + 1) where R is obtained, for example, edr = 0 is output from the determination means 32. The R signal from the RB component restoring means 7 may be output as it is.
[0051]
The processing for the B signal is the same as the processing for the R signal. In the second RB component restoring means 9, the B signal output B1 from the first RB component restoring means 7 is the B average value calculating means 37, B horizontal direction. The calculation means 38, the B vertical direction calculation means 39, and the B switching means 40 are input. FIG. 8B is a diagram for explaining the restoration of the B signal at the pixel position (2i, 2j + 1) in the second RB component restoration means 9, and the pixels indicated by B and b in the figure are imaging elements. The obtained signal and the pixel signal restored by the first RB restoration means 7, and therefore, the signal of the b ′ pixel indicated by the oblique line is obtained. First, the B average value calculation means 37 calculates the average value ba of four pixels adjacent in the diagonal direction at the pixel position (2i, 2j + 1) in the B signal B1 by the following equation.
ba (2i, 2j + 1) = {B (2i-1, 2j) + B (2i-1, 2j + 2)
+ B (2i + 1, 2j) + B (2i + 1, 2j + 2)} / 4
[0052]
The B horizontal direction calculation means 38 also receives the G signal G0 (FIG. 5) from the G component restoration means 6, and at the pixel position (2i, 2j + 1) in the B and G signals, the value via the horizontal low-pass filter. B2hlpf and G3hlpf are, for example, B2hlpf = (b (2i, 2j) + b (2i, 2j + 2)) / 2 (31)
G3hlpf = (G (2i, 2j) + G (2i, 2j + 2)) / 2 (32)
Based on the ratio of B2hlpf and G3hlpf and the pixel g (2i, 2j + 1), a pixel value bh having a correlation in the horizontal direction of (2i, 2j + 1) is calculated and output by the following equation.
bh (2i, 2j + 1) = g (2i, 2j + 1) × (B2hlpf / G3hlpf)
= G (2i, 2j + 1) × (b (2i, 2j) + b (2i, 2j + 2))
/ (G (2i, 2j) + G (2i, 2j + 2)) (33)
[0053]
In the B vertical direction calculating means 39, the G signal G0 from the G component restoring means 6 is also input, and the values B2vlpf and G3vlpf through the vertical low-pass filter are obtained at the pixel position (2i, 2j + 1) in the B and G signals. For example,

Based on the ratio of B2vlpf and G3vlpf and the pixel g (2i, 2j + 1), a pixel value bv having a correlation in the vertical direction of (2i, 2j + 1) is calculated by the following equation.
bv (2i, 2j + 1) = g (2i, 2j + 1) × (B2vlpf / G3vlpf)
= G (2i, 2j + 1) * (b (2i-1, 2j + 1) + b (2i + 1, 2j + 1))
/ (G (2i-1,2j + 1) + G (2i + 1,2j + 1)) (36)
[0054]
Note that, as described above, the above formulas (33) and (36) are based on the premise that the change of the color signal in the local region is small, that is, the ratio of each signal is almost equal in the local region. In addition, the calculation formulas for G3hlpf, G3vlpf, B2hlpf, and B2vlpf in the equations (31) to (36) are examples of calculating the low-pass filter output in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above. Instead, other tap numbers and coefficients may be used. The outputs bh and bv are both sent to the B switching means 40.
[0055]
In the B switching unit 40, the signal at the pixel position (2i, 2j + 1) is converted to the B average value calculating unit 37 based on the result edb of determining the signal level change of the peripheral pixels from the determining unit 32 in the second edge determining unit 8. , B horizontal direction calculating means 38, B vertical direction calculating means 39, and signals from the first RB component restoring means 7 are selected and switched. That is, when the output signal edb by the determination unit 32 indicates “2”, that is, when it is determined that there is an edge component in the pixel but ΔH> ΔV and it is determined that the correlation in the vertical direction is high, the vertical The signal bv of the R vertical direction calculation means 39 that outputs R having a correlation in the direction is selected. When the output signal edb by the determination unit 32 indicates “3”, that is, when it is determined that there is an edge component in the pixel but ΔH ≦ ΔV and it is determined that the correlation in the horizontal direction is high, the horizontal direction The signal of the B horizontal direction calculation means 38 that outputs B having a correlation to is selected. Further, when the output signal edb by the determination unit 32 indicates “1”, both ΔH and ΔV are equal to or less than a predetermined value th2, and it is determined that there is no change in the signal level in the surrounding pixels. The B signal ba by the B average value calculating means 37 is selected without considering the change in frequency. For example, edb = 0 is output from the determination unit 32 at the pixel positions (2i, 2j), (2i + 1, 2j), and (2i + 1, 2j + 1) from which B is obtained. The B signal from the RB component restoring means 7 may be output as it is.
[0056]
From the above, the second RB component restoring means 9 outputs R and B signals at the respective pixel positions (2i, 2j), (2i, 2j + 1), (2i + 1, 2j), (2i + 1, 2j + 1). That is, it becomes possible to obtain R and B signals having a resolution equivalent to the number of pixels of the image sensor. The G component restoration means 6 has already obtained a signal having a resolution equivalent to the number of pixels of the image sensor. Therefore, the G component is used to determine the edge component in the horizontal and vertical directions and the spatial frequency in the local region. And the generation of each color signal is switched based on the determination result, and the signal is calculated based on the ratio of the color signal in the local area when the edge component exceeds a predetermined value. When the value is equal to or less than a predetermined value, the average value of the surrounding pixels is obtained, and therefore, an image with reduced false color and false contour is obtained.
[0057]
FIG. 9 shows the relationship between the false color level (color-alias level, vertical axis) generated in the horizontal direction when the zone plate is processed by image simulation, and the resolution (horizontal axis). The false color level after processing according to the prior art is indicated by a wavy line, and the false color level after processing according to the first embodiment shown in FIG. 2 is indicated by a solid line. Although many false colors are generated in the processing according to the conventional technique, the generation of false colors is suppressed in the processing according to the first embodiment. When the level of the false color signal in the horizontal direction is integrated and compared in each processing, there is a suppression effect of about 18.4 dB.
[0058]
FIG. 10 shows the relationship between the false color level (color-alias level, vertical axis) generated in the vertical direction and the resolution (horizontal axis). Similarly to the above, the prior art according to FIG. The false color level after processing according to FIG. 2 is indicated by a wavy line, and the false color level after processing according to the first embodiment shown in FIG. 2 is indicated by a solid line. Although many false colors are generated in the processing according to the conventional technique, the generation of false colors is suppressed in the processing according to the first embodiment. When the levels of the false color signals in the vertical direction are integrated and compared, about There is a suppression effect of 18.5 dB.
[0059]
Further, FIG. 11 and FIG. 12 are diagrams in which the luminance signal level in the horizontal direction when the zone plate is processed by image simulation is calculated from the RGB signal, and the vertical axis represents the resolution and the horizontal axis represents the resolution. FIG. 11 shows the case after the processing according to the first embodiment, and FIG. 12 shows the case after the processing according to the conventional technique shown in FIG. FIG. 13 shows the relationship between the luminance signal level and the resolution in the vertical direction after processing according to the first embodiment shown in FIG. 2, and FIG. 14 shows the case after processing according to the conventional technique shown in FIG. In both the horizontal and vertical directions, the processing according to the first embodiment has a lower level of reduction and the horizontal and vertical resolution is improved as compared with the processing according to the conventional technique.
[0060]
In the first embodiment, the arrangement of the color filters of the image sensor 1 is the primary color filter shown in FIG. 1, and each photoelectric conversion element is an image sensor that calls each photoelectric conversion element in a pixel position (2i, 2j). And a pixel position (2i + 1, 2j + 1) (i = 0, 1, 2,..., J = 0, 1, 2,...) With a first color filter having spectral characteristics that allow the G signal to pass through the pixel position ( 2i, 2j + 1) is arranged with a second color filter having a spectral characteristic that allows an R signal to pass, and a third color filter having a spectral characteristic that allows a B signal to pass at a pixel position (2i + 1, 2j). As described above, the spectral characteristics of the first, second, and third color filters in the image sensor are not limited to R, G, and B. For example, a pixel mixture type image sensor may be used. As shown, pixel position (2i, 2j) and A first color filter having spectral characteristics that allows the first signal A to pass through to a prime position (2i + 1, 2j + 1) (i = 0, 1, 2,..., J = 0, 1, 2,...) A second color filter having a spectral characteristic that allows the second signal B to pass through the position (2i, 2j + 1) and a third color filter that has the spectral characteristic that allows the third signal C to pass through the pixel position (2i + 1, 2j). It is only necessary to reconstruct RGB color signals after restoring the A, B, and C signals corresponding to the number of pixels of the image sensor by the processing of the first embodiment, and the same effects as described above can be obtained. .
[0061]
In the first embodiment, the arrangement of the color filters of the image sensor shown in FIGS. 1 and 15 is the pixel position (2i, 2j) and the pixel position (2i + 1, 2j + 1) (i = 0, 1, 2,..., J = 0, 1, 2,...) When the first color filter is arranged at the pixel position (2i, 2j + 1), the third color filter is arranged at the pixel position (2i + 1, 2j) ( 1 and 15, the pixel position (2i, 2j + 1) and the pixel position (2i + 1, 2j) (i = 0, 1, 2,..., J = (0, 1, 2,...), The first color filter is arranged at the pixel position (2i, 2j), and the third color filter is arranged at the pixel position (2i + 1, 2j + 1) (FIG. 16 also has the same effect, and the first line is shown in the n-line every four upper and lower pixels. The filter and the second color filter are arranged, the third color filter and the first color filter are arranged in the n + 1 line, and the first filter is arranged in the diagonal pixels in the upper and lower lines. .
[0062]
In the first embodiment, the signal indicating the determination result in the first edge determination unit 5 and the second edge determination unit 8 is “1” when both ΔH and ΔV are equal to or less than a predetermined value. In the case where it is determined that there is an edge component, “2” is set when ΔH> ΔV, “3” is set when ΔH ≦ ΔV, and “0” is output at the pixel position where the pixel signal is obtained. Although described, the present invention is not limited to this, and any other value may be used as long as a signal that can distinguish each determination result is output.
[0063]
In the first embodiment, the case where the process of the configuration in FIG. 2 is performed by hardware has been described. Needless to say, the same process can be performed by software in the imaging apparatus. Has the same effect as.
[0064]
Embodiment 2. FIG.
In the first embodiment, the first edge determination means 5 determines the edge components in the vertical and horizontal directions at the pixel positions (2i, 2j + 1) and (2i + 1, 2j) in the G signal, and the second edge determination means 8 Is configured to determine an edge component at the pixel position (2i + 1, 2j) in the G signal for restoration of the R signal and an edge component at the pixel position (2i, 2j + 1) in the G signal for restoration of the B signal. However, in both cases, edge components of left and right and upper and lower pixels at pixel positions (2i, 2j + 1) and (2i + 1, 2j) are detected, and upper, lower, left and right pixel signals at these pixel positions are obtained from the image sensor. Therefore, the same edge component is obtained. Therefore, as shown in FIG. 17, it is possible to adopt a configuration in which the edge component at the pixel positions (2i, 2j + 1) and (2i + 1, 2j) in the G signal is determined by one G component edge determination means.
[0065]
In FIG. 17, 1 to 4, 6 to 7 and 9 are the same as those in the image pickup apparatus in the first embodiment, 41 is a G component edge determining means, horizontal edge detecting means 42, vertical direction. An edge detection unit 43 and a determination unit 44 are included.
[0066]
Next, the operation will be described. Each pixel signal R, G, B is read from the image sensor 1, A / D converted by the A / D converter 2, input to the frame memory 3, and each R, G, B signal is separated by the separating means 4, G Since the operation of generating and restoring each signal of the number of pixels of the image sensor by the component restoring means 6, the first RB restoring means 7 and the second RB restoring means 9 is the same as that in the first embodiment, detailed description thereof will be given. Is omitted.
[0067]
The G signal (FIG. 18) from the separating means 4 is input to the G component edge determining means 41 and is supplied to the horizontal edge detecting means 42 and the vertical edge detecting means 43 in the G component edge determining means 41. In the horizontal edge detection means 42 and the vertical edge detection means 43, the difference between the left and right and upper and lower pixels at the pixel positions (2i, 2j + 1), (2i + 1, 2j) (pixels indicated by diagonal lines in FIG. 18), that is, the edge component is calculated. To detect. That is, the horizontal edge detection means 42 obtains the absolute value ΔHg of the difference between the left and right pixels at the pixel position, and outputs this to the determination means 44. The vertical edge detection means 43 outputs the absolute value of the upper and lower pixels. The absolute value ΔVg of the difference is obtained and output to the determination means 55. For example, at the pixel position (2i, 2j + 1), the horizontal edge detection means 42
ΔHg = | G (2i, 2j) −G (2i, 2j + 2) |
And the vertical edge detection means 43
ΔVg = | G (2i−1,2j + 1) −G (2i + 1,2j + 1) |
And ΔHg and ΔVg are sent to the determination means 44.
[0068]
The determination means 44 determines a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the edge component ΔHg and the vertical edge component ΔVg in the horizontal direction, and according to each pixel position of the input G signal. The signal ed1 indicating the determination result is output to the G component restoring means 6 and edr and edb are outputted to the second RB component restoring means 9. When both ΔHg and ΔVg are equal to or less than a predetermined value, it is determined that there is no change in the signal level in the surrounding pixels, and each determination signal is set to “1”, for example. On the other hand, if ΔHg or ΔVg is larger than a predetermined value, it is determined that there is an edge component in the pixel. Further, if ΔHg> ΔVg, it is determined that the correlation is high in the vertical direction, for example, “2”. In the case of ΔHg ≦ ΔVg, it is determined that the correlation is high in the horizontal direction, for example, “3”. Here, since the G component restoration means 6 restores the pixels at the pixel positions (2i, 2j + 1) and (2i + 1, 2j), the edge determination result is output as ed1 at both pixel positions, and G is obtained. For example, ed1 = 0 is output at the pixel positions (2i, 2j) and (2i + 1, 2j + 1). Then, the output ed1 of the determination means 44 is sent to the switching means 25 in the G component restoration means 6.
[0069]
On the other hand, the second RB component restoration unit 9 restores the pixel position (2i + 1, 2j) of the R signal and restores the pixel position (2i, 2j + 1) of the B signal. Therefore, the result of determining the edge component at the pixel position (2i + 1, 2j) is set as edr and output in time with the pixel position to be processed by the second RB component restoration unit 9, and the other pixel positions (2i, In 2j), (2i, 2j + 1), (2i + 1, 2j + 1), for example, edr = 0 is output. Then, the output edr of the determination means 44 is sent to the R switching means 36 in the second RB component restoration means 9. On the other hand, the result of determining the edge component at the pixel position (2i, 2j + 1) is set as edb, and is output in time with the pixel position to be processed by the second RB component restoration unit 9, and the other pixel positions (2i, 2j, In 2j), (2i + 1, 2j), and (2i + 1, 2j + 1), for example, edb = 0 is output. Then, the output edb of the determination unit 44 is sent to the B switching unit 40 in the second RB component restoration unit 9.
[0070]
Therefore, in the G component edge determination means 41, the edge component determination result in the G signal corresponding to the pixel position where the G, R, and B signals are restored in the G component restoration means 6 and the second RB component restoration means 9 is obtained. Will be output.
[0071]
In the second embodiment, the G signal is at the pixel positions (2i, 2j) and (2i + 1, 2j + 1), the R signal is at the pixel position (2i, 2j + 1), and the B signal is at the pixel position (2i + 1, 2j + 1). As described in the first embodiment, the G signal is obtained at the pixel positions (2i, 2j + 1), (2i + 1, 2j), and the R signal is obtained at the pixel position (2i, 2j). B signals may be arranged at (2i + 1, 2j + 1), and the color signals are not limited to RGB.
[0072]
Further, as in the first embodiment, in the second embodiment, it is needless to say that the same processing as the processing in the configuration of FIG. 17 can be performed by software in the imaging apparatus. Play.
[0073]
【The invention's effect】
Since the present invention is configured as described above, the following effects can be obtained.
[0074]
According to the imaging apparatus according to the present invention, the first color signal at the pixel positions (2i, 2j) (i = 0, 1, 2,... And j = 0, 1, 2,...) And (2i + 1, 2j + 1). A first color filter having a spectral characteristic that allows the second color signal to pass through the pixel position (2i, 2j + 1), and a second color filter having a spectral characteristic that allows the second color signal to pass through the pixel position (2i + 1, 2j). In the image pickup device in which the upper and lower four pixels in which the third color filters having spectral characteristics that pass the third color signal are arranged are repeatedly arranged in the vertical and horizontal directions, the first color filter At the position of At the pixel positions (2i, 2j + 1), (2i + 1, 2j) of the first color signal, an edge component is determined based on the peripheral pixel signal, and based on the determination result, the first, second, and third color signals are used. A signal of the first color signal at the pixel position is calculated to restore the first color signal of the number of pixels of the image sensor, and based on the edge determination result, the restored first color signal and each color filter are used. By calculating the second and third color signals from the color signals and obtaining the second and third color signals of the number of pixels of the image sensor, an image with reduced false colors and false contours can be obtained.
[0075]
Further, according to the imaging apparatus of the present invention, the edge determination means detects the edge component ΔH in the horizontal direction by calculating the absolute value of the difference between the left and right adjacent pixels in the predetermined pixel of the first color signal. The edge value ΔV in the vertical direction is detected by calculating the absolute value of the difference between the pixels, and the edge component in the horizontal or vertical direction at the predetermined pixel is determined based on the edge component ΔH in the horizontal direction and the edge component ΔV in the vertical direction. By making the determination, it is possible to determine a change in the horizontal and vertical spatial frequencies in the local region, and an image with reduced false colors and false contours can be obtained.
[0076]
According to the imaging apparatus of the present invention, when the determination unit in the edge determination unit has an output ΔH from the horizontal direction edge detection unit or an output ΔV from the vertical direction edge detection unit greater than a predetermined value. Determines that an edge component is detected in a peripheral pixel of a predetermined pixel. Further, if ΔH> ΔV, it is determined that there is a correlation in the vertical direction, and if ΔH ≦ ΔV, it is determined that there is a correlation in the horizontal direction. When ΔV is smaller than a predetermined value, it is possible to determine a change in the spatial frequency in the horizontal and vertical directions in a local region by determining that no edge component is detected. Can be obtained.
[0077]
According to the imaging apparatus of the present invention, the means for calculating and restoring the first color signal is the position (l, m) of the predetermined pixel 1 row m column B (l, m) where the second color signal B is present. = 2i, m = 2j + 1 or l = 2i + 1, m = 2j), the value Ahlpf (l, m) through the horizontal low-pass filter for each of the first color signal A and the second color signal B , Bhlpf (l, m) is calculated, and the ratio of Ahlpf (l, m) and Bhlpf (l, m), which are output signals from the horizontal low-pass filter, and the pixel value B (l, m) at the pixel position are calculated. ), The pixel value A (l, m) in the first color signal A of l rows and m columns is represented by A (l, m) = B (l, m) × {Ahlpf (l, m) / Bhlpf (l , M)}, and the first color is similarly applied to other pixel positions having the third color signal C. Horizontal direction signal calculation means for calculating a pixel value in the signal A, and at each of the first color signal A and the second color signal B at the position of the predetermined pixel 1 row m column B (l, m). The values Avlpf (l, m) and Bvlpf (l, m) are calculated through the low-pass filter in the vertical direction, and Avlpf (l, m) and Bvlpf (l, m) which are output signals from the vertical low-pass filter are calculated. And the pixel value A (l, m) as the first color signal A in l rows and m columns, A (l, m) = B ( l, m) × {Avlpf (l, m) / Bvlpf (l, m)}, and the pixel value in the first color signal A is similarly calculated at other pixel positions where the third color signal C exists. Vertical direction signal calculating means for calculating, and the predetermined pixel in 1 row and m columns in the first color signal A Average value calculating means for calculating a pixel value A (l, m) in the first color signal A of l rows and m columns from the average value of the adjacent pixels in the upper, lower, left and right positions, and the output of the edge determining means Based on the output from the horizontal signal calculation means, the output from the vertical direction signal calculation means, or the output from the average value calculation means, the pixel value A of the first color signal A in the predetermined pixel 1 row m column is selected. 1 and m) and obtaining the first color signal of the number of pixels in the image sensor, an image with reduced false colors and false contours can be obtained.
[0078]
According to the imaging device of the present invention, when the means for calculating the first color signal determines that the output of the edge determination means does not detect an edge component at a predetermined pixel 1 row and m column position, When the output of the average value calculation means is selected and it is determined that there is a correlation in the vertical direction, the output of the vertical direction signal calculation means is selected, and when it is determined that there is a correlation in the horizontal direction, the horizontal direction signal calculation means And obtaining a first color signal of the number of pixels in the image sensor, an image with reduced false colors and false contours can be obtained.
[0079]
Further, according to the imaging apparatus of the present invention, the means for calculating and restoring the second and third color signals is the position of the predetermined pixel l rows and m columns (l = 2i, m = 2j or l = 2i + 1, m = 2j + 1), for the output A from the first color signal calculating means, the value A1hlpf (l, m) through the horizontal low-pass filter and the value A1vlpf (l, through the vertical low-pass filter) m) is calculated, and the value B1hlpf (l, m) through the low-pass filter in the horizontal direction for the second color signal B and the value C1vlpf through the low-pass filter in the vertical direction for the third color signal C (L, m) (or the value B1vlpf (l, m) through the low-pass filter in the vertical direction for the second color signal B and the low-pass filter in the horizontal direction for the third color signal C Value C1hlpf l, m)), and the ratio of A1hlpf (l, m) to B1hlpf (l, m) (or the ratio of C1hlpf (l, m)), A1vlpf (l, m) and C1vlpf (l , M) (or the ratio to B1vlpf (l, m)) and the pixel value A (l, m at the predetermined pixel 1 row m column in the output A from the means for calculating the first color signal ), The pixel values B (l, m) and C (l, m) in the second color signal B and the third color signal C in l rows and m columns are represented by B (l, m) = A (l, m) × {B1hlpf (l, m) / A1hlpf (l, m)}, C (l, m) = A (l, m) × {C1vlpf (l, m) / A1vlpf (l, m)}, ( Or, B (l, m) = A (l, m) × {B1vlpf (l, m) / A1vlpf (l, m)}, C (l, m) = A (l, m) × {C 1hlpf (l, m) / A1hlpf (l, m)}) and a position of a predetermined pixel x row y column different from the position of the l row m column (x = 2i + 1, y = 2j or x = 2i, y = 2j + 1), the value A2hlpf (x, y) through the horizontal low-pass filter for the output A from the means for calculating the first color signal, from the signal calculating means A value B2hlpf (x, y) is calculated through a horizontal low-pass filter for the second color signal B at the output, and the ratio of A2hlpf (x, y) to B2hlpf (x, y) Based on the pixel value A (x, y) at the pixel x row y column at the output A from the calculation means, the second color signal B (x, y) at the position of x row y column is represented by B (x, y). ) = A (x, y) × {B2hlpf (x, y) / A hlpf (x, y)}, and the third color signal C is also perpendicular to the output A from the horizontal direction signal calculation means for calculating the C signal in the same manner and the means for calculating the first color signal. A value A2vlpf (x, y) through the direction low-pass filter and a value B2vlpf (x, y) through the low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means X row y by the ratio of A2vlpf (x, y) and B2vlpf (x, y) and the pixel value A (x, y) in the pixel x row y column at the output A from the first calculating means. The second color signal B (x, y) at the column position is calculated by B (x, y) = A (x, y) × {B2vlpf (x, y) / A2vlpf (x, y)} In the third color signal C as well, a vertical direction signal calculator for calculating the C signal in the same manner. And an average value calculating means for calculating an average value of diagonally adjacent pixels at positions of predetermined pixels x rows and y columns in the second and third color signals in the output from the signal calculating means, Based on the output of the edge determination means, the second and third pixels in the predetermined pixel x row y column are selected from the outputs from the horizontal direction signal calculation means, vertical direction signal calculation means, and average value calculation means. By obtaining the second color signal and the second color signal of the number of pixels in the image sensor, an image with reduced false colors and false contours can be obtained.
[0080]
Further, according to the imaging apparatus of the present invention, the means for calculating the second and third color signals determines that the output of the edge determination means does not detect an edge component at the position of the predetermined pixel x row y column. If so, the output of the average value calculation means is selected. If it is determined that there is a correlation in the vertical direction, the output of the vertical direction signal calculation means is selected. If it is determined that there is a correlation in the horizontal direction, the horizontal By selecting the output of the direction signal calculating means and obtaining the second and third color signals of the number of pixels in the image sensor, an image with reduced false colors and false contours can be obtained.
[Brief description of the drawings]
FIG. 1 is a diagram showing an example of an arrangement of color filters of an image sensor according to Embodiment 1 of the present invention.
FIG. 2 is a block diagram illustrating an example of a configuration of an imaging apparatus according to Embodiment 1 of the present invention.
FIG. 3 is a block diagram showing an example of the configuration of first edge determination means 5 and G component restoration means 6 in the imaging apparatus according to Embodiment 1 of the present invention.
4 is a block diagram showing an example of the configuration of second edge determination means 8 and second RB component restoration means 9 in the imaging apparatus according to Embodiment 1 of the present invention. FIG.
FIG. 5 is a diagram illustrating G signal pixels for explaining an operation of restoring a G signal in the imaging apparatus according to Embodiment 1 of the present invention;
6 is a diagram showing pixels of R and B signals for explaining an operation of restoring a G signal in the imaging apparatus according to Embodiment 1 of the present invention. FIG.
7 is a diagram showing pixels of R and B signals for explaining the operation of the first RB component restoration device 7 in the imaging apparatus according to Embodiment 1 of the present invention; FIG.
FIG. 8 is a diagram showing R and B signal pixels for explaining operations of second edge determination means 8 and second RB component restoration means 9 in the imaging apparatus according to Embodiment 1 of the present invention;
FIG. 9 is a diagram showing false color levels at horizontal resolution in simulation images processed by the imaging apparatus according to Embodiment 1 of the present invention and a conventional apparatus.
FIG. 10 is a diagram illustrating false color levels at a vertical resolution in a simulation image processed by the imaging apparatus according to Embodiment 1 of the present invention and a conventional apparatus.
FIG. 11 is a diagram showing the level of a luminance signal at a horizontal resolution in a simulation image processed by the imaging apparatus according to Embodiment 1 of the present invention.
FIG. 12 is a diagram illustrating the level of a luminance signal at a horizontal resolution in a simulation image processed by a conventional apparatus.
FIG. 13 is a diagram showing the level of a luminance signal at a vertical resolution in a simulation image processed by the imaging apparatus according to Embodiment 1 of the present invention.
FIG. 14 is a diagram showing the level of a luminance signal at a vertical resolution in a simulation image processed by a conventional apparatus.
FIG. 15 is a diagram showing an example of another color filter array of the image sensor according to the first embodiment of the present invention.
FIG. 16 is a diagram showing an example of another color filter array of the image sensor according to the first embodiment of the present invention.
FIG. 17 is a block diagram showing an example of the configuration of an imaging apparatus according to Embodiment 2 of the present invention.
FIG. 18 is a diagram illustrating pixels of a G signal for explaining the operation of the G component edge determination unit 41 in the imaging apparatus according to Embodiment 2 of the present invention.
FIG. 19 is a diagram illustrating an example of a color filter array of an imaging element in a conventional imaging device.
FIG. 20 is a block diagram illustrating an example of a configuration of a conventional imaging device.
FIG. 21 is a diagram illustrating pixels of each signal for explaining the operation of a conventional imaging device.
FIG. 22 is a block diagram illustrating an example of another configuration of a conventional imaging device.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Image pick-up element, 2 A / D converter, 3 frame memory, 4 separation means, 5 1st edge determination means, 6 G component restoration means, 7 1st RB component restoration means, 8 2nd edge determination means, 9 Second RB component restoration means, 11, 30, 42 horizontal edge detection means, 12, 31, 43 Vertical edge detection means, 13 determination means, 14 average value calculation means, 15-17 horizontal low-pass filter, 18, 19 computing means, 20-22 vertical low-pass filter, 23, 24 computing means, 25 switching means, 32 judging means, 33 R average value computing means, 34 R horizontal direction computing means, 35 R vertical direction computing means, 36 R switching Means, 37 B average value calculating means, 38 B horizontal direction calculating means, 39 B vertical direction calculating means, 40 B switching means, 41 G component edge determining means, 44 Constant means.

Claims (7)

The first color filter having the spectral sensitivity characteristic with respect to the first color signal and the second color filter having the spectral sensitivity characteristic with respect to the second color signal are arranged in the first row in four pixels in the horizontal and vertical 2 rows and 2 columns. In the second row, a third color filter having spectral sensitivity characteristics for the third color signal is arranged in the same column as the pixel position where the first color filter in the first row is arranged. A first color filter having spectral sensitivity characteristics with respect to the first color signal is arranged in the same column as the second color filter, and the four-pixel color filter in the second row and the second column is sequentially repeated in the vertical and horizontal directions. In an imaging apparatus including an arrayed imaging device,
First edge determination means for determining an edge component at a predetermined pixel position based on a first color signal around the predetermined pixel position of the second or third color signal at the position of the second or third color filter. When,
First, second, and third horizontal low-pass filters that generate horizontal low-frequency signal components of the first, second, and third color signals;
First, second, and third vertical low-pass filters that generate vertical low frequency signal components of the first, second, and third color signals;
The low-frequency signal component in the horizontal direction of the first and second color signals at the second color filter position obtained from the first and second horizontal low-pass filters, and the first at the second color filter position. First calculation means for calculating the first color signal at the second color filter position from the two color signals;
The low frequency signal component in the vertical direction of the first and second color signals at the second color filter position obtained from the first and second vertical low-pass filters, and the first low frequency signal component at the second color filter position. Second calculation means for calculating the first color signal at the second color filter position from the two color signals;
The low-frequency signal component in the horizontal direction of the first and third color signals at the third color filter position obtained from the first and third horizontal low-pass filters, and the first low frequency signal component at the third color filter position. 3rd calculation means for calculating the 1st color signal in the 3rd color filter position from 3 color signals,
The low-frequency signal component in the vertical direction of the first and third color signals at the third color filter position obtained from the first and third vertical low-pass filters, and the first low-frequency signal component at the third color filter position. A fourth calculation means for calculating the first color signal at the third color filter position from the three color signals;
Fifth arithmetic means for calculating an average value of the surrounding first color signals;
A first color signal including: a first color signal at a first color filter position; and a switching unit that switches an output signal of the first to fifth arithmetic units based on an output of the first edge determination unit. Signal restoration means;
At a predetermined position of the first of the first color signal of the color filter position, the low-frequency component of the second color signals at a predetermined position of the first color signal, and the low frequency component of the first color signal, said A second color signal is generated from the first color signal at the first color filter position output from the first color signal restoration means , and the third color signal at a predetermined position of the first color signal is generated. A third color signal is generated from the low frequency component, the low frequency component of the first signal, and the first color signal at the first color filter position output from the first color signal restoration means . Two color signal restoration means;
Based on the first color signal around the predetermined pixel position of the second or third color signal at the position of the second or third color filter output from the first color signal restoration means, at the predetermined pixel position Second edge determination means for determining an edge component;
The first color signal output from the first color signal restoration means and the first color signal restoration means output from the first color signal restoration means at a predetermined position of the second color signal at the second color filter position . The second color signal at a predetermined position of the second color signal is determined from the horizontal low frequency component of the first color signal and the horizontal low frequency component of the third color signal output from the second color signal restoration means . First horizontal direction calculating means for calculating a color signal of 3;
The first color signal output from the first color signal restoration means and the first color signal restoration means output from the first color signal restoration means at a predetermined position of the second color signal at the second color filter position . From the low-frequency component in the vertical direction of the first color signal and the low-frequency component in the vertical direction of the third color signal output from the second color signal restoration means, First vertical direction calculation means for calculating the color signal of 3;
First average value calculating means for calculating an average value of the surrounding third color signals output from the second color signal restoring means ;
The first color signal output from the first color signal restoration unit and the first color signal output from the first color signal restoration unit at a predetermined position of the third color signal at the third color filter position . The third color signal at a predetermined position from the horizontal low frequency component of the first color signal and the horizontal low frequency component of the second color signal output from the second color signal restoration means . Second horizontal direction calculating means for calculating the color signal of 2;
The first color signal output from the first color signal restoration unit and the first color signal output from the first color signal restoration unit at a predetermined position of the third color signal at the third color filter position . From the low-frequency component in the vertical direction of the first color signal and the low-frequency component in the vertical direction of the second color signal output from the second color signal restoration means, Second vertical direction calculation means for calculating the color signal of 2;
Second average value calculating means for calculating an average value of the surrounding second color signals output from the second color signal restoring means ;
The outputs of the first horizontal direction calculating means, the first vertical direction calculating means, and the first average value calculating means are switched based on the output of the second edge determining means, and the second horizontal direction calculating means A third color signal restoring means comprising: a switching means for switching the output of the second vertical direction calculating means and the second average value calculating means based on the output of the second edge determining means; An imaging apparatus comprising: The first edge determining means is
Horizontal edge detection means for calculating a difference between the first color signals of left and right adjacent pixels at a predetermined pixel position of the second or third color signal to detect a horizontal edge component;
Vertical edge detection means for detecting a vertical edge component by calculating a difference between first color signals of upper and lower adjacent pixels at a predetermined pixel position of the second or third color signal;
The imaging apparatus according to claim 1, further comprising a determination unit that determines a horizontal or vertical edge component of the predetermined pixel based on outputs from the horizontal edge detection unit and the vertical edge detection unit.   When the determination means in the first edge determination means has an output from the horizontal edge detection means or an output from the vertical edge detection means larger than a predetermined value, an edge component is added to the peripheral pixels of the predetermined pixel. Furthermore, if the output from the horizontal edge detection means is larger than the output from the vertical edge detection means, there is a correlation in the vertical direction, and the output from the horizontal edge detection means is the vertical edge detection means. If the output from the horizontal direction edge detection means and the vertical direction edge detection means are both smaller than a predetermined value, it is determined that no edge component is detected. The imaging apparatus according to claim 1, wherein: In the first color signal restoration means,
The first horizontal low-pass filter outputs a low-frequency signal component Ahlpf (2i, 2j + 1) in the horizontal direction of the first color signal at the second color filter position (2i, 2j + 1) (i and j are integers). Calculate
The second horizontal low-pass filter calculates a horizontal low-frequency signal component Bhlpf (2i, 2j + 1) of the second color signal at the second color filter position (2i, 2j + 1),
The first vertical low-pass filter calculates a low frequency signal component Avlpf (2i, 2j + 1) in the vertical direction of the first color signal at the second color filter position (2i, 2j + 1),
The second vertical low-pass filter calculates a low-frequency signal component Bvlpf (2i, 2j + 1) in the vertical direction of the second color signal at the second color filter position (2i, 2j + 1),
The first calculating means outputs the first and second horizontal low-pass filter outputs and the second color signals B (2i, 2j + 1) at the (2i, 2j + 1) position to the first at the (2i, 2j + 1) position. The color signal Ah (2i, 2j + 1) of B (2i, 2j + 1) × (Ahlpf (2i, 2j + 1) / Bhlpf (2i, 2j + 1))
As
The second computing means outputs the first and second vertical low-pass filter outputs and the second color signal B (2i, 2j + 1) at the (2i, 2j + 1) position to the first at the (2i, 2j + 1) position. The color signal Av (2i, 2j + 1) is calculated as B (2i, 2j + 1) × (Avlpf (2i, 2j + 1) / Bvlpf (2i, 2j + 1))
The first horizontal low-pass filter further includes a horizontal low-frequency signal component Ahlpf (2i + 1, 2j) of the first color signal at the third color filter position (2i + 1, 2j) (i and j are integers). To calculate
The third horizontal low-pass filter calculates a horizontal low-frequency signal component Chlpf (2i + 1, 2j) of the third color signal at the third color filter position (2i + 1, 2j),
The first vertical low-pass filter further calculates a low-frequency signal component Avlpf (2i + 1, 2j) in the vertical direction of the first color signal at the third color filter position (2i + 1, 2j),
The third vertical low-pass filter calculates a low-frequency signal component Cvlpf (2i + 1, 2j) in the vertical direction of the third color signal at the third color filter position (2i + 1, 2j);
The third calculation means outputs the first and third horizontal low-pass filters and the first color signal at the (2i + 1, 2j) position from the third color signal C (2i + 1, 2j) at the (2i + 1, 2j) position. Color signal Ah (2i + 1, 2j) is calculated as C (2i + 1, 2j) × (Ahlpf (2i + 1, 2j) / Chlpf (2i + 1, 2j)),
The fourth computing means outputs the first color signal Av (2i, 2j, 1) from the outputs of the first and third vertical low-pass filters and the third color signal C (2i + 1, 2j) at the (2i + 1, 2j) position. 2j + 1) is calculated as C (2i + 1, 2j) × (Avlpf (2i + 1, 2j) / Cvlpf (2i + 1, 2j)),
The fifth computing means calculates an average value of the first, upper, lower, left and right color signals at the positions (2i, 2j + 1) and (2i + 1, 2j),
Based on the output of the first edge determination means, the switching means outputs either the output of the first calculation means, the output of the second calculation means, or the output of the fifth calculation means (2i, 2j + 1) as the first color signal, and the output of the third calculation means or the output of the fourth calculation means or the output of the sixth calculation means is set at the position (2i + 1, 2j). The imaging device according to claim 1, wherein the imaging device is a first color signal. The switching means of the first color signal restoration means is
When the output of the first edge determination means determines that no edge component is detected at the second color filter position (2i, 2j + 1), the output of the fifth calculation means is selected, and the horizontal direction is correlated. If it is determined that there is a correlation, the output of the first calculation means is selected. If it is determined that there is a correlation in the vertical direction, the output of the second calculation means is selected, and the third color filter position (2i + 1, 2 If it is determined in 2j) that no edge component is detected, the output of the fifth arithmetic means is selected, and if it is determined that there is a correlation in the horizontal direction, the output of the third arithmetic means is selected, and the vertical The imaging apparatus according to claim 4, wherein when it is determined that there is a correlation in the direction, an output of the fourth calculation means is selected. In the third color signal restoration means,
The first horizontal direction arithmetic means calculates the horizontal color of the first color signal obtained from the output value of the first color signal restoration means at the second color filter position (2i, 2j + 1) (i and j are integers). Low-frequency signal component ahlpf (2i, 2j + 1) in the direction, horizontal low-frequency signal component chlpf (2i, 2j + 1) of the third color signal obtained from the output value of the second color signal restoration means, From the first color signal a (2i, 2j + 1) at the position (2i, 2j + 1) restored by the color signal restoration means of
A third color signal Ch (2i, 2j + 1) at the position (2i, 2j + 1) is calculated as a (2i, 2j + 1) × (chlpf (2i, 2j + 1) / ahhlpf (2i, 2j + 1))
The first vertical direction calculation means is a low frequency signal component avlpf in the vertical direction of the first color signal obtained from the output value of the first color signal restoration means at the second color filter position (2i, 2j + 1). (2i, 2j + 1), the low-frequency signal component cvlpf (2i, 2j + 1) in the vertical direction of the third color signal obtained from the output value of the second color signal restoration means, and restoration by the first color signal restoration means From the first color signal a (2i, 2j + 1) at the (2i, 2j + 1) position, the third color signal Cv (2i, 2j + 1) at the (2i, 2j + 1) position is a (2i, 2j + 1) × ( cvlpf (2i, 2j + 1) / avlpf (2i, 2j + 1))
As
The second horizontal direction calculation means is the horizontal of the first color signal obtained from the output value of the first color signal restoration means at the third color filter position (2i + 1, 2j) (i and j are integers). Low-frequency signal component ahlpf (2i + 1, 2j) in the direction, horizontal low-frequency signal component bhlpf (2i + 1, 2j) of the second color signal obtained from the output value of the second color signal restoration means, From the first color signal a (2i + 1, 2j) at the position (2i + 1, 2j) restored by the color signal restoration means, the second color signal Bh (2i + 1, 2j) at the position (2i + 1, 2j) is a (2i + 1, 2j) x (bhlpf (2i + 1, 2j) / ahhlpf (2i + 1, 2j))
As
The second vertical direction calculation means is a low-frequency signal component avlpf in the vertical direction of the first color signal obtained from the output value of the first color signal restoration means at the third color filter position (2i + 1, 2j). (2i + 1, 2j), the low-frequency signal component bvlpf ((2i + 1, 2j)) in the vertical direction of the second color signal obtained from the output value of the second color signal restoration means, and the first color signal restoration means From the restored first color signal a (2i + 1, 2j) at the (2i + 1, 2j) position, the second color signal Bv (2i + 1, 2j) at the (2i + 1, 2j) position is a (2i + 1, 2j) × (Bvlpf (2i + 1, 2j) / avlpf (2i + 1, 2j))
As
The first average value calculating means calculates an average value of the third color signal diagonally up, down, left and right at the position (2i, 2j + 1),
The second average value calculating means calculates an average value of the second color signals diagonally up, down, left and right at the position of (2i + 1, 2j),
Based on the output of the second edge determining means, the switching means is either the output of the first horizontal direction calculating means, the output of the first vertical direction calculating means or the output of the first average value calculating means. Is the third color signal at the position (2i, 2j + 1), and the output of the second horizontal direction calculation means, the output of the second vertical direction calculation means or the output of the second average value calculation means The imaging apparatus according to claim 1, wherein any one of the second color signals is at a position of (2i + 1, 2j).   When the switching unit of the third color signal restoration unit determines that the output of the second edge determination unit does not detect an edge component at the second color filter position (2i, 2j + 1), the first color signal restoration unit When the output of the average value calculation means is selected and it is determined that there is a correlation in the horizontal direction, the output of the first horizontal direction calculation means is selected, and when it is determined that there is a correlation in the vertical direction, the first If the output of the vertical direction calculation means is selected and it is determined that the edge component is not detected at the third color filter position (2i + 1, 2j), the output of the second average value calculation means is selected and the horizontal direction If it is determined that there is a correlation, the output of the second horizontal direction calculation means is selected, and if it is determined that there is a correlation in the vertical direction, the output of the second vertical direction calculation means is selected. Claim 6 Placing the imaging device.

Images