JP3893489B2 - Signal processing apparatus and signal processing method - Google Patents

Description

に示すようにこれらのブロック間の位置に対応するブロックBL2 を生成する。
【００４６】
このように構成要素を選択して取り出す必要があることから、ブロック生成部12D には、図示しないが各ブロックを一旦格納するメモリとブロックを水平方向に隣接させて並べて列方向に２画素を一度に間引く選択を行う選択部が備えられる。このメモリには２ブロックの格納によって形式的に２行４列の要素が含まれていることになる。選択部は、これらの要素の内、両端側に位置する２行１列をそれぞれ間引く。換言すると、中央に位置する２行２列分の要素だけを抽出すると言える。ブロック生成部12D は、この生成したブロックのデータをブロック対応抽出部12E に出力する。
【００４７】
ブロック対応抽出部12E は、画素位置を重視しながら画素データを抽出する。このとき、ブロック対応抽出部12E は、この新たに生成されたブロックに対するデータ抽出を行う。この場合、サンプルホールド回路は、たとえば生成した画素数と同じく４つ設けてデータ抽出を行うことになる。データ抽出のタイミングは、図示しないがデータ抽出部12C のブロックにおけるサンプリングタイミングに対して3/4 周期だけブロック対応抽出部12E のブロックにおけるサンプリングタイミングがずれた関係になっている。
【００４８】
代表値算出部12F は、ブロック内の任意に一つの画素をブロックの代表画素として規定し、この規定に基づいてこの画素のRGB の画素値を算出する。この算出を行うため、代表値算出部12F は、少なくとも加算器および1/2 を係数に乗算する乗算器あるいは加算器および除算器からなる演算回路で構成される。代表値算出部12F は、ブロック分割して残ったブロックに対する演算も新たに生成されたブロックに対する演算も同じ代表値を算出する演算方法を用いる。ただし、入力される代表値の位置が異なることになる。この演算回路を用いた演算は後段で詳述する。ここで得られたRGB の値は出力形式選択部124cおよびYC変換部122cに供給される。
【００４９】
図４に示すYC変換部122cは、輝度信号Y と色差信号C_b, C_rを算出する。この算出には、図示しないがRGB に対する３つの係数が格納された乗算部とこれら乗算結果を加算する加算器がある。RGB に対する３つの係数が格納された乗算部の結果を加算して輝度信号Y が生成される。また、色差信号C_b, C_rは、それぞれ算出用に加算器を一つずつ備える。色差信号C_bの場合、加算器は、一方の側から代表値算出部12F で算出されたB の画素値を加算入力し、他方の側から上述したように算出された輝度信号Y を減算入力し、その出力を求める。同様に色差信号C_rも算出する。
【００５０】
このように代表値算出部12F およびYC変換部122cを備えていることによってRGB 出力とYC出力に対応できる。しかしながら、図１の表示部11i には、いずれか一方の出力形式で表示させることになる。したがって、ユーザがデータ入力部11h を介してシステム制御部13にどのような表示をさせるかモード等の設定が行われる。システム制御部13は、このユーザの意向を反映した制御信号を出力選択部124cに供給する。図４の出力選択部124cは、システム制御部13の制御によって所望の出力形式を選択して出力する。出力選択部124cは、代表値算出部12F だけしか配設されていないとき当然不要になる。YC変換の処理には代表値算出部12F およびYC変換部122cが必要だからである。
【００５１】
データ評価制御部12d には、図６に示すように、エッジ検出部120d、帯域指示部122d、輝度信号積算部124d、色信号積算部126dおよび比較制御部128dが備えられている。さらに各部を説明する。
【００５２】
エッジ検出部120dには、フィルタバンク部、ラインメモリ、乗算器および加算器が備えられている。具体的な構成を図７に示し、後段で説明する。帯域指示部122dは、図示しないがたとえば、露出等によって得られたデータから画像の種類がデータとして供給される。画像の種類とは、たとえば順光、過順光等の撮像状況を表すような情報である。この情報が供給されたとき、帯域指示部122dは、情報に応じた帯域指示信号をエッジ検出部120dに出力している。
【００５３】
ここで、エッジ検出部120dの構成について説明する。エッジ検出部120dは、帯域指示部122dからの帯域指示信号に応じた輝度信号データだけを通過させるようにフィルタバンク部1200がある（図７(a), (b)を参照）。フィルタバンク部1200には、信号の通過帯域の異なる複数種類のディジタルフィルタが設けられている。これにより、ある特定の周波数範囲の成分だけについて以後の信号処理が行われることになる。
【００５４】
また、画像の垂直方向に対して行うと効果的であるから、エッジ検出処理にはラインメモリが用いられる。エッジ検出を単に２ラインで行う場合とより広い範囲にわたって行う３ラインで行う場合を例示する。ライン数の違いによってエッジ検出の構成がそれぞれ図７(a), (b)に示すようになる。すなわち、フィルタバンク部1200を経たデータが図７(a),(b) ではラインメモリ1202と加算器1204の一端側に加算供給されている。図７(a) ではラインメモリ1202を介して１ライン分遅延させられたデータy_n-1が加算器1204の他端側に減算供給される。これにより、加算器1204の出力、すなわち検出データe(n)は、y_n-y_n-1 となる。
【００５５】
また、図７(b) では、上述した図７(a) の構成の他に乗算器1208および加算器1210が用いられている。ラインメモリ1202は、１ライン分遅延させられたデータをラインメモリ1206および加算器1204の他端側に供給する。加算器1204の他端側には加算入力する。ここで、ラインメモリ1202の出力をy_nとすると、ラインメモリ1206の出力とフィルタバンク部1200からの出力は、それぞれ２ライン分遅延させられたy_n+1, 全く遅延を受けていないy_n-1という関係にある。加算器1204は、それぞれ供給されるデータを加算し、係数1/2 の乗算器1208に出力する。乗算器1208は、乗算結果を加算器1210の一端側に減算入力する。また、加算器1210の他端側には、ラインメモリ1206からの出力が加算入力される。このようにデータが各端部に供給されることにより、加算器1210の出力、すなわち検出データe(n)は、y_n-(y_n-1+y_n+1)/2となる。このようにして得られたエッジ検出のデータが、図６の輝度信号積算部124dに供給される。
【００５６】
輝度信号積算部124dには、絶対値変換部1212および積算部1214が備えられている。絶対値変換部1212は、供給されたエッジ検出のデータに対してこのデータが正の値となるように絶対値をとる変換を行う。絶対値変換されたデータは積算部1214に供給される。積算部1214は、単に供給されるデータを積算して比較制御部128dに出力する。積算部1214の積算は、画像全体のライン検出が終了するまで繰り返される。このような処理は、輝度信号積算部124dだけでなく、色信号積算部126dでも同様に行われる。色信号積算部126dは、色差信号(R-Y), (B-Y)のデータに対してそれぞれ絶対値変換部1216, 1220と積算部1218, 1222を含んでいる。色信号積算部126dも画像全体に対して色差信号(R-Y), (B-Y)のデータを積算して比較制御部128dに出力する。
【００５７】
比較制御部128dには、スレッショルドレベル格納部1224、比較部1226, 1228, 1230、評価制御部1232が備えられている。スレッショルドレベル格納部1224には、輝度信号Y のデータ、および色差信号(R-Y), (B-Y)のデータに対するスレッショルドレベルが記憶されている。比較制御部128dは、図６に図示していないが図４に示すように、システム制御部13により制御されている。この制御によりスレッショルドレベル格納部1224は、比較部1226, 1228, および1230にそれぞれ画像に対する適切なスレッショルドレベルを供給している。比較部1226, 1228, および1230は、スレッショルドレベルと対応する積算されデータとのレベル比較を行う。比較部1226, 1228, および1230は、比較結果を評価制御部1232に供給する。この供給は、評価制御部1232にエッジ検出部120d、輝度信号積算部124dおよび色信号積算部126dで検出された周波数成分、エッジ検出成分および色信号成分の情報が比較部1226, 1228, および1230にそれぞれ供給されていることを意味している。
【００５８】
評価制御部1232は、各比較部1226, 1228, および1230からの比較結果に基づいて間引きして得られた画像の評価を行い、評価結果に応じて各種の制御信号を出力する。評価については動作で詳述する。評価制御部1232は、各種の制御信号としてデータ評価制御信号、およびアパーチャ制御信号等を出力するとともに、評価データをシステム制御部13に出力する。このように評価データを供給することにより、システム制御部13が各部を制御するようにしてもよい。
【００５９】
また、図４の画像処理部12e には、たとえば図８に示すようにアパーチャ調整部120eおよび色ゲイン制御部122eが備えられている。アパーチャ調整部120eは、輝度信号(Y) のデータに対してデータ評価制御部12d からの制御信号に応じて信号調整の一つであるアパーチャー補正により、たとえば、輪郭強調やコアリング等の効果が得られるように補正をかける。また、色ゲイン制御部122eもデータ評価制御部12d からの制御信号に応じて色信号(C) のレベルを調整してたとえば彩度の向上等を図っている。
【００６０】
システム制御部13は、具体的に図示しないが電子スチルタメラ10を制御するシステムコントローラ、タイミング信号発生部およびアドレス制御部等を含んでいる。システム制御部13は、図１に示すように、CCD イメージセンサ部11c 、前置処理部11d 、A/D 変換部11e 、信号処理部12およびデータ圧縮部11f を制御している。特に、信号処理部12では、図２のタイミング信号生成部12a や切換スイッチSW1 を制御している。
【００６１】
駆動信号生成部14は、システム制御部13からの駆動制御信号に応じてCCD イメージ部11c の撮像および信号の読出しといった駆動タイミングを供給している。表示モードおよび記録モードにおけるデータ間引き処理において、この駆動信号生成部14とタイミング信号生成部12a の両方の役割を適切に分けて動作させると、処理の効率を向上させながら、装置の構成を簡素化することができる。
【００６２】
データ入力部11h は、ユーザの操作するモード選択、シャッタ・レリーズボタン（いずれも図示せず）等の設定や押圧指示の情報をモード選択信号11m としてシステム制御部13に送っている。ユーザの操作するモードには、スチル撮影、ムービー撮影、RGB 出力およびYC出力等があって、これらのモードの中から所望のモードを電子スチルカメラ10に対して選択的に行えるようにしている。
【００６３】
また、電子スチルカメラ10は、記録モードで使用する本体部11の構成を簡単に説明する。データ圧縮部11f は、信号処理部12から供給される輝度データY, 色差データ(B-Y), (R-Y)のビット数を低減する圧縮処理を行っている。データ圧縮方法には、たとえばハフマン符号化や差分PCM 等を用いている。これによりデータ圧縮部11f は、データ量を抑える処理を施して入出力インターフェース部11g に出力する。
【００６４】
メモリカード部20には、コネクタ30、入出力インターフェース部21を介してメモリカード22にデータが供給される。逆に、メモリカード22に格納しているデータの読出しは、入出力インターフェース部21、コネクタ30を介してカメラ本体11に供給する順で行われる。
【００６５】
また、前述したデータ評価制御部12d の構成をより簡単なものにする変形例の構成について図９を参照しながら説明する。垂直方向に間引き処理を行う場合、輝度データを垂直方向に見たときに輪郭の境界に偽色が発生しやすい。このことを考慮すると、データ評価制御部12d は、エッジ検出部120d、絶対値変換部1212, 1216、スレッショルドレベル格納部1224、比較部1226、積算制御部1234、積算部1218および評価制御部1232で済ますことができる。この構成は、積算制御部1234以外、前述のデータ評価制御部12d の構成要素を抽出して用いている。積算制御部1234は、エッジ成分がスレッショルド値よりも大きい部分、すなわちその画像領域だけに対して色差信号の積算を行う制御信号を積算部1218に出力する。
【００６６】
この場合、色差信号は、輝度信号Y の半分に間引いて(R-Y)/(B-Y) を点順次で供給される。このため、色差信号に対する処理構成が図６の構成に比べて半分に削減することができる。
【００６７】
次に電子スチルカメラ10の動作について説明する。色フィルタアレイ11b がCCD イメージセンサ11c の各画素であるセルに対応して前面に配されている。色フィルタアレイ11b には、基本的なベイヤパターンが適用されている（図10(a) や図12(a) を参照）。図10の左端に書かれた数字は、色フィルタアレイ11b のライン番号を示している。また、セルに対応した位置は、行列表現を用いて各色R, G, B の添字の数字で表している。
【００６８】
CCD イメージセンサ11c から読み出したデータに対して、信号処理部12は、ラインの間引き処理を行う。すなわち、CCD イメージセンサ11c は、通常の通りラインを間引くことなく撮像信号をすべて出力している。このラインの間引き処理には、隣接した２ラインをライン毎に読み出しながら、垂直方向にラインを間引くライン隣接読出しで得られたデータを用いるライン隣接間引き処理と、所定の関係を保ってサンプリングされたラインを用いて垂直方向にラインを間引くライン選択間引き処理がある。ここで、所定の関係とは、三原色R, G, B の内、一方のサンプリングしたラインが含む色と異なる色を含むラインを次のラインとして選択する関係である。この２ラインによって三原色R, G, B が揃うことになる。
【００６９】
まず、ライン隣接間引き処理について説明する。システム制御部13の制御によって信号処理部12内のタイミング信号生成部12a が動作する。また、タイミング信号生成部12a は、データ評価制御部12d からのデータ評価制御信号によっても制御されている。タイミング信号生成部12a は、図２に示すように、システム制御部13からの垂直選択信号Vsと水平選択信号Hsによって出力するタイミング信号を選択される。垂直選択信号生成部12V は、ライン隣接間引き処理とライン選択間引き処理に対応した垂直タイミング信号がライン隣接信号生成部120aとライン選択信号生成部122aがそれぞれ出力される。ライン隣接信号生成部120aとライン選択信号生成部122aは、データ評価制御信号のレベルでいずれか一方が選択される。たとえば、データ評価制御信号は、レベル"H" でライン隣接信号生成部120a、レベル"L" でライン選択信号生成部122aをイネーブルにする。タイミング信号生成部12a は、垂直タイミング信号Tvと水平タイミング信号Thをそれぞれ図４のラインデータサンプリング部12b とライン生成部12c に出力する。
【００７０】
ラインデータサンプリング部12b は、入力データをシステム制御部13の制御に応じてラインサンプリング部120bに供給される垂直タイミング信号V1でラインをサンプリングする。このとき、ラインサンプリング部120bには、２ラインを同時読出し的にサンプリングするように垂直タイミング信号V1が供給されている。この垂直タイミング信号V1に応じてサンプリングすると、図10(a) の場合、一点鎖線40で囲まれた領域、すなわちライン0, 1, 8, 9がサンプリングされる。これらのラインは、ライン0, 1, 8, 9が示すように、隣接した２ラインずつサンプリングされている。
【００７１】
このサンプリングしたデータは、合成ラインサンプリング部122bおよび水平画素処理部120cに供給される。合成ラインサンプリング部122bは、供給される垂直タイミング信号V2により、２ライン同時的なサンプリングをして得られたデータの内、一方のラインを選択する。この選択により、次に得られる２ラインの内、選択される、もう一方のラインが前述した所定の関係によって決まる。合成ラインサンプリング部122bは、垂直タイミング信号V2によりこのように読み出したラインから新たな２ラインを合成する。図10(a) の一点鎖線42で囲まれたラインが合成ラインのライン要素である。ラインサンプリング部120bでは、このようなラインのサンプリングにより隣接ラインの組を得て、さらに隣接ラインの組からラインを組み合わせて新たな隣接ラインの組を合成ラインとして求めて水平画素処理部120cに出力する。これにより、破線44で示すライン2 〜7 は間引かれる。この場合、入力データを合成ラインサンプリング部122bに直接的に入力する必要がないので、合成ラインサンプリング部122bは、ラインサンプリング部120bからだけのデータを入力し、直接A/D 変換部11e からのデータ入力を禁止してもよい。
【００７２】
水平画素処理部120cには、図10(b) に示すように、２ライン同時読出し的に３組のラインが供給される。一点鎖線で区切られた第１および第３のラインの組は、単に同時読出し的にラインサンプリング部120bで得られたラインを表し、真ん中の第２のラインの組は合成ラインサンプリング部122bで得られたラインを表している。第２のラインの組は間引かれたライン4, 5に相当するデータとして用いる。水平画素処理部120cでは、図10(b) に示すラインからライン0, 4, 8 あるいはライン1, 5, 9 を生成する。ライン0, 4, 8 を生成するとき、代表値の位置を記号P と行列表現を用いて添字の数字で表して、図10(c) に示す位置でのデータ算出が行われる。
【００７３】
このデータ算出において最初に、図５のブロック分割部12A でシステム制御部13からの制御信号に応じて供給される２ラインを２画素ずつに分ける。すなわち、ブロックは、計４画素単位で扱われる。この処理を行った後、ブロック単位での間引きがシステム制御部13からの制御信号に応じてブロック間引き部12B で行われる。１ブロック毎に間引きされずに残ったブロックがデータ抽出部12C およびブロック生成部12D に供給される。データ抽出部12C は、供給されるブロック内の画素として得られたデータをシステム制御部13の制御信号に応じて水平タイミングで読み出す。読み出したデータは、代表値算出部12F に供給される。
【００７４】
また、ブロック生成部12D では、間引きされずに残ったブロックの画素を用いて間びいたブロックと異なる新たなブロックを生成する。基本的に生成するブロックの構成要素は、残ったブロックと同じ構成で、計４画素である。この新たに生成されるブロックは、間引いたブロックを挟んで、たとえばブロック（P₀₀, P₀₁, P₁₀, P₁₁）とブロック（P₀₄, P₀₅, P₁₄, P₁₅）の２つ残したブロックから生成する。生成されるブロックは、残したブロックの各画素位置を用いて表すと、要素が（P₀₁, P₀₄, P₁₁, P₁₄）で構成される。同様に、要素は（P₀₅, P₀₈, P₁₅, P₁₈）となる。第２および第３の組においても前述した一連の処理を施す。このようにして再ブロック化されたブロック内の要素は、ブロック対応抽出部12E でシステム制御部13の制御信号に応じて水平タイミング信号のタイミングで要素となるデータがサンプリングされる。このようにして得られたデータが、図11(a) に示すように代表値算出部12F に供給される。
【００７５】
ここで、図５に示すデータに対する通常の間引き抽出を行うブロック間引き部12B およびデータ抽出部12C の構成と残したデータによる補間抽出を行うブロック生成部12D およびブロック対応抽出部12E の構成とを区別して動作させることができるようにシステム制御部13から供給される制御信号13A をイネーブル信号としてその信号レベルをたとえば、レベル"L" とレベル"H" により動作選択できるようにしている。
【００７６】
代表値算出部12F は、演算処理を供給されたブロックに対して行って代表位置における三原色RGB それぞれの値を算出する。たとえば、画素位置P₀₀ の三原色RGB を代表値PR₀₀, PG₀₀, PB₀₀を算出する。この算出は、代表値PR₀₀, PG₀₀, PB₀₀に対する各画素位置での色の画素値の代入および画素値を用いた演算を次に示すように式(1)
【００７７】
【数１】
PR₀₀=P₀₁
PG₀₀=(P₀₀+P₁₁)/2 ・・・(1)
PB₀₀=P₁₀
で行う。具体的に式(1) に画素値を入れて演算すると、
【００７８】
【数２】
PR₀₀=R₀₁
PG₀₀=(G₀₀+G₁₁)/2 ・・・(2)
PB₀₀=B₁₀
が得られる。PR₀₀, PG₀₀, PB₀₀は、図11(b) のR₀₀, G₀₀, B₀₀ にそれぞれ対応している。このようにして、代表位置P₀₀ だけでなく、間引いて残ったブロックに対する代表値の算出を式(1) と同様の関係から行って、同一ライン上の代表位置P₀₄ での代表値が算出される。
【００７９】
また、補間されるブロックにおける代表位置は、P₀₂ とする。代表値PR₀₂, PG₀₂, PB₀₂に対する各画素位置での色の画素値の代入および画素値を用いた演算を式(3)
【００８０】
【数３】
PR₀₂=P₀₂
PG₀₂=(P₀₃+P₁₂)/2 ・・・(3)
PB₀₂=P₁₃
で行う。具体的に式(3) に画素値を入れて演算すると、
【００８１】
【数４】
PR₀₂=R₀₁
PG₀₂=(G₀₄+G₁₁)/2 ・・・(4)
PB₀₂=B₁₄
が得られる。PR₀₂, PG₀₂, PB₀₂は、図11(b) のR₀₂, G₀₂, B₀₂ にそれぞれ対応している。この場合も代表位置P₀₂ だけでなく、代表位置P₀₆ に対しても同様に演算を施して求める。これらの算出により、ライン0 における画素が、図11(b) に示すように水平方向に色フィルタアレイの画素を見ると、一つおきに三原色RGB の値が得られる。この算出は、ライン8 に対しても同様に行われる。
【００８２】
これに対して、新たに生成されたラインをライン4 算出のデータとして用いる場合の算出を簡単に説明する。残したブロックに対する処理は、前述した式(3) の関係を用いて代表位置P₄₀, P₄₄の三原色RGB を求める。すなわち、
【００８３】
【数５】
PR₄₀=P₅₁
PG₄₀=(P₄₁+P₅₀)/2 ・・・(5)
PB₄₀=P₄₀
で行う。この位置にある画素データをそのまま式(5) に代入すると、
【００８４】
【数６】
PR₄₀=R₈₁
PG₄₀=(G₁₁+G₈₀)/2 ・・・(6)
PB₄₀=B₁₀
が算出される。代表位置P₄₄ も同様の関係により算出される。この補間されたラインにおける水平方向の補間処理は、前述した式(1) の関係を用いて代表位置P₄₂, P₄₆の三原色RGB を算出する。すなわち、
【００８５】
【数７】
PR₄₂=P₅₂
PG₄₂=(P₄₂+P₅₃)/2 ・・・(7)
PB₄₂=P₄₃
で行う。この位置にある画素データをそのまま式(5) に代入すると、
【００８６】
【数８】
PR₄₂=R₈₁
PG₄₂=(G₄₁+G₅₀)/2 ・・・(8)
PB₄₂=B₁₀
が算出される。代表位置P₄₆ も同様の関係により算出される。このように算出することにより、ライン4 の画素が補間される。このデータ読出しを２ライン同時読出しと同様に行いながら、ライン隣接間引き処理を行う場合、全データを用いて処理する場合に比べてほぼ1/4 のデータ量で処理できることになる。また、２ライン同時読出しにより１ライン生成が行われるとき、補間ラインが生成されることにより、実際に垂直方向の画像は、倍のレートで読み出されたときに得られる画像と同じ関係で得られる。代表値算出部12F で得られたデータ（三原色RGB ）は、YC変換部122cおよび出力形式選択部124cに供給される。
【００８７】
YC変換部122cは、供給された代表位置におけるRGB データ（代表値）を用いてたとえば、代表位置の輝度信号PY₀₀と色差信号PC_r00, PC_b00を算出する。この算出は、式(9)
【００８８】
【数９】
PY₀₀=0.3*PR₀₀+0.59*PG₀₀+0.11*PB₀₀
C_r00=PR₀₀-PY₀₀ ・・・(9)
C_b00=PB₀₀-PY₀₀
によって得られる。他の画素や他のラインにおいても式(9) の関係を用いてYCのデータを算出する。この算出されたYCデータは、出力形式選択部124cに供給される。
【００８９】
出力形式選択部124cは、図３の水平画素処理部120cとYC変換部122cからそれぞれ供給されるデータのいずれか一方をシステム制御部13から供給される制御信号により選択し、選択されたデータを画像処理部12e に出力する。また、このデータを記録に用いる場合に鑑み、図２の切換スイッチSW1 の端子b 側にも供給している。ところで、切換スイッチSW1 の端子a には、間引き処理の何等施されていない生のデータが供給される。切換スイッチSW1 は、システム制御部13からの制御に応じて記録側に供給するデータを選択している。
【００９０】
画像処理部12e は、出力形式選択部124cから供給されるデータの中で、特にYCデータに対して画像信号処理を施す。図示しないが、RGB に対しては、何も処理しないでスルーさせている。Y データには、アパーチャ調整部120eで後段で詳述するデータ評価制御部12d からの制御信号に応じて輪郭強調やコアリング等に相当する補償処理が行われる。また、C データには、色ゲイン制御部122eでデータ評価制御部12d からの制御信号に応じて色レベルの調整を行っている。このように画像処理を受けたYCデータが表示部11i に供給される。
【００９１】
なお、前述した水平画素処理部120c以降の処理は、用いる画素が異なるものの、処理の手順はライン選択間引き処理と同じ処理手順で処理が施される。
【００９２】
次に電子スチルカメラ10のもう一つの間引き処理であるライン選択間引き処理について説明する。ここで、図12の左端側の数字および画素の色を表すRGB の添字は、前述したライン隣接間引き処理の設定と同じである。
【００９３】
ラインサンプリング部120bは、システム制御部13の制御信号130 に応じて垂直タイミング信号V1のタイミングである１ラインをサンプリングする。サンプリングしたラインデータは、合成ラインサンプリング部122bにも出力供給される。このとき、ラインサンプリング部120bは水平画素処理部120cにもデータを出力しているが、図４の水平画素処理部120cはラインサンプリング部120bからの入力を禁止し、合成ラインサンプリング部122bからのデータだけを入力するように動作させる。この水平画素処理部120cのこの入力に対する動作は、システム制御部13からの制御信号13A により行なわれる（図５を参照）。図４の合成ラインサンプリング部122bには、過去にサンプリングしたラインデータが一方の入力側から供給されることになる。また、合成ラインサンプリング部122bは、他端入力側からラインデータサンプリング部12b に供給されるデータを直接入力する。合成ラインサンプリング部122bは、システム制御部13からの制御信号132 により入力制御されている。この入力制御は、ラインデータサンプリング部12b に既に供給されたラインに対して前述した所定の関係にあるラインをサンプリングするようにデータ入力が行われる。
【００９４】
たとえば、ライン0 を最初のラインとし、ラインサンプリング部120bが制御信号130 によりイネーブルのとき垂直タイミング信号V1でサンプリングする。サンプリングの後に、このラインデータは、合成ラインサンプリング部122bに供給され、たとえばラインバッファメモリに格納される。次の偶数ラインのサンプリングは８ライン後に合成ラインサンプリング部122bで行われる。このライン0 に対して合成ラインサンプリング部122bで選択されるラインは、隣接するライン1 を除くと、ライン3 あるいはライン5 が該当する。この場合のようにラインが偶数ラインのとき、次に入力するラインは、奇数ラインを選択する。ただし、ライン7 は、ライン8 と隣接ライン関係になるので除外される。図12(a) の場合、ライン3 をライン0 に対する対のもう一方のラインとして選択している（図12(a) の一点鎖線46で囲まれたラインを参照）。このとき、合成ラインサンプリング部122bは、ライン3 をライン0 に対する対のもう一方のラインとして選択するとともに、ラインサンプリング部120bは次の対の一方のラインに用いるためラインサンプリング部120bに格納される。したがって、順次、サンプリングする一方のラインをもう一方のラインに切り換えてサンプリングが行われる。
【００９５】
たとえば、ライン3 に対するもう一方のラインデータは、ライン3 と対をなす選択候補のラインはライン6 あるいはライン8 となる。ここで、偶数ラインを８ライン毎にサンプリングする条件および前述した所定の関係から、本実施例では、ライン3 に対するラインは、ライン8 になる。このライン8 を取り込むとき、この合成ラインサンプリング部122bは、システム制御部13からの制御信号132 に応じて取り込みをイネーブルにする。取り込みのタイミングは、垂直タイミング信号V2である。このように合成ラインサンプリング部122bにもう一方のラインデータも入力することによって、同時読出し的にライン0, 3とライン3, 8（一点鎖線48で囲まれた２ライン）といった２ラインデータが得られる。この得られた２ラインを水平画素処理部120cに出力する。
【００９６】
そして、また、このライン8 がさらに次の対をなすラインの一方のラインになる（一点鎖線50で囲まれたラインを参照）。ラインデータサンプリング部12b は、この場合、図12(a) の破線52a で囲まれたライン1, 2、破線52b で囲まれたライン4 〜7 およびライン9, 10 が飛ばされる、すなわち間引かれる。このようにして合成ラインサンプリング部122bで得られた２ラインの組が、図12(b) に示す関係となることが判る。この関係にあるデータを用いて、ライン0, 3, 8 のデータを算出する場合、その代表位置は、たとえば図12(c) に示すようにする。
【００９７】
水平画素処理部120cでは、前述したライン隣接間引き処理で説明したように、供給された２ラインの組を２画素ずつ区切ってブロックに分割する。この分割したブロックが１ブロックずつ間引きされ、残したブロックの画素に対応するデータの抽出が行われるとともに、さらに、残したブロックの画素データを用いて間引いたブロックを再構成する。この再ブロック化処理した際の状況は、たとえば図13(a) のような関係にある。再ブロック化されたブロックに対してもブロック内の画素データが抽出される。これらのブロックから抽出された画素データを対応する代表値算出の式に入力して代表位置P₀₀, P₀₂, P₀₄, P₀₆,・・・; P₃₀, P₃₂, P₃₄, P₃₆,・・・での三原色RGB を求める（図13(b) を参照）。たとえば、代表位置P₀₂ を算出するとき、
【００９８】
【数１０】
PR₀₂=R₀₁
PG₀₂=(G₃₁+G₀₄)/2 ・・・(10)
PB₀₂=B₃₄
が算出される。このように供給される画素データは同じ代表位置であってもライン隣接間引き処理で用いた際の画素データとは異なっている。このようにして算出されたRGB は、出力形式選択部124cとYC変換部122cに供給される。YC変換部122cでは、供給されたRGB データを用いてY データおよびC データを算出する。算出されたYCデータは、出力形式選択部124cとデータ評価制御部12d に出力する。出力形式選択部124cで選択された形式のデータは、画像処理部12e を介して表示部11i に供給される。
【００９９】
このようにライン隣接間引き処理およびライン選択間引き処理のいずれの処理によっても色G の算出に斜め相関を算出する場合と同じ演算を行うことにより、偽色の発生を抑えているので、間引かれた画像でも良好な画像が得られるようになる。
【０１００】
ところで、ライン隣接間引き処理とライン選択間引き処理において偽色と解像度の点について比較すると、ライン隣接間引き処理は、ライン選択間引き処理よりも偽色の発生が少ない。また、ライン選択間引き処理は、ライン隣接間引き処理よりも解像度の低下を抑えて表示部11i に表示させることができる。電子スチルカメラ10は、撮像した画像に対して偽色の改善処理や解像度の低下抑制処理のいずれかを行うか特に静止画モードでデータ評価制御部122cで画像評価を行う。
【０１０１】
ここで、データ評価制御部122cは、図６に示すように輝度データY と色差データ(R-Y)=C_r, (B-Y)=C_bを用いて撮像した画像の評価を行う。一般的に、モニタ表示画像は、色の少ない画像では偽色が目立つ傾向にあり、色の多い画像では偽色として色が目立たないことが知られている。この評価には、輝度データY に対する周波数検出処理を含むエッジ検出処理と、色差データ(R-Y)=C_r, (B-Y)=C_bに対する色信号検出処理とが行われる。
【０１０２】
エッジ検出処理は、最も簡単な構成として図７(a) のフィルタバンク部1200で帯域指示信号に応じて特定の周波数（たとえば、バンドパス- フィルタから）のデータだけを取り出す。このとき、この帯域を変化させることによって、評価制御部1232での最終的な処理選択の範囲を変更することもできる。このデータは、加算器1204の一端側にラインメモリ1202で１ライン遅延されたデータを減算入力し、遅延を受けていないデータを加算器1204の他端側に加算入力する。加算器1204の出力データは、ライン間の差分をとったデータでエッジ検出データになる。また、中央差分的なラインに対するエッジ検出処理を行う場合、図７(b) のように構成してライン間のエッジ検出を行う。得られたエッジ検出データは、輝度信号積算部124dで絶対値変換され、そして変換されたデータの積算が行われる。
【０１０３】
色信号検出処理は、図６に示すように、それぞれ供給される色差データ(R-Y)=C_r, (B-Y)=C_bを色信号積算部126dで絶対値変換を施し、さらにその変換されたデータを積算している。この色信号検出処理や上述したエッジ検出処理で得られたデータは、それぞれ比較制御部128dに送られる。図６に図示しないがたとえば、露出データ等のデータから画像がどのような画像かの判定結果に対応した判定信号がスレッショルドレベル格納部1224に供給されている。この判定信号を制御信号としてこの画像における適切と考えられるスレッショルドレベルを各比較部1226, 1228, 1230に出力する。比較部1226, 1228, 1230は、それぞれ供給される積算値とこのスレッショルドレベルとを比較する。これらの比較結果は、評価制御部1232に供給される。
【０１０４】
評価制御部1232では、輝度データY の積算値がスレッショルドレベル以上のとき、エッジ成分の多い画像としてライン選択間引き処理を行うようにデータ評価制御信号をタイミング信号生成部12a に出力する。また、輝度データY の積算値がスレッショルドレベルより小さかったとき、ライン隣接間引き処理を行うようにデータ評価制御信号をタイミング信号生成部12a に出力する。
【０１０５】
評価制御部1232は、色信号積算部126dでの(R-Y), (B-Y)データの積算値がスレッショルドレベル以上のとき、ライン選択間引き処理を行うようにデータ評価制御信号をタイミング信号生成部12a に出力する。また、評価制御部1232は、(R-Y), (B-Y)データの積算値がスレッショルドレベルより小さいとき、ライン隣接間引き処理を行うようにデータ評価制御信号をタイミング信号生成部12a に出力する。
【０１０６】
このライン隣接間引き処理が選択される場合、解像度が低下する傾向にあるので、評価制御部1232は、アパーチャ調整部120eにアパーチャ制御信号を供給する。アパーチャ調整部120eは、供給されるアパーチャ制御信号に応じて輪郭強調等の処理を輝度信号のY データに施す。この結果、輝度信号のY データは、解像度を向上させて画像処理部12e から出力される。また、評価制御部1232は、データの評価に応じた色制御信号を色ゲイン制御部122eに供給する。色ゲイン制御部122eは、色信号のC （C_b, C_r）データに対する色制御信号に基づいて適切なゲイン調整を行う。これにより、色信号のC データは、適切な色レベルとして表示部11i に供給される。この評価制御部1232は、評価データをシステム制御部13に供給する。システム制御部13は、上述した制御および／またはこの他の制御を供給される評価データに基づいて行うようにしてもよい。
【０１０７】
画像表示において垂直方向のエッジ部分には偽色が生じ易いことから、図６のデータ評価制御部12d の構成を簡素化しながら、偽色の検出を行う構成およびその動作について図９を参照し、説明する。共通する部分には図６で表記した同じ参照符号を用いている。輝度信号のY データには、エッジ検出部120dで帯域指示信号に応じたフィルタ処理が行われ、得られたデータに対してエッジ検出が行われる。エッジ検出部120dは、検出されたデータを絶対値変換部1212に供給する。絶対値変換部1212は、供給されるデータに対して符号を正にする変化処理を施して比較部1226に出力する。比較部1226には、スレッショルドレベル格納部1224からエッジ成分の参照レベルとして供給されている。この参照レベルは、図６のデータ評価制御部12d で供給されたレベルが積算結果に対するレベルとは異なり、たとえばライン間の差を比較するためのレベルを示す。
【０１０８】
比較部1234では供給される両方のレベルを比較している。比較部1234は、エッジ成分のレベルが参照レベル以上のときとエッジ成分のレベルが参照レベルより小さいときに分けて結果を積算制御部1236に出力する。積算制御部1234では、エッジ成分のレベルが参照レベル以上のときだけ積算部1218に積算制御信号を出力する。この輝度信号のY データのエッジ検出から積算制御までの処理の間に、画像のこの処理が行われている同じ領域における色信号のC （(R-Y)/(B-Y) ）データにもタイミング調整しながら、絶対値変換処理を絶対値変換部1216で行う。絶対値変換処理された色信号のC データは積算部1218に供給される。積算部1218は、積算制御信号が供給されるときだけ積算処理を行わせる。積算部1218は、この積算したC データを評価制御部1232に出力する。評価制御部1232では、偽色の量が多いかどうか判断が行われる。結果として偽色が多いと判断した場合、評価制御部1232は、ライン隣接間引き処理を行うようにデータ評価制御信号を出力する。このような簡素化した構成にしても的確な偽色検出を行うことができる。
【０１０９】
前述した動作は、主に電子スチルカメラ10の静止画モードで行われる。電子スチルカメラ10は、データ入力部11h でムービーモードも選択できるようになっている。このムービーモードは、静止画モードと異なり、画像表示が時間経過に応じて変化する。したがって、ムービーモードでは、時間変化も考慮した処理が要求される。この要求に対応すべく電子スチルカメラ10では、ムービーモードにおいてシステム制御部13がたとえばデータ評価制御部12d あるいは直接的にタイミング信号生成部12a の制御を行う。このとき、システム制御部13は、垂直ブランキング期間のたとえば、立下がり（図14(a) を参照）に同期して選択する各処理の切換えを行う。この切換えを行う処理は、ライン隣接間引き処理とライン選択間引き処理である。ムービーモードが指定された際にたとえば、最初の撮影画像は、垂直ブランキング信号の立下がりVB1 に同期してライン隣接間引き処理を行って表示部11i に表示させる。データ評価制御信号は、このとき図３から明らかなようにレベル"H" である。この状態が図14(b) のデータ評価制御信号の立上がりに示されている。インタレース方式で供給されるデータを表示部11i に表示させている場合、１フィールド期間中に表示用のデータを用いて前述したエッジおよび偽色の評価が行われている。
【０１１０】
次の垂直ブランキングの開始時、すなわち図14(a) のVB2 に同期してライン選択間引き処理を行って表示部11i に表示させる。ライン選択間引き処理の選択には、データ評価制御信号がレベル"L" となる（図３や図14(b) を参照）。この垂直ブランキング終了後の以降、すなわち１フィールド期間中に上述したと同様にエッジおよび偽色の評価が行われる。
【０１１１】
システム制御部13あるいはデータ評価制御部12d において、ライン隣接間引き処理とライン選択間引き処理のいずれが最適なライン間引きか判定して垂直ブランキングVB以降のライン間引き処理を決定する。図14(b) ではライン選択間引き処理がシステム制御部13あるいはデータ評価制御部12d で選択されたことを示している。
【０１１２】
この選択は以後確定的に行われるものでなく、電子スチルカメラ10の向きの変更や一定の時間経過後にこのデータ評価に基づくライン間引き処理の選択が行われる。この選択を行う際に、先と同様にライン隣接間引き処理、ライン選択間引き処理の順に行ってもよいし、逆にライン選択間引き処理、ライン隣接間引き処理の順で行ってもよい。このように動作させることにより、ムービーモードで垂直方向の解像度や偽色等を含む画像に対する最適なライン間引き処理を行っても、画質劣化の少ない画像を表示させることができるようになる。このようなデータが供給されることにより、インデックス画またはいわゆるサムネイル画と呼ばれる小さな表示部に表示される画像でも細部までの認識ができるようになる。
【０１１３】
このように構成することにより、基本的なベイヤパターンの色フィルタアレイを用いて得られたデータをライン間引き処理しても解像度の低下を抑え、かつ垂直方向に発生し易い偽色を抑えた信号処理を行うことによって表示部に良い画質の画像を表示させることができるようになる。これにより、表示部の小型化が可能になり、電子スチルカメラのより一層の小型化および表示における消費電力等の削減にも大いに貢献させることができる。
【０１１４】
なお、前述した実施例は、固体撮像装置としてCCD イメージセンサを用い、このCCD イメージセンサからの撮像信号の読出しについて説明したが、この説明に限定されるものでなく、MOS タイプのイメージセンサからの撮像信号の読出しについても前述した実施例と同様の読出しを行って信号処理を行い、有用な効果を奏することは言うまでもない。
【０１１５】
【発明の効果】
このように本発明の信号処理装置によれば、第１のサンプリング手段がタイミング信号生成手段からの出力に応じてサンプリングを行って、この際に得られたラインを用いてライン生成手段でライン合成を行うとともに、間引いたライン間の一ラインを生成して垂直方向にライン間引きされた画像を生成する。この画像は、静止画モードの際にデータ評価制御手段でこの画像のデータの評価およびその評価結果に応じてタイミング信号生成手段および信号処理を施す信号処理手段を制御して、画像に発生する偽色の抑制や垂直解像度を低下させない信号処理が行われることにより、表示部に良い画質の画像を表示させることができるようになる。さらに、これにより、表示部の小型化が可能になり、電子スチルカメラのより一層の小型化および表示における消費電力等の削減にも大いに貢献させることができる。
【０１１６】
また、本発明の信号処理方法によれば、静止画モードあるいはムービーモードの選択に応じて制御を行う際に、偶数ラインと奇数ラインを一組とし、このライン関係を保ちながら、複数の水平ライン毎にサンプリングを行うこと、すなわちライン間引きを行うタイミング信号を生成する。タイミング信号によってサンプリングして得られたラインは、当然、偶数ラインと奇数ラインの関係にあってこれらのラインから一つのラインを合成するとともに、間引いたライン間の一ラインをこれらのラインから生成する。このようにして得られたデータに基づいて静止画モードの際にデータの評価および評価結果に応じてタイミング信号の選択および施す信号処理を制御して、画像に発生する偽色の抑制や垂直解像度を低下させない信号処理が行われることにより、表示部に良い画質の画像を表示させることができる。さらに、電子スチルカメラのより一層の小型化が可能になることから、表示における消費電力等の削減にも大いに貢献させることができる。
【図面の簡単な説明】
【図１】本発明の信号処理装置を適用した電子スチルカメラの概略的な構成のブロック図である。
【図２】図１に示した信号処理部の概略的な構成のブロック図である。
【図３】図２に示したタイミング信号生成部の概略的な構成のブロック図である。
【図４】図２に示したラインデータサンプリング部およびライン生成部の概略的な構成のブロック図である。
【図５】図４に示したライン生成部における水平画素処理部の概略的な構成のブロック図である。
【図６】図４に示したデータ評価制御部の概略的な構成のブロック図である。
【図７】図６に示したエッジ検出部における(a) ２ライン処理および(b) ３ライン処理での概略的な構成のブロック図である。
【図８】図４に示した画像処理部の概略的な構成のブロック図である。
【図９】図４に示したデータ評価制御部の概略的な構成を簡素化するとともに、偽色検出を行うえるようにした変形例のブロック図である。
【図１０】本発明の信号処理方法におけるライン隣接間引き処理の手順により得られるラインデータの関係を説明し、(a) サンプリングするライン関係、(b) サンプリングで得られたラインの組の関係および(c) 各ラインの代表位置の関係を説明する模式図である。
【図１１】図10(b) に示したラインを(a) ブロック分割処理、ブロック間引き処理および再ブロック化処理の順に処理して得られたブロックの位置関係および(b) 各ラインの代表位置の関係とこの位置が有する代表値を表した模式図である。
【図１２】本発明の信号処理方法におけるライン選択間引き処理の手順により得られるラインデータの関係を説明し、(a) サンプリングするライン関係、(b) サンプリングで得られたラインの組の関係および(c) 各ラインの代表位置の関係を説明する模式図である。
【図１３】図12(b) に示したラインを(a) ブロック分割処理、ブロック間引き処理および再ブロック化処理の順に処理して得られたブロックの位置関係および(b) 各ラインの代表位置の関係とこの位置が有する代表値を表した模式図である。
【図１４】本発明を適用した電子スチルカメラにおいてムービーモードを適用する際の手順およびそのタイミングを説明するタイミングチャートである。
【符号の説明】
10 電子スチルカメラ
11 カメラ本体部
12 信号処理部
13 システム制御部
14 駆動信号生成部
20 メモリカード部
30 コネクタ
11b 色フィルタアレイ
11c CCD イメージセンサ部
11e A/D 変換部
11h データ入力部
11i 表示部
12a タイミング信号生成部
12b ラインデータサンプリング部
12c ライン生成部
12d データ評価制御部
12e 画像処理部
120a ライン隣接信号生成部
120c 水平画素処理部
122a ライン選択信号生成部
122b 合成ラインサンプリング部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a signal processing device and a signal processing method for generating display data using data for a signal supplied from a pixel when imaging of a scene is performed through a color filter, and more particularly to a color filter array having a Bayer. A pattern is used, and is suitable for use when, for example, a still image captured with hundreds of thousands to millions of pixels and a movie image are switched and displayed in an electronic still camera, a movie camera, or the like.
[0002]
[Prior art]
For example, an electronic still camera, a video camera, or the like (hereinafter, representatively referred to as an electronic still camera) photoelectrically converts light incident from an object scene in a pixel cell including, for example, a solid-state imaging device. . In particular, when a single-plate color filter is used immediately before the image pickup unit, the image pickup unit obtains a signal of only one of the three primary colors R, G, and B corresponding to the color of the color filter from each pixel. I can't. And the image imaged using the signal of each obtained color is displayed on the display part arrange | positioned in the electronic still camera etc. which were mentioned above. In general, a large-sized liquid crystal display or the like is used as the display unit when emphasizing good visibility, and a small liquid crystal display or the like is used as a display device such as an index image or a movie image when power consumption is important. Used as
[0003]
When this small liquid crystal display is used for the display unit, the number of pixels of the display unit is smaller than that of the imaging unit. Therefore, the electronic still camera thins out the obtained signal and supplies it to the display unit. Usually, a Bayer pattern is often used for the color filter array. Since two lines are used to obtain RGB from the characteristics of the color filter array, a line memory is used.
[0004]
In general, there is a concern about resolution degradation due to the thinning of the signal. Therefore, even if the signal is thinned out, for example, the solid-state imaging device disclosed in Japanese Patent Publication No. 8-2587225 reads the horizontal line signal every 1 / P horizontal period by the image sensor driving means and writes it to a plurality of memory units. The horizontal resolution is artificially improved by simultaneously reading out the data of the P horizontal line every horizontal period by the memory driving means and generating the signal by the synthesizing means.
[0005]
In addition, when a Bayer array is used for the color filter array, a color reproducibility in which a false signal is generated becomes a problem because a luminance signal having a correct spectral characteristic cannot be obtained by a line from which the signal is extracted regardless of signal thinning. There is a case. A proposal of a color image pickup apparatus that pays attention to improvement of such false signals is described in, for example, Japanese Patent Laid-Open No. 4-329786. This color image pickup apparatus uses a color filter array in which only the first color filter has an offset sampling structure and the other second and third color filters have a rectangular lattice sampling structure, and the first image filter is used to synchronize signals. The color difference difference between the signal from the pixel of the first color filter and the signal from the second color filter in the same row or column as the pixel of this color filter is taken, and similarly, the signal from the pixel of the first color filter The color difference between the pixel of this color filter and the signal from the third color filter in the same row or the same column is taken, and these color difference signals are subjected to, for example, matrix synthesis and calculation to generate a luminance signal, and spectral characteristics Correction is performed. In solid-state imaging devices using a Bayer array color filter array, attempts have been made to ensure color reproducibility by using various methods in addition to the signal processing described above.
[0006]
[Problems to be solved by the invention]
By the way, as described above, signals from the solid-state imaging device in which the Bayer array is used include a line including RG and a line including BG. When generating an image with thinning out in the vertical (V) direction, using only the lines of the same color set will not be able to reproduce the correct color because all RGB colors are not aligned. Also, if the continuous RG line and BG line are thinned out, the false signal will be improved, but the resolution will be reduced. As described above, false colors may increase or resolution may be difficult to obtain depending on how to set the lines to be used by thinning out the lines and whether the supplied image has a strong tendency to generate false colors. . Further, in the signal processing performed for improving the false color and the image quality, the readout pixels are increased in total by reading out the extracted signals. As a result, this signal processing results in an increase in circuit configuration and an increase in power consumption.
[0007]
Further, for example, when a movie image is displayed using a signal read from a solid-state imaging device that requires several million pixels, thinning processing becomes important. In this case, the high frequency component Y of the luminance signal from the information of all pixels_HIf the thinning process is performed after the pixel is generated corresponding to the pixel, it takes a lot of time.
[0008]
The present invention eliminates the drawbacks of the prior art, suppresses the generation of false colors even if line thinning processing is performed with a simple configuration, and can perform signal processing corresponding to moving image reproduction. It is another object of the present invention to provide a signal processing method.
[0009]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention provides first, second, and third color filters having different spectral sensitivity characteristics arranged in a Bayer pattern and supplied from pixels arranged corresponding to each color filter. A signal processing device that generates display data by performing signal processing according to a still image mode and a movie mode using data for a signal to be processed, wherein even lines and odd lines in sampling of lines in which data is arranged in a horizontal direction Timing signal generating means for generating and outputting a timing signal for sampling for each of a plurality of horizontal lines as a set, and a plurality of horizontal lines while thinning out the plurality of lines in which data is arranged in the horizontal direction according to the timing signal A first sampling means that performs sampling every time, and a line obtained by using the line obtained by the first sampling means. A line generation unit that combines and generates one line between the lines thinned out by the first sampling unit, and a luminance signal and a color signal obtained based on these data output from the line generation unit Signal processing means for correcting the image quality, and data evaluation control means for evaluating the data based on the data obtained by the line generation means, and controlling the timing signal generation means and the signal processing means according to the evaluation result, And a mode correspondence control means for controlling the timing generation means via the data evaluation control means in accordance with the still image mode and the movie mode.
[0010]
Here, the timing signal generation means is a sampling of adjacent lines as a set of even lines and odd lines, and is opposed to two sets of lines separated by a plurality of horizontal lines with this set of lines as a unit. Line adjacent decimation signal generating means for generating a first timing signal for sampling even lines and odd lines, and maintaining the sampling relationship between even lines and odd lines, and even lines and odd lines are separated by a plurality of horizontal lines. It is preferable to include a line separation thinning signal generating unit that generates a second timing signal for performing sampling twice at different timings. As a result, two types of timing signals are generated: a timing signal for sampling every two lines at a predetermined interval by the line adjacent thinning signal generation means, and a timing signal for sampling the opposing lines at a predetermined interval. Further, the line separation thinning signal generating means generates a timing signal for sampling at different timings twice for each line in which the horizontal lines are separated in the vertical direction at a predetermined interval.
[0011]
The line generating means is obtained by the line adjacent synthesizing means for synthesizing and generating one line from the adjacent two lines obtained by the first sampling means using the output of the line adjacent decimation signal generating means, and the first sampling means. The two adjacent lines are used as two sets, and the lines between the lines thinned out using the data of the lines respectively obtained from the second sampling means for sampling the even lines and odd lines of these lines as one set are used. It is desirable to include line interpolation generation means for generating a line by interpolation. As a result, an image in which generation of false colors in the generated line is suppressed is generated.
[0012]
The line generation means constitutes a set of lines with lines in which the positions of the even lines and the odd lines obtained by the first sampling means in response to the first timing signal are separated, and It is preferable to include line separation / synthesis means for synthesizing and generating any one of the set of lines from the set of lines. As a result, an image in which a decrease in vertical resolution in the generated line is suppressed is generated.
[0013]
The data evaluation control means includes a color component detection means for detecting the level of the color component using the data obtained by the line generation means, and a color component threshold value using the detection output of the color component detection means as a color component evaluation reference. Color component comparison / determination means for comparing and determining, edge component detection means for detecting an edge component of an image generated using data obtained by the line generation means, and detection output of the edge component detection means Edge component comparison / determination means for comparing with an edge component threshold value used as an evaluation reference, frequency component detection means for detecting the level of the frequency component included in the data using the data obtained by the line generation means, and this frequency A frequency component comparison / determination unit, a color component comparison / determination unit, and an edge component ratio are determined by comparing the detection output of the component detection unit with a level threshold value used as an evaluation criterion for the frequency component. It is advantageous to include signal control means for controlling the signal generation of the timing signal generation means and the signal processing of the signal processing means for the obtained data by comprehensively judging based on the determination results of the determination means and the frequency component comparison determination means. It is. As a result, it is possible to determine what kind of feature the image that has undergone thinning includes, and therefore, it is possible to take a countermeasure in consideration of this feature.
[0014]
The timing signal generating means is an operation switching means for switching between the line adjacent thinning signal generating means and the line separation thinning signal generating means at least every other vertical blanking signal as a switching control signal in the movie mode. The operation switching means outputs a timing signal of either one of the line adjacent decimation signal generation means and the line separation decimation signal generation means selected based on the evaluation result of the data evaluation control means for a fixed period. It is preferable to repeat the evaluation of the data evaluation control means after a predetermined period. As a result, signal processing is performed with a timing signal suitable for the object scene.
[0015]
In the signal processing apparatus of the present invention, the first sampling unit performs sampling in accordance with the output from the timing signal generation unit, performs line synthesis by the line generation unit using the line obtained at this time, and thins out the line. One line between the lines is generated to generate an image that is thinned out in the vertical direction. This image is generated in the image by the data evaluation control means in the still image mode by controlling the data of the image and the signal processing means for performing the signal processing according to the evaluation result. Signal processing that suppresses false colors and does not reduce vertical resolution is performed.
[0016]
In the present invention, color filters of first, second and third colors having different spectral sensitivity characteristics are arranged in a Bayer pattern, and data for signals supplied from pixels arranged corresponding to the respective color filters is used. A signal processing method for generating display data by performing signal processing according to still image mode and movie mode, and selecting a mode corresponding to the still image mode and movie mode, and the data in the horizontal direction A timing signal generation step of generating a timing signal for sampling for each of the plurality of horizontal lines while thinning out a plurality of lines in which the data are arranged in the horizontal direction in a set of even lines and odd lines in the sampling of the lines arranged in Depending on the timing signal obtained in the timing signal generation process, the data are arranged in a horizontal direction and thinned out. A first step of sampling for each of a plurality of horizontal lines and a line are synthesized using the lines obtained in the first step, and one line between the lines thinned out by the timing signal generation step is generated. Based on the line generation step, the signal processing step for correcting the image quality of the luminance signal and the color signal obtained based on these data output from the line generation means, and the data obtained in the line generation step And a data evaluation control step for controlling the signal processing in the signal processing step and selection of a timing signal used in the timing signal generation step according to the evaluation result.
[0017]
Here, in the timing signal generation step, adjacent lines are sampled as a set of even lines and odd lines, and two sets of lines separated by a plurality of horizontal lines with this set of lines as a unit are opposed to each other. The line adjacent decimation signal generation process for generating timing signals for sampling even lines and odd lines, and the sampling relationship between even lines and odd lines are maintained, and sampling in which even lines and odd lines are separated by a plurality of horizontal lines is different. It is preferable to operate any one of the line separation thinning signal generation processes for generating a timing signal that is performed twice at the timing.
[0018]
In the line generation step, adjacent data obtained in the first step of performing sampling for each of the plurality of horizontal lines while thinning out the plurality of lines in which the data are arranged in the horizontal direction according to the timing signal obtained in the timing signal generation step. Using two sets of adjacent line synthesis process to generate one line from the two lines and two adjacent lines obtained in the first process, sampling is performed with a set of even lines and odd lines of these lines. It is desirable to include a second step to be performed, and a line interpolation generation step of interpolating and generating one line between the thinned lines using the line data obtained from the second step.
[0019]
Further, the line generation step is obtained in the first step of performing sampling for each of the plurality of horizontal lines while thinning out the plurality of lines in which the data are arranged in the horizontal direction according to the timing signal obtained in the timing signal generation step. Line separation / combination that forms a set of lines with the lines in which the positions of the even and odd lines are separated, and generates one of the lines from the set of lines. It is desirable to include a process.
[0020]
The data evaluation control step includes a color component detection step for detecting the level of the color component using the data obtained by the line generation means in the still image mode, and the detection output of the color component detection step is used as an evaluation criterion for the color component. A color component comparison determination step for comparing with a color component threshold to be determined, an edge component detection step for detecting an edge component of an image generated using data obtained in the line generation step, and detection of the edge component detection step Frequency component detection that detects the level of the frequency component included in the data using the data obtained in the edge component comparison and determination process and the line generation process that compares the output with the edge component threshold value used as the evaluation criterion for the edge component A frequency component comparison and determination process, and a color component comparison and determination process A signal generation control step for controlling the signal generation of the timing signal generation step and the signal processing for the obtained data by comprehensively determining based on the determination results of the edge component comparison determination step and the frequency component comparison determination step Is advantageous.
[0021]
The timing signal generation step includes at least one line adjacent thinning signal generation step or line separation thinning signal generation step before performing the line adjacent thinning signal generation step or line separation thinning signal generation step for each predetermined period in the movie mode. Including an operation switching step of switching every other vertical blanking signal supplied as a switching control signal, and thinning out the evaluation result for the thinning obtained by the data evaluation control step through the operation switching step. After continuing the period, it is preferable to repeat the operation of performing the data evaluation control by alternately switching the line adjacent thinning signal generation process or the line separation thinning signal generation process. Thereby, even in the movie mode, it is possible to display an image in which lines are thinned with a timing signal that takes image characteristics into consideration.
[0022]
The line generation process treats a plurality of pixels in a horizontal direction as a single block for pixels corresponding to a plurality of lines corresponding to a Bayer pattern, and associates data supplied from each pixel of this block for each pixel position. A block sampling step for sampling, a pixel value calculation step for calculating pixel values of the first, second and third colors in each block using data of a plurality of pixels obtained in the block sampling step; A display data generation step for generating a luminance signal and a color difference signal or three primary color signals as display data with the pixel values calculated in this pixel value calculation step, and the block sampling step is a timing signal of sampling timing in the horizontal direction of the data Data based on the timing signal from the horizontal timing control process that generates Block dividing step of horizontally dividing the image in such a manner that at least one of each of the first, second and third colors is included in the block, and a block divided in this block dividing step. A block thinning step that thins out every other horizontal direction, and a block that is thinned out in this block thinning step and the pixels arranged in the column direction located at both the left and right ends and the pixels of the block adjacent to each side meet the new condition. It is preferable to include a reblocking step for forming the block, and a block data extraction step for extracting data from each block in order by making the block of the reblocking step correspond to the block at the thinning position. With this process, it is possible to generate a horizontal line while suppressing generation of false colors even if pixels are thinned out.
[0023]
In the signal processing method of the present invention, after the control by selecting the still image mode and the movie mode is selected, the even-numbered line and the odd-numbered line are set as one set, and sampling is performed for each of the plurality of horizontal lines while maintaining this line relation. That is, a timing signal for line thinning is generated. The lines obtained by sampling with the timing signal are naturally in the relationship between the even lines and the odd lines, and one line is synthesized from these lines, and one line between the thinned lines is generated from these lines. . Based on the data obtained in this way, the evaluation of the data and the signal processing that controls the selection of the timing signal and the signal processing to be performed according to the evaluation result, thereby suppressing the false color generated in the image and reducing the vertical resolution Is done.
[0024]
DETAILED DESCRIPTION OF THE INVENTION
Next, an embodiment of a signal processing apparatus and a signal processing method according to the present invention will be described in detail with reference to the accompanying drawings.
[0025]
The present invention performs signal processing for generating an image with the remaining lines and lines obtained by optimal interpolation using the remaining lines while thinning a plurality of lines in the vertical direction, thereby suppressing generation of false colors and vertical It is characterized by preventing resolution degradation. In this embodiment, the case where the present invention is applied to an electronic still camera will be described with reference to FIGS.
[0026]
As shown in FIG. 1, the electronic still camera 10 includes a camera main body 11 and a memory card unit 20, both of which are integrally connected via a connector 30. The camera body 11 includes an optical system 11a, a color filter array 11b, a CCD image sensor 11c, a preprocessing unit 11d, an A / D conversion unit 11e, a signal processing unit 12, a data compression unit 11f, an input / output interface unit 11g, a data input The memory card unit 20 includes an input / output interface unit 21 and a memory card 22. The unit 11h includes a display unit 11i and a system control unit 13.
[0027]
The optical system 11a includes a photographing lens 110a, a diaphragm (not shown), and the like. The color filter array 11b is a single plate type as described above. The arrangement of this single-plate color filter array will be described with the three primary colors R, G, and B. In this embodiment, the electronic still camera 10 uses a basic Bayer pattern for the arrangement of the color filter array.
[0028]
A case where this pattern is applied will be described. This color filter array 11b is disposed immediately before the imaging surface of the CCD image sensor 11c.
[0029]
  In the CCD image sensor 11c, the cells of the CCD image sensor 11c corresponding to the elements of the color filter array 11b are arranged in an array as pixels. When incident light from the object field forms an image in this cell, the incident light is photoelectrically converted in this cell. In order to perform signal processing of the signal processing unit 12 efficiently, signal reading is performed with simultaneous reading of two lines or with even-numbered lines being spaced apart.Odd numberIt is preferable to carry out the above method by reading the lines one by one. In this reading method, a drive signal is supplied from the drive signal generation unit 14 to the CCD image sensor 11c so as to operate according to the control of the system control unit 13, and the CCD image sensor 11c outputs an imaging signal in response to the drive signal. You may make it carry out.
[0030]
The preprocessing unit 11d amplifies the supplied signal to a predetermined level and outputs it to the A / D conversion unit 11e. The A / D changing unit 11e converts the analog signal output from the preprocessing unit 11d into a digital signal.
[0031]
The signal processing unit 12 performs signal processing with various signal processing units built therein in accordance with still signal processing (that is, still mode) and movie signal processing (that is, movie mode). In the still signal processing, signal processing that is performed when displaying a still image in which resolution is particularly important is performed. In addition, the signal processing for movies performs signal processing or the like so that an easy-to-see image can be obtained while dynamically displaying. In these modes, operations corresponding to the respective modes are performed under the control of the system control unit 13.
[0032]
As shown in FIG. 2, the signal processing unit 12 includes a timing signal generation unit 12a, a line data sampling unit 12b, a line generation unit 12c, a data evaluation control unit 12d, an image processing unit 12e, and a changeover switch SW1. . Further, the configuration of the signal processing unit 12 in FIG. 2 will be described in order with reference to FIGS.
[0033]
The timing signal generation unit 12a includes a vertical selection signal unit 12V that generates a vertical timing signal in an image and a horizontal selection signal generation unit 12H that generates a horizontal timing signal in the image. The vertical selection signal unit 12V includes a line adjacent signal generation unit 120a and a line selection signal generation unit 122a. The line adjacent signal generation unit 120a generates, for example, a timing signal for stopping the simultaneous line reading three times while simultaneously reading the data every two lines. In addition, the line adjacent signal generation unit 120a reads one line at a time in the vertical direction of the simultaneous lines read by this timing signal, and generates a timing signal for synthesizing the read lines to form a new simultaneous line.
[0034]
The line selection signal generator 122a reads out one line of data, and generates a timing signal that reads out one line separated by an even number of lines as the next line. As long as the relationship of the number of lines opened with reading, that is, the relationship of an even number is maintained, the number of lines opened (that is, thinned lines) may be arbitrary. Two lines are read out line by line, and two lines including all three primary colors RGB are generated. From the viewpoint of efficient line reading, the next line described above is further used as one line paired with the next line.
[0035]
In the timing signal generation unit 12a, since the process of reading the line once read is performed, the supplied data is preferably stored in a non-destructive memory. Although not shown, the number of thinning lines may be set in advance in each unit via the system control unit 13. Two types of signals supplied from the timing signal generator 12a to the line data sampling unit 12b in FIG. 4 are vertical timing signals Tv.
[0036]
The timing signal generator 12a generates not only the vertical timing signal Tv but also the horizontal timing signal Th by the horizontal selection signal generator 12H. The horizontal selection signal generation unit 12H is configured to divide two lines into two pixels, a timing signal for dividing a total of four pixels into blocks, a timing signal to thin out blocks every other block, and two rows and one column adjacent to the thinned blocks Are used to generate a timing signal for generating a new block and a timing signal for extracting data corresponding to the pixels in each block. In FIG. 4 and FIG. 5, these timing signals are collectively expressed as a horizontal timing signal Th. As described above, the timing signal generator 12a generates a plurality of types of timing signals.
[0037]
Further, the line adjacent signal generation unit 120a and the line selection signal generation unit 122a also have a function of selecting one according to the level of the supplied data evaluation control signal. The operations of the line adjacent signal generation unit 120a and the line selection signal generation unit 122a are controlled by the vertical selection signal Vs from the system control unit 13, and the horizontal selection signal generation unit 12H is controlled by the horizontal selection signal Hs from the system control unit 13. Operation is controlled.
[0038]
As shown in FIG. 4, the line data sampling unit 12b includes a line sampling unit 120b and a combined line sampling unit 122b. The line sampling unit 120b is provided with two sample and hold circuits or two line memories for each line so as to support simultaneous reading of two lines. When a timing signal for sampling only one line is supplied from the line selection signal generator 122a, one sample hold circuit or line memory may be operated. The data obtained by the two-line simultaneous reading is supplied as it is to the horizontal pixel processing unit 120c of the line generation unit 12c.
[0039]
Since the combined line sampling unit 122b also forms two lines, two sample and hold circuits or two line memories are provided for each line. The combined line sampling unit 122b reads the inner two lines of the four lines formed by the two sets of two-line simultaneous reading as new combined lines with the timing signal from the line adjacent signal generation unit 120a. Further, when the timing signal from the line selection signal generation unit 122a is supplied to the line sampling unit 120b and the combined line sampling unit 122b, for example, the line sampling unit 120b receives the data for the first one line to be supplied. Store by vertical timing signal. When the next vertical timing signal is supplied, the line sampling unit 120b outputs to the combined line sampling unit 122b. At this time, the data for the other line is supplied to the combined line sampling unit 122b. By simultaneously taking in data for these two lines, it is possible to make the lines at a distance apart from each other into data having a relationship equivalent to one set of two-line simultaneous reading. In this way, a composite line is formed, and this composite line is supplied to the horizontal pixel processing unit 120c.
[0040]
As shown in FIG. 4, the line generation unit 12c includes a horizontal pixel processing unit 120c, a YC conversion unit 122c, and an output format selection unit 124c. As shown in FIG. 5, the horizontal pixel processing unit 120c includes a block dividing unit 12A, a block thinning unit 12B, a data extracting unit 12C, a block generating unit 12D, a block correspondence extracting unit 12E, and a representative value calculating unit 12F. . The operation of each of these units is controlled by a control signal 13A supplied from the system control unit 13.
[0041]
In the case of signal processing for a basic Bayer pattern in the present embodiment, the block dividing unit 12A performs processing so that, for example, pixels for two lines can be treated as a block in units of two pixels in the horizontal direction. Therefore, in this embodiment, one block is composed of a total of four pixels. The block division unit 12A performs block memory management of pixel data for two lines by dividing the horizontal direction with a counter, for example, and adding block numbers. Further, a configuration may be adopted in which division processing is simply performed by a plurality of first-in first-out (FIFO) memories that collectively handle four pixels. By this division, processing for each block can be facilitated as will be described later. Also, the thinning process is performed by switching between the counter and the selection switch for each pixel so as to repeat the process of outputting two pixels and not outputting the next two pixels when thinning is performed while paying attention to the pixels in the horizontal direction. May be.
[0042]
The block thinning unit 12B performs block selection by stopping the supply of block data to the data extraction unit 12C every other divided block in the horizontal direction. That is, the block thinning unit 12B is a selection switch for each block. The block thinning unit 12B is enabled to operate when the control signal 13A from the system control unit 13A is at the level “L” and is operated at the timing of the supplied horizontal timing signal Th.
[0043]
The data extraction unit 12C extracts a signal corresponding to each pixel position in the block. Each pixel position is the block position P₀₀, P₀₁, P_Ten, P₁₁Sampling is performed corresponding to the four positions. The numbers in the subscripts indicate the positions as in the matrix elements. For this reason, although not shown, the data extraction unit 12C has four sample and hold circuits. By sampling with a fixed pixel position, every other pixel is sampled at the same pixel position in an adjacent block (for example, G₀₀, G₀₂, G₀₄,...), When sampling each pixel one by one (for example, G₀₀, R₀₁, G₀₂, R₀₃), The sampling frequency can be reduced to half the frequency. The sampled data is supplied to a representative value calculation unit 12F that generates display data. If the same processing as the block thinning function can be performed at the extraction timing of the data extracting unit 12C, the block thinning unit 12B may be omitted. Naturally, the data extraction unit 12C must supply the output at this time to the block generation unit 12D. The operation timing of the data extraction unit 12C is the same as that of the block thinning unit 12B described above.
[0044]
The block generation unit 12D generates a new block different from the skipped block using the pixels of the block remaining after the thinning process. Basically, the components of the block to be generated are the same configuration as the remaining blocks, and a total of four pixels. This newly generated block is, for example, a block (P₀₀, P₀₁, P_Ten, P₁₁) And block (P₀₄, P₀₅, P₁₄, P₁₅) Are generated from the remaining two blocks. When the generated block is expressed using each pixel position of the remaining block, the element is (P₀₁, P₀₄, P₁₁, P₁₄). That is, for example, from the blocks BL1 and BL3 left behind
[0045]
[Table 1]

As shown in FIG. 4, a block BL2 corresponding to the position between these blocks is generated.
[0046]
Since it is necessary to select and take out the components in this way, the block generation unit 12D arranges the memory for temporarily storing each block and the blocks adjacent to each other in the horizontal direction (not shown), and sets two pixels in the column direction once. Is provided with a selection unit for performing selection to be thinned out. This memory contains elements of 2 rows and 4 columns formally by storing 2 blocks. The selection unit thins out two rows and one column located at both ends of these elements. In other words, it can be said that only the elements of 2 rows and 2 columns located at the center are extracted. The block generator 12D outputs the generated block data to the block correspondence extractor 12E.
[0047]
The block correspondence extracting unit 12E extracts pixel data while placing importance on the pixel position. At this time, the block correspondence extracting unit 12E performs data extraction for the newly generated block. In this case, for example, four sample hold circuits are provided in the same manner as the number of generated pixels to perform data extraction. Although not shown, the data extraction timing has a relationship in which the sampling timing in the block corresponding extraction unit 12E is shifted by 3/4 period from the sampling timing in the data extraction unit 12C block.
[0048]
The representative value calculation unit 12F defines any one pixel in the block as the representative pixel of the block, and calculates the RGB pixel value of this pixel based on this definition. In order to perform this calculation, the representative value calculation unit 12F includes an arithmetic circuit including at least an adder and a multiplier that multiplies a coefficient by 1/2 or an adder and a divider. The representative value calculation unit 12F uses a calculation method that calculates the same representative value for the calculation for the block remaining after the block division and the calculation for the newly generated block. However, the position of the input representative value is different. The calculation using this calculation circuit will be described in detail later. The RGB values obtained here are supplied to the output format selection unit 124c and the YC conversion unit 122c.
[0049]
The YC converter 122c shown in FIG._b, C_rIs calculated. For this calculation, there are a multiplier (not shown) storing three coefficients for RGB and an adder for adding the multiplication results. The luminance signal Y is generated by adding the results of the multiplication unit storing the three coefficients for RGB. Also, color difference signal C_b, C_rEach have one adder for calculation. Color difference signal C_bIn this case, the adder adds and inputs the B pixel value calculated by the representative value calculation unit 12F from one side, and subtracts and inputs the luminance signal Y calculated as described above from the other side, and outputs it. Ask for. Similarly color difference signal C_rIs also calculated.
[0050]
Thus, by providing the representative value calculation unit 12F and the YC conversion unit 122c, it is possible to cope with RGB output and YC output. However, the display unit 11i in FIG. 1 displays in either one of the output formats. Accordingly, the mode and the like are set for the user to display on the system control unit 13 via the data input unit 11h. The system control unit 13 supplies a control signal reflecting the user's intention to the output selection unit 124c. The output selection unit 124c in FIG. 4 selects and outputs a desired output format under the control of the system control unit 13. The output selection unit 124c is naturally unnecessary when only the representative value calculation unit 12F is provided. This is because the YC conversion process requires the representative value calculation unit 12F and the YC conversion unit 122c.
[0051]
As shown in FIG. 6, the data evaluation control unit 12d includes an edge detection unit 120d, a band instruction unit 122d, a luminance signal integration unit 124d, a color signal integration unit 126d, and a comparison control unit 128d. Further, each part will be described.
[0052]
The edge detection unit 120d includes a filter bank unit, a line memory, a multiplier, and an adder. A specific configuration is shown in FIG. 7 and will be described later. Although not shown, for example, the band instruction unit 122d is supplied with the image type as data from data obtained by exposure or the like. The type of image is information that represents an imaging situation such as forward light, excessive forward light, and the like. When this information is supplied, the band instruction unit 122d outputs a band instruction signal corresponding to the information to the edge detection unit 120d.
[0053]
Here, the configuration of the edge detection unit 120d will be described. The edge detection unit 120d includes a filter bank unit 1200 so as to pass only luminance signal data corresponding to the band instruction signal from the band instruction unit 122d (see FIGS. 7A and 7B). The filter bank unit 1200 is provided with a plurality of types of digital filters having different signal pass bands. As a result, subsequent signal processing is performed only for components in a specific frequency range.
[0054]
In addition, since it is effective when performed in the vertical direction of an image, a line memory is used for edge detection processing. An example in which the edge detection is simply performed on two lines and a case where the edge detection is performed on three lines over a wider range will be exemplified. Depending on the number of lines, the configuration of edge detection is as shown in FIGS. 7A and 7B, respectively. That is, the data passed through the filter bank unit 1200 is added and supplied to one end side of the line memory 1202 and the adder 1204 in FIGS. 7 (a) and 7 (b). In FIG. 7 (a), the data y delayed by one line via the line memory 1202_n-1Is subtracted and supplied to the other end of the adder 1204. Thus, the output of the adder 1204, that is, the detection data e (n) is y_n-y_n-1 It becomes.
[0055]
Further, in FIG. 7B, a multiplier 1208 and an adder 1210 are used in addition to the configuration of FIG. 7A described above. The line memory 1202 supplies the data delayed by one line to the other end side of the line memory 1206 and the adder 1204. Addition is input to the other end of the adder 1204. Here, the output of the line memory 1202 is y_nThen, the output from the line memory 1206 and the output from the filter bank unit 1200 are each delayed by two lines._{n + 1}, No delay_n-1There is a relationship. The adder 1204 adds the supplied data and outputs the added data to the multiplier 1208 having a coefficient of 1/2. Multiplier 1208 subtracts the multiplication result to one end of adder 1210. Further, the output from the line memory 1206 is added to the other end of the adder 1210. By supplying data to each end in this way, the output of the adder 1210, that is, the detection data e (n) is y_n-(y_n-1+ y_{n + 1}) / 2. The edge detection data thus obtained is supplied to the luminance signal integrating unit 124d in FIG.
[0056]
The luminance signal integration unit 124d includes an absolute value conversion unit 1212 and an integration unit 1214. The absolute value conversion unit 1212 performs conversion that takes an absolute value for the supplied edge detection data so that the data becomes a positive value. The absolute value-converted data is supplied to the integrating unit 1214. The accumulating unit 1214 simply accumulates the supplied data and outputs the accumulated data to the comparison control unit 128d. Integration by the integration unit 1214 is repeated until line detection of the entire image is completed. Such processing is performed not only in the luminance signal integrating unit 124d but also in the color signal integrating unit 126d. The color signal integration unit 126d includes absolute value conversion units 1216 and 1220 and integration units 1218 and 1222, respectively, for the data of the color difference signals (R−Y) and (B−Y). The color signal integration unit 126d also integrates the data of the color difference signals (R-Y) and (B-Y) over the entire image and outputs the result to the comparison control unit 128d.
[0057]
The comparison control unit 128d includes a threshold level storage unit 1224, comparison units 1226, 1228, 1230, and an evaluation control unit 1232. The threshold level storage unit 1224 stores threshold levels for the data of the luminance signal Y 1 and the data of the color difference signals (R−Y) and (B−Y). Although not shown in FIG. 6, the comparison control unit 128d is controlled by the system control unit 13 as shown in FIG. By this control, the threshold level storage unit 1224 supplies appropriate threshold levels for the images to the comparison units 1226, 1228, and 1230, respectively. Comparison units 1226, 1228, and 1230 perform a level comparison between the threshold level and the corresponding accumulated data. The comparison units 1226, 1228, and 1230 supply the comparison results to the evaluation control unit 1232. This supply is performed by comparing the frequency component, edge detection component, and color signal component information detected by the edge detection unit 120d, the luminance signal integration unit 124d, and the color signal integration unit 126d with the comparison units 1226, 1228, and 1230. It means that it is supplied to each.
[0058]
The evaluation control unit 1232 evaluates an image obtained by thinning based on the comparison results from the comparison units 1226, 1228, and 1230, and outputs various control signals according to the evaluation results. The evaluation will be described in detail in the operation. The evaluation control unit 1232 outputs a data evaluation control signal, an aperture control signal, and the like as various control signals, and outputs evaluation data to the system control unit 13. By supplying evaluation data in this way, the system control unit 13 may control each unit.
[0059]
The image processing unit 12e shown in FIG. 4 includes an aperture adjustment unit 120e and a color gain control unit 122e as shown in FIG. 8, for example. The aperture adjustment unit 120e can achieve effects such as contour enhancement and coring by performing aperture correction, which is one of signal adjustments on the luminance signal (Y) data according to the control signal from the data evaluation control unit 12d. Apply corrections to obtain. Further, the color gain control unit 122e also adjusts the level of the color signal (C) in accordance with the control signal from the data evaluation control unit 12d to improve the saturation, for example.
[0060]
Although not specifically illustrated, the system control unit 13 includes a system controller that controls the electronic still lamella 10, a timing signal generation unit, an address control unit, and the like. As shown in FIG. 1, the system control unit 13 controls a CCD image sensor unit 11c, a preprocessing unit 11d, an A / D conversion unit 11e, a signal processing unit 12, and a data compression unit 11f. In particular, the signal processor 12 controls the timing signal generator 12a and the changeover switch SW1 in FIG.
[0061]
The drive signal generation unit 14 supplies drive timing such as imaging of the CCD image unit 11c and signal reading in accordance with the drive control signal from the system control unit 13. In the data thinning process in the display mode and the recording mode, when the roles of both the drive signal generation unit 14 and the timing signal generation unit 12a are appropriately divided and operated, the configuration of the apparatus is simplified while improving the processing efficiency. can do.
[0062]
The data input unit 11h sends the mode selection signal operated by the user, setting of a shutter / release button (both not shown), and information on pressing instructions to the system control unit 13 as a mode selection signal 11m. The modes operated by the user include still shooting, movie shooting, RGB output, YC output, and the like, and a desired mode can be selectively performed on the electronic still camera 10 from these modes.
[0063]
The electronic still camera 10 will be briefly described with respect to the configuration of the main body 11 used in the recording mode. The data compression unit 11f performs compression processing for reducing the number of bits of the luminance data Y, color difference data (B-Y), and (R-Y) supplied from the signal processing unit 12. For example, Huffman coding or differential PCM is used as a data compression method. As a result, the data compression unit 11f performs processing for reducing the amount of data and outputs it to the input / output interface unit 11g.
[0064]
Data is supplied to the memory card 22 via the connector 30 and the input / output interface unit 21. Conversely, the data stored in the memory card 22 is read in the order of being supplied to the camera body 11 via the input / output interface unit 21 and the connector 30.
[0065]
Further, a configuration of a modified example that simplifies the configuration of the data evaluation control unit 12d described above will be described with reference to FIG. When thinning processing is performed in the vertical direction, false color tends to occur at the boundary of the contour when the luminance data is viewed in the vertical direction. Considering this, the data evaluation control unit 12d includes an edge detection unit 120d, absolute value conversion units 1212, 1216, threshold level storage unit 1224, comparison unit 1226, integration control unit 1234, integration unit 1218, and evaluation control unit 1232. I can do it. In this configuration, the constituent elements of the data evaluation control unit 12d described above are extracted and used in addition to the integration control unit 1234. The integration control unit 1234 outputs to the integration unit 1218 a control signal that integrates the color difference signals only for the portion where the edge component is larger than the threshold value, that is, only the image region.
[0066]
In this case, the color difference signal is thinned out by half of the luminance signal Y and (R−Y) / (B−Y) is supplied in a dot sequential manner. For this reason, the processing configuration for the color difference signal can be reduced by half compared to the configuration of FIG.
[0067]
Next, the operation of the electronic still camera 10 will be described. The color filter array 11b is arranged on the front surface corresponding to the cell which is each pixel of the CCD image sensor 11c. A basic Bayer pattern is applied to the color filter array 11b (see FIG. 10 (a) and FIG. 12 (a)). The numbers written at the left end of FIG. 10 indicate the line numbers of the color filter array 11b. In addition, the position corresponding to the cell is represented by a subscript number of each color R, G, B using a matrix expression.
[0068]
The signal processing unit 12 performs line thinning processing on the data read from the CCD image sensor 11c. In other words, the CCD image sensor 11c outputs all the imaging signals without thinning out the lines as usual. This line thinning process was sampled while maintaining a predetermined relationship with the line adjacent thinning process using the data obtained by the line adjacent reading that thins out the lines in the vertical direction while reading two adjacent lines line by line. There is a line selection thinning process that thins lines in the vertical direction using lines. Here, the predetermined relationship is a relationship in which, among the three primary colors R, G, and B, a line including a color different from the color included in one sampled line is selected as the next line. With these two lines, the three primary colors R, G, and B are aligned.
[0069]
First, the line adjacent thinning process will be described. Under the control of the system controller 13, the timing signal generator 12a in the signal processor 12 operates. The timing signal generator 12a is also controlled by a data evaluation control signal from the data evaluation controller 12d. As shown in FIG. 2, the timing signal generator 12a selects a timing signal to be output by a vertical selection signal Vs and a horizontal selection signal Hs from the system control unit 13. The vertical selection signal generation unit 12V outputs vertical timing signals corresponding to the line adjacent thinning process and the line selection thinning process from the line adjacent signal generation unit 120a and the line selection signal generation unit 122a, respectively. One of the line adjacent signal generation unit 120a and the line selection signal generation unit 122a is selected based on the level of the data evaluation control signal. For example, the data evaluation control signal enables the line adjacent signal generation unit 120a at the level “H” and the line selection signal generation unit 122a at the level “L”. The timing signal generation unit 12a outputs the vertical timing signal Tv and the horizontal timing signal Th to the line data sampling unit 12b and the line generation unit 12c in FIG. 4, respectively.
[0070]
The line data sampling unit 12b samples the line with the vertical timing signal V1 supplied to the line sampling unit 120b according to the control of the system control unit 13. At this time, the vertical timing signal V1 is supplied to the line sampling unit 120b so as to sample two lines simultaneously. When sampling is performed according to the vertical timing signal V1, in the case of FIG. 10 (a), the region surrounded by the alternate long and short dash line 40, that is, the lines 0, 1, 8, and 9 are sampled. These lines are sampled by two adjacent lines as indicated by lines 0, 1, 8, and 9.
[0071]
The sampled data is supplied to the combined line sampling unit 122b and the horizontal pixel processing unit 120c. The combined line sampling unit 122b selects one of the data obtained by sampling two lines simultaneously by the supplied vertical timing signal V2. By this selection, the other line to be selected from the next two lines to be obtained is determined by the above-described predetermined relationship. The synthesized line sampling unit 122b synthesizes two new lines from the lines read out in this way by the vertical timing signal V2. A line surrounded by an alternate long and short dash line 42 in FIG. 10 (a) is a line element of the composite line. The line sampling unit 120b obtains a set of adjacent lines by sampling such lines, and further combines the lines from the set of adjacent lines to obtain a new set of adjacent lines as a composite line and outputs it to the horizontal pixel processing unit 120c. To do. Thereby, the lines 2 to 7 indicated by the broken line 44 are thinned out. In this case, since it is not necessary to input the input data directly to the combined line sampling unit 122b, the combined line sampling unit 122b inputs the data only from the line sampling unit 120b and directly inputs the data from the A / D conversion unit 11e. Data input may be prohibited.
[0072]
As shown in FIG. 10B, three sets of lines are supplied to the horizontal pixel processing unit 120c so as to simultaneously read out two lines. The set of the first and third lines separated by an alternate long and short dash line simply represents a line obtained by the line sampling unit 120b in simultaneous readout, and the second set of the middle line is obtained by the combined line sampling unit 122b. Represents the drawn line. The second set of lines is used as data corresponding to the thinned lines 4 and 5. In the horizontal pixel processing unit 120c, lines 0, 4, 8 or lines 1, 5, 9 are generated from the lines shown in FIG. When lines 0, 4, and 8 are generated, the position of the representative value is represented by a subscript number using the symbol P and matrix representation, and data calculation at the position shown in FIG. 10 (c) is performed.
[0073]
In this data calculation, first, the two lines supplied according to the control signal from the system control unit 13 are divided into two pixels by the block dividing unit 12A of FIG. That is, a block is handled in units of a total of 4 pixels. After this processing is performed, thinning out in units of blocks is performed by the block thinning unit 12B according to a control signal from the system control unit 13. Blocks remaining without being thinned out for each block are supplied to the data extraction unit 12C and the block generation unit 12D. The data extraction unit 12C reads out data obtained as pixels in the supplied block at a horizontal timing in accordance with a control signal from the system control unit 13. The read data is supplied to the representative value calculation unit 12F.
[0074]
Further, the block generation unit 12D generates a new block different from the skipped block using the pixels of the block remaining without being thinned. Basically, the components of the block to be generated are the same configuration as the remaining blocks, and a total of four pixels. This newly generated block is, for example, a block (P₀₀, P₀₁, P_Ten, P₁₁) And block (P₀₄, P₀₅, P₁₄, P₁₅) Are generated from the remaining two blocks. When the generated block is expressed using each pixel position of the remaining block, the element is (P₀₁, P₀₄, P₁₁, P₁₄). Similarly, the element is (P₀₅, P₀₈, P₁₅, P₁₈) The above-described series of processing is also performed in the second and third groups. As for the elements in the block reblocked in this way, the data corresponding to the elements is sampled at the timing of the horizontal timing signal in accordance with the control signal of the system control unit 13 by the block correspondence extracting unit 12E. The data obtained in this way is supplied to the representative value calculation unit 12F as shown in FIG. 11 (a).
[0075]
Here, the configuration of the block decimation unit 12B and the data extraction unit 12C that perform normal decimation extraction on the data shown in FIG. 5 and the configuration of the block generation unit 12D and the block correspondence extraction unit 12E that perform interpolation extraction using the remaining data are separated. The control signal 13A supplied from the system control unit 13 is used as an enable signal so that the signal level can be selected depending on, for example, the level “L” and the level “H” so that they can be operated separately.
[0076]
The representative value calculation unit 12F calculates the values of the three primary colors RGB at the representative position by performing arithmetic processing on the supplied block. For example, pixel position P₀₀ PR of the three primary colors RGB₀₀, PG₀₀, PB₀₀Is calculated. This calculation is representative value PR₀₀, PG₀₀, PB₀₀Substitution of color pixel value at each pixel position and calculation using the pixel value for
[0077]
[Expression 1]
PR₀₀= P₀₁
PG₀₀= (P₀₀+ P₁₁)/twenty one)
PB₀₀= P_Ten
To do. Specifically, when calculating by putting the pixel value in equation (1),
[0078]
[Expression 2]
PR₀₀= R₀₁
PG₀₀= (G₀₀+ G₁₁)/twenty two)
PB₀₀= B_Ten
Is obtained. PR₀₀, PG₀₀, PB₀₀R in Fig. 11 (b)₀₀, G₀₀, B₀₀ It corresponds to each. In this way, the representative position P₀₀ In addition, the representative value for the remaining block after thinning is calculated from the same relationship as in equation (1), and the representative position P on the same line is calculated.₀₄ The representative value at is calculated.
[0079]
The representative position in the block to be interpolated is P₀₂ And Representative value PR₀₂, PG₀₂, PB₀₂Substituting the pixel value of the color at each pixel position and the calculation using the pixel value with respect to (3)
[0080]
[Equation 3]
PR₀₂= P₀₂
PG₀₂= (P₀₃+ P₁₂)/twenty three)
PB₀₂= P₁₃
To do. Specifically, when calculating with the pixel value in equation (3),
[0081]
[Expression 4]
PR₀₂= R₀₁
PG₀₂= (G₀₄+ G₁₁)/twenty four)
PB₀₂= B₁₄
Is obtained. PR₀₂, PG₀₂, PB₀₂R in Fig. 11 (b)₀₂, G₀₂, B₀₂ It corresponds to each. In this case as well, representative position P₀₂ As well as the representative position P₀₆ The same calculation is performed for. As a result of these calculations, when the pixels on line 0 look at the pixels of the color filter array in the horizontal direction as shown in FIG. 11 (b), the values of the three primary colors RGB are obtained. This calculation is similarly performed for the line 8.
[0082]
On the other hand, the calculation when the newly generated line is used as data for calculating the line 4 will be briefly described. The processing for the remaining blocks is performed by using the relationship of the above-described equation (3) to represent the representative position P.₄₀, P₄₄Find the three primary colors RGB. That is,
[0083]
[Equation 5]
PR₄₀= P₅₁
PG₄₀= (P₄₁+ P₅₀)/twenty five)
PB₄₀= P₄₀
To do. If the pixel data at this position is directly substituted into equation (5),
[0084]
[Formula 6]
PR₄₀= R₈₁
PG₄₀= (G₁₁+ G₈₀) / 2 ・ ・ ・ (6)
PB₄₀= B_Ten
Is calculated. Representative position P₄₄ Is calculated by the same relationship. The horizontal interpolation process for this interpolated line is performed by using the relationship of the above-described equation (1) to represent the representative position P.₄₂, P₄₆The three primary colors RGB are calculated. That is,
[0085]
[Expression 7]
PR₄₂= P₅₂
PG₄₂= (P₄₂+ P₅₃) / 2 (7)
PB₄₂= P₄₃
To do. If the pixel data at this position is directly substituted into equation (5),
[0086]
[Equation 8]
PR₄₂= R₈₁
PG₄₂= (G₄₁+ G₅₀) / 2 (8)
PB₄₂= B_Ten
Is calculated. Representative position P₄₆ Is calculated by the same relationship. By calculating in this way, the pixel of line 4 is interpolated. When performing the line adjacent thinning process while performing this data reading in the same manner as the two-line simultaneous reading, it is possible to process with a data amount of about 1/4 compared with the case of processing using all data. In addition, when one line is generated by simultaneous readout of two lines, an interpolation line is generated, so that an image in the vertical direction can be obtained in the same relationship as an image obtained when read at a double rate. It is done. The data (three primary colors RGB) obtained by the representative value calculation unit 12F is supplied to the YC conversion unit 122c and the output format selection unit 124c.
[0087]
The YC conversion unit 122c uses, for example, the RGB signal (representative value) at the supplied representative position, for example, the luminance signal PY at the representative position.₀₀And color difference signal PC_r00, PC_b00Is calculated. This calculation is given by equation (9)
[0088]
[Equation 9]
PY₀₀= 0.3 * PR₀₀+ 0.59 * PG₀₀+ 0.11 * PB₀₀
C_r00= PR₀₀-PY₀₀                                               ... (9)
C_b00= PB₀₀-PY₀₀
Obtained by. YC data is also calculated for other pixels and other lines using the relationship of equation (9). The calculated YC data is supplied to the output format selection unit 124c.
[0089]
The output format selection unit 124c selects any one of the data supplied from the horizontal pixel processing unit 120c and the YC conversion unit 122c in FIG. 3 according to the control signal supplied from the system control unit 13, and selects the selected data. The image is output to the image processing unit 12e. In view of the case where this data is used for recording, the data is also supplied to the terminal b side of the changeover switch SW1 in FIG. By the way, raw data that has not been subjected to any thinning process is supplied to the terminal a of the changeover switch SW1. The changeover switch SW1 selects data to be supplied to the recording side in accordance with control from the system control unit 13.
[0090]
Among the data supplied from the output format selection unit 124c, the image processing unit 12e particularly performs image signal processing on YC data. Although not shown, RGB is passed through without any processing. The Y data is subjected to compensation processing corresponding to contour enhancement, coring, and the like in accordance with a control signal from the data evaluation control unit 12d described in detail later in the aperture adjustment unit 120e. For the C data, the color gain control unit 122e adjusts the color level according to the control signal from the data evaluation control unit 12d. The YC data subjected to the image processing in this way is supplied to the display unit 11i.
[0091]
Note that the processing after the horizontal pixel processing unit 120c described above is performed using the same processing procedure as the line selection thinning process, although the pixels used are different.
[0092]
Next, a line selection thinning process which is another thinning process of the electronic still camera 10 will be described. Here, the numbers on the left end side of FIG. 12 and the RGB subscripts representing the pixel colors are the same as the settings for the line adjacent thinning process described above.
[0093]
The line sampling unit 120b samples one line which is the timing of the vertical timing signal V1 according to the control signal 130 of the system control unit 13. The sampled line data is also output and supplied to the combined line sampling unit 122b. At this time, the line sampling unit 120b also outputs data to the horizontal pixel processing unit 120c, but the horizontal pixel processing unit 120c in FIG. 4 prohibits input from the line sampling unit 120b, and outputs from the combined line sampling unit 122b. Operate to input data only. The horizontal pixel processing unit 120c operates on this input in response to a control signal 13A from the system control unit 13 (see FIG. 5). The combined line sampling unit 122b in FIG. 4 is supplied with line data sampled in the past from one input side. The combined line sampling unit 122b directly inputs data supplied to the line data sampling unit 12b from the other end input side. The combined line sampling unit 122b is input-controlled by a control signal 132 from the system control unit 13. In this input control, data input is performed so as to sample a line having the predetermined relationship described above with respect to a line already supplied to the line data sampling unit 12b.
[0094]
For example, line 0 is the first line, and sampling is performed with vertical timing signal V1 when line sampling unit 120b is enabled by control signal 130. After sampling, this line data is supplied to the combined line sampling unit 122b and stored in, for example, a line buffer memory. The next even line sampling is performed by the combined line sampling unit 122b after 8 lines. For the line 0, the line selected by the combined line sampling unit 122b corresponds to the line 3 or the line 5 except for the adjacent line 1. When the line is an even line as in this case, an odd line is selected as the next input line. However, line 7 is excluded because it has an adjacent line relationship with line 8. In the case of FIG. 12 (a), the line 3 is selected as the other line of the pair with respect to the line 0 (see the line surrounded by the alternate long and short dash line 46 in FIG. 12 (a)). At this time, the combined line sampling unit 122b selects the line 3 as the other line of the pair with respect to the line 0, and the line sampling unit 120b is stored in the line sampling unit 120b for use in one line of the next pair. . Therefore, sampling is performed by sequentially switching one line to be sampled to the other line.
[0095]
For example, the other line data for line 3 is line 6 or line 8 as a selection candidate line paired with line 3. Here, from the condition for sampling even lines every 8 lines and the above-mentioned predetermined relationship, the line for line 3 is line 8 in this embodiment. When the line 8 is captured, the combined line sampling unit 122b enables the capture according to the control signal 132 from the system control unit 13. The capture timing is the vertical timing signal V2. In this way, by inputting the other line data to the combined line sampling unit 122b, two line data such as lines 0 and 3 and lines 3 and 8 (two lines surrounded by a one-dot chain line 48) are obtained simultaneously. It is done. The obtained two lines are output to the horizontal pixel processing unit 120c.
[0096]
Further, this line 8 further becomes one of the next paired lines (see the line surrounded by the alternate long and short dash line 50). In this case, the line data sampling unit 12b skips the lines 1 and 2 surrounded by the broken line 52a in FIG. 12 (a), the lines 4 to 7 and the lines 9 and 10 surrounded by the broken line 52b, that is, is thinned out. . It can be seen that the pair of two lines obtained by the combined line sampling unit 122b in this way has the relationship shown in FIG. 12 (b). When data of lines 0, 3, and 8 is calculated using data having this relationship, the representative positions are as shown in FIG. 12 (c), for example.
[0097]
In the horizontal pixel processing unit 120c, as described in the line adjacent thinning process described above, the set of the supplied two lines is divided into blocks by dividing each pixel by two pixels. The divided blocks are thinned out one by one, and data corresponding to the pixels of the remaining blocks are extracted, and further, the thinned blocks are reconstructed using the pixel data of the remaining blocks. The situation when this reblocking process is performed has a relationship as shown in FIG. 13 (a), for example. Pixel data in the block is also extracted from the reblocked block. Input the pixel data extracted from these blocks into the corresponding representative value calculation formula and enter the representative position P₀₀, P₀₂, P₀₄, P₀₆, ...; P₃₀, P₃₂, P₃₄, P₃₆,... Are obtained (see FIG. 13 (b)). For example, representative position P₀₂ When calculating
[0098]
[Expression 10]
PR₀₂= R₀₁
PG₀₂= (G₃₁+ G₀₄) / 2 ・ ・ ・ (10)
PB₀₂= B₃₄
Is calculated. The pixel data supplied in this way is different from the pixel data used in the line adjacent thinning process even at the same representative position. The RGB thus calculated is supplied to the output format selection unit 124c and the YC conversion unit 122c. The YC conversion unit 122c calculates Y data and C data using the supplied RGB data. The calculated YC data is output to the output format selection unit 124c and the data evaluation control unit 12d. Data in the format selected by the output format selection unit 124c is supplied to the display unit 11i through the image processing unit 12e.
[0099]
In this way, the occurrence of false colors is suppressed by performing the same operation as the case of calculating the diagonal correlation in the calculation of the color G in both the line adjacent thinning process and the line selection thinning process. A good image can be obtained even with a different image.
[0100]
By the way, when comparing the false color and the resolution point in the line adjacent thinning process and the line selection thinning process, the line adjacent thinning process generates fewer false colors than the line selective thinning process. In addition, the line selection thinning process can be displayed on the display unit 11i with a lower resolution than the line adjacent thinning process. The electronic still camera 10 performs either false color improvement processing or resolution reduction suppression processing on the captured image, or performs image evaluation by the data evaluation control unit 122c in still image mode.
[0101]
Here, the data evaluation control unit 122c performs luminance data Y and color difference data (R−Y) = C as shown in FIG._r, (B-Y) = C_bThe image captured using is evaluated. Generally, it is known that a monitor display image has a tendency that false colors are conspicuous in an image with few colors, and that colors are not conspicuous as false colors in an image with many colors. This evaluation includes edge detection processing including frequency detection processing for luminance data Y and color difference data (R-Y) = C_r, (B-Y) = C_bAnd a color signal detection process is performed.
[0102]
In the edge detection process, as the simplest configuration, only the data of a specific frequency (for example, from a bandpass filter) is extracted in accordance with the band instruction signal by the filter bank unit 1200 of FIG. At this time, the range of the final process selection in the evaluation control unit 1232 can be changed by changing the band. For this data, the data delayed by one line in the line memory 1202 is subtracted and input to one end of the adder 1204, and the data not subjected to delay is added and input to the other end of the adder 1204. The output data of the adder 1204 is data obtained by taking the difference between lines and becomes edge detection data. Further, when performing edge detection processing on a central differential line, the edge detection between lines is performed as shown in FIG. 7B. The obtained edge detection data is subjected to absolute value conversion by the luminance signal integrating unit 124d, and the converted data is integrated.
[0103]
In the color signal detection process, as shown in FIG. 6, the color difference data (R−Y) = C supplied respectively._r, (B-Y) = C_bThe color signal integrating unit 126d performs absolute value conversion, and further integrates the converted data. Data obtained by the color signal detection process and the edge detection process described above are sent to the comparison control unit 128d. Although not shown in FIG. 6, for example, a determination signal corresponding to a determination result of what an image is from data such as exposure data is supplied to the threshold level storage unit 1224. Using this determination signal as a control signal, a threshold level considered appropriate in this image is output to each of the comparison units 1226, 1228, and 1230. Comparison units 1226, 1228, and 1230 compare the supplied integrated values with the threshold levels. These comparison results are supplied to the evaluation control unit 1232.
[0104]
When the integrated value of the luminance data Y is equal to or higher than the threshold level, the evaluation control unit 1232 outputs a data evaluation control signal to the timing signal generation unit 12a so that the line selection thinning process is performed as an image with many edge components. When the integrated value of the luminance data Y is smaller than the threshold level, a data evaluation control signal is output to the timing signal generator 12a so as to perform line adjacent thinning processing.
[0105]
The evaluation control unit 1232 sends the data evaluation control signal to the timing signal generation unit 12a so that the line selection thinning process is performed when the integrated value of the (RY) and (BY) data in the color signal integration unit 126d is equal to or higher than the threshold level. Output. Further, when the integrated value of the (R−Y) and (B−Y) data is smaller than the threshold level, the evaluation control unit 1232 outputs a data evaluation control signal to the timing signal generation unit 12a so as to perform line adjacent thinning processing.
[0106]
When this line adjacent thinning process is selected, the resolution tends to decrease, so the evaluation control unit 1232 supplies an aperture control signal to the aperture adjustment unit 120e. The aperture adjustment unit 120e performs processing such as contour enhancement on the Y data of the luminance signal in accordance with the supplied aperture control signal. As a result, the Y data of the luminance signal is output from the image processing unit 12e with improved resolution. The evaluation control unit 1232 supplies a color control signal corresponding to the data evaluation to the color gain control unit 122e. The color gain control unit 122e outputs the color signal C (C_b, C_r) Perform appropriate gain adjustment based on the color control signal for the data. Thereby, the C data of the color signal is supplied to the display unit 11i as an appropriate color level. The evaluation control unit 1232 supplies the evaluation data to the system control unit 13. The system control unit 13 may perform the above-described control and / or other control based on the supplied evaluation data.
[0107]
Since false colors are likely to occur in the vertical edge portion in the image display, refer to FIG. 9 for the configuration and operation for detecting false colors while simplifying the configuration of the data evaluation control unit 12d of FIG. explain. The same reference numerals shown in FIG. 6 are used for common parts. The Y signal of the luminance signal is subjected to filter processing according to the band instruction signal by the edge detection unit 120d, and edge detection is performed on the obtained data. The edge detection unit 120d supplies the detected data to the absolute value conversion unit 1212. The absolute value conversion unit 1212 performs a change process to make the sign positive for the supplied data, and outputs it to the comparison unit 1226. The comparison unit 1226 is supplied from the threshold level storage unit 1224 as an edge component reference level. The reference level is a level for comparing the difference between lines, for example, the level supplied by the data evaluation control unit 12d in FIG. 6 is different from the level for the integration result.
[0108]
The comparison unit 1234 compares both supplied levels. The comparison unit 1234 outputs the result to the integration control unit 1236 separately when the edge component level is equal to or higher than the reference level and when the edge component level is lower than the reference level. The integration control unit 1234 outputs an integration control signal to the integration unit 1218 only when the level of the edge component is equal to or higher than the reference level. During the process from the edge detection of the Y signal of this luminance signal to the integration control, the timing is also adjusted to the C ((RY) / (BY)) data of the color signal in the same area where this processing of the image is performed. However, the absolute value conversion processing is performed by the absolute value conversion unit 1216. The C data of the color signal subjected to the absolute value conversion process is supplied to the integrating unit 1218. The integrating unit 1218 performs the integrating process only when the integration control signal is supplied. The accumulating unit 1218 outputs the accumulated C data to the evaluation control unit 1232. The evaluation control unit 1232 determines whether the amount of false color is large. As a result, when it is determined that there are many false colors, the evaluation control unit 1232 outputs a data evaluation control signal so as to perform the line adjacent thinning process. Even with such a simplified configuration, accurate false color detection can be performed.
[0109]
The above-described operation is mainly performed in the still image mode of the electronic still camera 10. The electronic still camera 10 can also select a movie mode with the data input unit 11h. In the movie mode, unlike the still image mode, the image display changes with time. Therefore, in the movie mode, processing that takes time changes into account is required. In response to this request, in the electronic still camera 10, in the movie mode, the system control unit 13 controls, for example, the data evaluation control unit 12d or directly the timing signal generation unit 12a. At this time, the system control unit 13 switches each process to be selected in synchronization with, for example, a fall (see FIG. 14 (a)) in the vertical blanking period. The process for performing the switching is a line adjacent thinning process and a line selection thinning process. When the movie mode is designated, for example, the first photographed image is displayed on the display unit 11i by performing line adjacent thinning processing in synchronization with the falling VB1 of the vertical blanking signal. At this time, the data evaluation control signal is at the level “H” as apparent from FIG. This state is shown at the rise of the data evaluation control signal in FIG. 14 (b). When the data supplied by the interlace method is displayed on the display unit 11i, the above-described evaluation of the edge and the false color is performed using the display data during one field period.
[0110]
At the start of the next vertical blanking, that is, in synchronization with VB2 in FIG. 14 (a), the line selection thinning process is performed and displayed on the display unit 11i. For the selection of the line selection thinning process, the data evaluation control signal becomes level “L” (see FIG. 3 and FIG. 14B). After this vertical blanking is completed, that is, during one field period, the edge and false color are evaluated in the same manner as described above.
[0111]
In the system control unit 13 or the data evaluation control unit 12d, it is determined whether the line adjacent thinning process or the line selection thinning process is the optimum line thinning process, and the line thinning process after the vertical blanking VB is determined. FIG. 14 (b) shows that the line selection thinning process is selected by the system control unit 13 or the data evaluation control unit 12d.
[0112]
This selection is not made afterwards, but the line thinning process based on the data evaluation is selected based on a change in the orientation of the electronic still camera 10 or a certain time. When this selection is performed, the line adjacent thinning process and the line selection thinning process may be performed in the same order as described above, or conversely, the line selection thinning process and the line adjacent thinning process may be performed in this order. By operating in this way, it is possible to display an image with little deterioration in image quality even when an optimal line thinning process is performed on an image including vertical resolution, false color, and the like in the movie mode. By supplying such data, even an image displayed on a small display unit called an index image or a so-called thumbnail image can be recognized in detail.
[0113]
With this configuration, even if the data obtained using a basic Bayer pattern color filter array is processed by thinning the line, the resolution is reduced and the false color that is likely to occur in the vertical direction is suppressed. By performing the processing, an image with good image quality can be displayed on the display unit. This makes it possible to reduce the size of the display unit, which can greatly contribute to further downsizing of the electronic still camera and reduction of power consumption in display.
[0114]
In the above-described embodiment, the CCD image sensor is used as the solid-state imaging device and the readout of the imaging signal from the CCD image sensor has been described. However, the present invention is not limited to this description. Needless to say, the readout of the image pickup signal is also performed in the same manner as in the above-described embodiment, and the signal processing is performed.
[0115]
【The invention's effect】
As described above, according to the signal processing apparatus of the present invention, the first sampling unit performs sampling according to the output from the timing signal generation unit, and the line generation unit uses the line obtained at this time to perform line synthesis. And one line between the thinned lines is generated to generate an image whose lines are thinned in the vertical direction. In this image, the data evaluation control means controls the timing signal generation means and the signal processing means for performing signal processing in accordance with the evaluation result of the image data in the still image mode, and the fake generated in the image. By performing signal processing that does not reduce color or reduce vertical resolution, an image with good image quality can be displayed on the display unit. Further, this makes it possible to reduce the size of the display unit, and can greatly contribute to further downsizing of the electronic still camera and reduction of power consumption in display.
[0116]
Further, according to the signal processing method of the present invention, when control is performed according to the selection of the still image mode or the movie mode, an even number line and an odd number line are set as one set, and a plurality of horizontal lines are maintained while maintaining this line relationship. Every time sampling is performed, that is, a timing signal for line thinning is generated. The lines obtained by sampling with the timing signal are naturally in the relationship between the even lines and the odd lines, and one line is synthesized from these lines, and one line between the thinned lines is generated from these lines. . Based on the data obtained in this way, the evaluation of the data in the still image mode and the selection of the timing signal according to the evaluation result and the signal processing to be performed are controlled to suppress false colors generated in the image and the vertical resolution. By performing signal processing that does not reduce the image quality, it is possible to display an image with good image quality on the display unit. Furthermore, since the electronic still camera can be further reduced in size, it can greatly contribute to reduction of power consumption in display.
[Brief description of the drawings]
FIG. 1 is a block diagram of a schematic configuration of an electronic still camera to which a signal processing apparatus of the present invention is applied.
2 is a block diagram of a schematic configuration of a signal processing unit shown in FIG. 1. FIG.
FIG. 3 is a block diagram of a schematic configuration of a timing signal generator shown in FIG. 2;
4 is a block diagram of a schematic configuration of a line data sampling unit and a line generation unit shown in FIG. 2;
5 is a block diagram of a schematic configuration of a horizontal pixel processing unit in the line generation unit shown in FIG. 4;
6 is a block diagram of a schematic configuration of a data evaluation control unit shown in FIG. 4;
7 is a block diagram of a schematic configuration in (a) 2-line processing and (b) 3-line processing in the edge detection unit shown in FIG. 6;
8 is a block diagram of a schematic configuration of an image processing unit shown in FIG. 4;
9 is a block diagram of a modified example in which the schematic configuration of the data evaluation control unit shown in FIG. 4 is simplified and false color detection can be performed.
FIG. 10 explains the relationship of line data obtained by the procedure of line adjacent thinning processing in the signal processing method of the present invention; (a) the relationship of lines to be sampled; (b) the relationship of sets of lines obtained by sampling; (c) It is a schematic diagram explaining the relationship of the representative position of each line.
11 is a block positional relationship obtained by processing the lines shown in FIG. 10B in the order of (a) block division processing, block thinning processing, and reblocking processing; and (b) representative position of each line. FIG. 6 is a schematic diagram showing the relationship and the representative value of this position.
FIGS. 12A and 12B illustrate the relationship of line data obtained by the procedure of line selection thinning processing in the signal processing method of the present invention; (a) the relationship of lines to be sampled; (b) the relationship of sets of lines obtained by sampling; (c) It is a schematic diagram explaining the relationship of the representative position of each line.
FIG. 13 is a block positional relationship obtained by processing the lines shown in FIG. 12B in the order of (a) block division processing, block thinning processing, and reblocking processing; and (b) representative position of each line. FIG. 6 is a schematic diagram showing the relationship and the representative value of this position.
FIG. 14 is a timing chart for explaining a procedure and timing for applying a movie mode in an electronic still camera to which the present invention is applied.
[Explanation of symbols]
10 Electronic still camera
11 Camera body
12 Signal processor
13 System controller
14 Drive signal generator
20 Memory card section
30 connectors
11b color filter array
11c CCD image sensor
11e A / D converter
11h Data input part
11i display
12a Timing signal generator
12b Line data sampling unit
12c line generator
12d Data evaluation controller
12e Image processing unit
120a Line adjacent signal generator
120c horizontal pixel processor
122a Line selection signal generator
122b Composite line sampling unit

Claims (13)

Color filters of first, second, and third colors having different spectral sensitivity characteristics are arranged in a Bayer pattern, and a still image mode is obtained by using data for signals supplied from pixels arranged corresponding to the color filters. A signal processing device that generates display data by performing signal processing according to a movie mode,
Timing signal generation means for generating and outputting a timing signal for sampling for each of a plurality of horizontal lines, with a set of even lines and odd lines in the sampling of lines in which the data are arranged in a horizontal direction;
First sampling means for performing sampling for each of the plurality of horizontal lines while thinning out the plurality of lines in which the data is arranged in the horizontal direction according to the timing signal;
A line generating means for synthesizing lines using the lines obtained by the first sampling means, and for generating one line between the lines thinned out by the first sampling means;
Signal processing means for performing image quality correction on luminance signals and color signals obtained based on these data output from the line generation means;
A data evaluation control unit that evaluates the data based on the data obtained by the line generation unit and controls the timing signal generation unit and the signal processing unit according to an evaluation result;
A signal processing apparatus comprising: a mode correspondence control unit that controls the timing generation unit via the data evaluation control unit according to the still image mode and the movie mode. 2. The apparatus according to claim 1, wherein the timing signal generation unit samples adjacent lines as a set of the even lines and the odd lines, and separates the set lines for each of a plurality of horizontal lines. Line adjacent decimation signal generating means for generating a first timing signal for sampling the even line and the odd line facing each other in two sets of lines;
Line separation thinning signal generation for maintaining a sampling relationship between the even lines and the odd lines and generating a second timing signal for performing sampling at different timings twice with the even lines and the odd lines separated by a plurality of horizontal lines And a signal processing apparatus. 2. The apparatus according to claim 1, wherein the line generation unit combines and generates one line from two adjacent lines obtained by the first sampling unit using an output of the line adjacent thinning signal generation unit. Combining means;
Data of each line obtained from the second sampling means that performs sampling using two sets of two adjacent lines obtained by the first sampling means and sets even and odd lines as a set. And a line interpolation generation means for interpolating and generating one line between the thinned lines using the signal. 3. The apparatus according to claim 2, wherein the line generation means is a line in which the positions of the even lines and the odd lines obtained by the first sampling means are separated in response to the first timing signal. And a line separating / combining means for composing and generating any one line of the set of lines from the set of lines. The apparatus according to claim 1, wherein the data evaluation control unit detects a color component level using data obtained by the line generation unit, and
A color component comparison / determination unit for comparing the detection output of the color component detection unit with a color component threshold value used as an evaluation criterion for the color component;
Edge component detection means for detecting an edge component of an image generated using the data obtained by the line generation means;
An edge component comparison / determination unit for comparing the detection output of the edge component detection unit with an edge component threshold value used as an evaluation criterion for the edge component;
Frequency component detection means for detecting the level of the frequency component included in the data using the data obtained by the line generation means;
Comparing the detection output of the frequency component detection means with a level threshold value used as an evaluation criterion for the frequency component, and determining the frequency component comparison determination means;
Signal generation of the timing signal generation means and signal processing means for the obtained data by comprehensively judging based on the determination results of the color component comparison determination means, the edge component comparison determination means, and the frequency component comparison determination means And a signal control means for controlling the signal processing. 3. The apparatus according to claim 2, wherein the timing signal generating means supplies at least every other line adjacent thinning signal generating means and line separation thinning signal generating means in the movie mode. Including an operation switching means for switching using the signal as a switching control signal,
The operation switching means outputs the timing signal of either one of the line adjacent decimation signal generation means and the line separation decimation signal generation means selected according to the evaluation result of the data evaluation control means in a fixed period during a predetermined period. Let
The signal processing apparatus characterized by repeating the evaluation of the data evaluation control means after the predetermined period. Color filters of first, second, and third colors having different spectral sensitivity characteristics are arranged in a Bayer pattern, and a still image mode is obtained by using data for signals supplied from pixels arranged corresponding to the color filters. A signal processing method for generating display data by performing signal processing according to a movie mode, the method comprising:
A mode-compatible selection step of selecting control according to the still image mode and the movie mode;
Timing for generating a timing signal for performing sampling for each of the plurality of horizontal lines while thinning out the plurality of lines in which the data is arranged in the horizontal direction, with a set of even lines and odd lines in the sampling of the lines in which the data are arranged in the horizontal direction A signal generation process;
A first step of performing sampling for each of a plurality of horizontal lines while thinning out a plurality of lines in which the data are arranged in the horizontal direction according to the timing signal obtained in the timing signal generation step;
A line generation step for synthesizing a line using the line obtained in the first step and generating one line between the lines thinned out by the timing signal generation step;
A signal processing step of performing image quality correction on the luminance signal and the color signal obtained based on these data output from the line generation means;
A data evaluation control step that evaluates the data based on the data obtained in the line generation step and controls selection of a timing signal used in the timing signal generation step and signal processing in the signal processing step according to the evaluation result And a signal processing method. 8. The method according to claim 7, wherein in the timing signal generation step, adjacent lines are sampled as a set of the even-numbered line and the odd-numbered line, and a plurality of horizontal lines are separated in units of the set of lines. A line adjacent decimation signal generating step for generating a timing signal for sampling the even line and the odd line facing each other in two sets of lines;
A line separation thinning signal generating step for maintaining a sampling relationship between the even line and the odd line and generating a timing signal for performing sampling at different timings twice, wherein the even line and the odd line are separated by a plurality of horizontal lines. A signal processing method comprising operating any one of the steps. 8. The method according to claim 7, wherein the line generation step performs sampling for each of the plurality of horizontal lines while thinning out the plurality of lines in which the data are arranged in the horizontal direction according to the timing signal obtained in the timing signal generation step. A line adjacent synthesis step of synthesizing and generating one line from two adjacent lines obtained in the first step of performing
A second step of performing sampling using two sets of two adjacent lines obtained in the first step, and even lines and odd lines of these lines as a set;
And a line interpolation generation step of interpolating and generating one line between the thinned lines using the line data obtained from the second step. The method according to claim 7, wherein the line generation step performs sampling for each of a plurality of horizontal lines while thinning out the plurality of lines in which the data are arranged in a horizontal direction according to the timing signal obtained in the timing signal generation step. A set of lines is formed by lines in which the positions of the even lines and the odd lines obtained in the first step are separated, and any one of the lines is set to the set. A signal processing method characterized by including a line separation / synthesis step of synthesizing and generating from the lines. The method according to claim 7, wherein the data evaluation control step includes a color component detection step of detecting a level of a color component using data obtained by the line generation unit in the still image mode;
A color component comparison determination step for comparing the detection output of the color component detection step with a color component threshold value used as an evaluation criterion for the color component;
An edge component detection step of detecting an edge component of an image generated using the data obtained in the line generation step;
An edge component comparison determination step for comparing the detection output of the edge component detection step with an edge component threshold value used as an evaluation criterion for the edge component;
A frequency component detection step of detecting the level of the frequency component included in the data using the data obtained in the line generation step;
A frequency component comparison and determination step for comparing the detection output of the frequency component detection step with a level threshold value used as an evaluation criterion for the frequency component;
Controls signal generation of the timing signal generation step and signal processing for the obtained data by comprehensively judging based on the determination results of the color component comparison determination step, the edge component comparison determination step, and the frequency component comparison determination step And a signal control process. 9. The method according to claim 8, wherein the timing signal generation step includes the line adjacent thinning signal generation step or the line adjacent thinning signal generation step or the line separation thinning signal generation step in the movie mode. Including an operation switching step of switching at least every other vertical blanking signal supplied as a switching control signal.
From the evaluation result for the thinning obtained by the data evaluation control step, the thinning is continuously continued for a predetermined period through the operation switching step, and then the line adjacent thinning signal generation step or the line separation thinning signal generation step is performed. A signal processing method characterized by repeating the operation of performing data evaluation control by switching alternately. The method according to claim 7, wherein the line generation step includes:
A block that handles a plurality of pixels in a horizontal direction as a single block with respect to pixels for a plurality of lines corresponding to the Bayer pattern, and samples data supplied from each pixel of the block corresponding to each pixel position A sampling process;
A pixel value calculating step of calculating pixel values of the first, second and third colors in each block using data of a plurality of pixels obtained in the block sampling step;
A display data generation step of generating a luminance signal and a color difference signal or a three primary color signal as the display data with the pixel value calculated in the pixel value calculation step,
The block sampling step performs horizontal sampling of the data based on a timing signal from a horizontal timing control step of generating a timing signal of sampling timing in the horizontal direction of the data, and the first, second and second A block dividing step for horizontally dividing a condition that includes at least one of the three colors in the block;
A block thinning step for horizontally thinning every other block divided in the block dividing step;
A reblocking step in which the pixels arranged in the column direction located at the left and right ends in the block thinned out in the block thinning step and the pixels of the blocks adjacent to the sides respectively satisfy the above condition to form a new block;
And a block data extracting step of extracting data from each block in order by making the block of the reblocking step correspond to the block at the thinning position.

Images