JP2004172726A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technology capable of suppressing a computing amount even when parts other than an edge are smoothed while keeping the edge in which a change in the pixel value is steep. <P>SOLUTION: A nonlinear smoothing unit 22 is provided with ε filters 53A to 53C connected in cascade. The ε filters are set from the pre-stage ε filter 53A toward the post-stage ε filter 53C in a way of being formed from a narrow band filter with many numbers of taps to a broad band filter with a few numbers of taps. In this case, the interval of the number of taps from the ε filter 53A to the ε filter 53C is set to the power of two. Threshold values ε<SB>1</SB>, ε<SB>2</SB>, ε<SB>3</SB>of the ε filter 53A to the ε filter 53C are set to be a relation of ε<SB>1</SB><ε<SB>2</SB><ε<SB>3</SB>. The image processing apparatus and method is applicable to video cameras. <P>COPYRIGHT: (C)2004,JPO

Description

【００８８】
すなわち、εフィルタ５３は、フィルタ処理の中心画素ｐｎ の画素値Ｘｎ と画素ｐｎ−ｋ の画素値Ｘｎ−ｋ との差の絶対値｜Ｘｎ −Ｘｎ−ｋ ｜を所定の閾値εと比較する。その結果、εフィルタ５３Ａは、絶対値｜Ｘｎ −Ｘｎ−ｋ ｜が所定の閾値εよりも小さいと判断した場合には、ｗｎ−ｋ に画素値Ｘｎ−ｋ を代入し、ａｋ を各タップ係数とした通常の線形ローパスフィルタと同様の処理を実行することにより、中心画素ｐｎ を中心として画像を一様に平滑化する。
【００８９】
これに対して、εフィルタ５３は、絶対値｜Ｘｎ −Ｘｎ−ｋ ｜が所定の閾値εよりも大きいと判断した場合には、ｗｎ−ｋ に画素値Ｘｎ を代入し、画素ｐｎ−ｋ の画素値Ｘｎ−ｋ を中心画素ｐｎ の画素値Ｘｎ に置き換えた後に当該中心画素ｐｎ を中心としてローパスフィルタ処理を行うことにより、画素値Ｘｎ−ｋ を無視して画素値Ｘｎ 近傍の画素値のみで平滑化を行う。
【００９０】
図１１は、εフィルタ５３Ａ（１７タップのεフィルタ）のレジスタ１０１乃至１１６に保持される画素値を説明する図である。
【００９１】
図１１において、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＸ１，Ｘ２，Ｘ３，Ｌ１，Ｘ４，Ｘ５，Ｘ６，およびＬ２とされている。また、中心画素より右側の画素値がＹ１，Ｙ２，Ｙ３，Ｒ１，Ｙ４，Ｙ５，Ｙ６，およびＲ２とされている。
【００９２】
通常、ラインの走査は、左上から右下方向に行なわれるので、図中、最も左側の画素値Ｌ２が最も先に走査され、最も右側の画素値Ｒ２が最後に走査されるので、画素値Ｌ２がレジスタ１１６に保持され、画素値Ｘ６がレジスタ１１５に保持され、画素値Ｘ５がレジスタ１１４に保持される。以下、順次、図１１において左側の画素値から順番にレジスタ１１３，１１２，１１１，・・・に保持され、画素値Ｙ６がレジスタ１０１に保持され、画素値Ｒ２がセレクタ１４４に直接供給される（画素値Ｒ２をラッチするレジスタを設けてもよい）。
【００９３】
上述したようにεフィルタ５３Ａのタップ係数は｛１，０，０，０，２，０，０，０，２，０，０，０，２，０，０，０，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされ、Ｘ１乃至Ｘ６（それぞれ、レジスタ１１５，１１４，１１３，１１１，１１０，１０９に保持される）、並びに、Ｙ１乃至Ｙ６（それぞれ、レジスタ１０７，１０６，１０５，１０３，１０２，１０１に保持される）の係数は０とされる（各レジスタの出力が加算器１８１に供給されていない）。
【００９４】
上述した式（１）を適用すると、例えば、画素値Ｃと画素値Ｌ１との差の絶対値｜Ｃ−Ｌ１｜が、所定の閾値ε_１と比較され、所定の閾値ε_１よりも大きいとされた場合、画素値Ｌ１が、画素値Ｃに置き換えられる。式（１）によりＲ２，Ｙ６，Ｙ５，Ｙ４，Ｒ１，Ｙ３，Ｙ２，Ｙ１，Ｘ１，Ｘ２，Ｘ３，Ｌ１，Ｘ４，Ｘ５，Ｘ６，およびＬ２のそれぞれに対して演算された結果の画素値をＲ２’，Ｙ６’，Ｙ５’，Ｙ４’，Ｒ１’，Ｙ３’，Ｙ２’，Ｙ１’，Ｘ１’，Ｘ２’，Ｘ３’，Ｌ１’，Ｘ４’，Ｘ５’，Ｘ６’，およびＬ２’とすると、εフィルタ５３Ａから出力される画素値Ｃ’は、式（２）が演算されることにより行なわれる（式（２）は、図１０のステップＳ１０６乃至ステップＳ１０８の処理をまとめた式とされる）。
【数２】

【００９５】
図１２は、εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける閾値εの関係を説明する図である。
【００９６】
εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける閾値εは、後段のεフィルタ５３になるほど、大きくなる。すなわち、この場合、閾値ε_１＜ε_２＜ε_３とされる。
【００９７】
閾値εは、大きい値であるほど、ナマリが生じやすく（すなわち、ボケが生じるため、ノイズが取れる）、小さい値であるほど、ナマリが生じにくい（すなわち、ボケが生じにくく、ノイズが残る）。
【００９８】
例えば、タップ数が３とされ、タップ係数が｛１，２，１｝とされ、Ｌ１，Ｃ，Ｒ１が、それぞれＬ１＝３０，Ｃ＝１００およびＲ１＝８０とされ、閾値εが５０とされる場合、式（１）により、Ｌ１’＝１００とされる（Ｒ１’はそのままの値（Ｒ１）とされる）。また、式（２）により、Ｃ’＝｛１×１００＋２×１００＋１×８０｝／４が演算され、出力画素Ｃ’＝９５とされる。一方、例えば、タップ数が３とされ、タップ係数が｛１，２，１｝とされ、Ｌ１＝３０，Ｃ＝１００，Ｒ１＝８０とされ、閾値εが８０とされる場合（すなわち、上述した例と閾値εの値のみが異なる場合）、式（１）から、Ｌ１’とＲ１’はそのままの値（Ｌ１，Ｒ１）とされる。また、式（２）により、Ｃ’＝｛１×３０＋２×１００＋１×８０｝／４が演算され、出力画素Ｃ’＝７７．５とされる。このように、閾値εが大きいほど（いまの例の場合、閾値εが５０より８０であるほど）、出力画像にナマリが生じる（ボケが生じる）こととなる。
【００９９】
図１３は、εフィルタ５３Ｂ（９タップのεフィルタ）のレジスタ３０１乃至３０８に保持される画素値を説明する図であり、図１４は、εフィルタ５３Ｃのレジスタ３０１乃至３０８に保持されている画素値を説明する図である。
【０１００】
図１３において、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＸ１，Ｌ１，Ｘ２，およびＬ２とされている。また、中心画素より右側の画素値がＹ１，Ｒ１，Ｙ２，およびＲ２とされている。上述したようにεフィルタ５３Ｂのタップ係数は｛１，０，２，０，２，０，２，０，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされ、Ｘ１，Ｘ２，Ｙ１，およびＹ２（それぞれ、レジスタ３０７，３０５，３０３，３０１に保持される）の係数は０とされる。
【０１０１】
また、図１４のεフィルタ５３Ｃにおいて、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＬ１，およびＬ２とされている。また、中心画素より右側の画素値がＲ１，およびＲ２とされている。上述したようにεフィルタ５３Ｃのタップ係数は｛１，２，２，２，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされる。
【０１０２】
このように、εフィルタ５３Ａのインターバルは、４（＝２^２）（インターバル設定のためのレジスタの数は３個）、εフィルタ５３Ｂのインターバルは、２（＝２^１）（インターバル設定の為のレジスタの数は１個）、εフィルタ５３Ｃのインターバルは、１（＝２^０）（インターバル設定のためのレジスタの数は０個）とされる。
【０１０３】
ここで、それぞれの画素値に対して上述した式（１）と式（２）が適用されることにより、出力画素値Ｃ’が得られる。このとき、タップ数が大きいεフィルタ５３Ａに比較して、εフィルタ５３Ｂにより演算される出力画素値Ｃ’は、ノイズが多くなる。また、タップ数が大きいεフィルタ５３Ｂに比較して、εフィルタ５３Ｃにより演算される出力画素値Ｃ’は、ノイズが多くなる。すなわち、ノイズの大きさは、タップ数の多いεフィルタ５３Ａにおいてはノイズが少なく、後段のεフィルタ５３Ｃ（タップ数が少ない）になるに連れて、ノイズが多くなる。
【０１０４】
そこで、図１２に示されるように、εフィルタ５３の閾値εを、εフィルタのタップ数により変化させる（タップ数の多いεフィルタ５３Ａにおいては、閾値ε_１を小さくさせ、タップ数の少ないεフィルタ５３Ｃにおいては、閾値ε_３を大きくさせる）ことにより、εフィルタ５３Ａ乃至εフィルタ５３Ｃの全ての出力画素値Ｃ’のノイズを抑えることができる。
【０１０５】
図１５は、平滑化処理される画像データを説明する図である。
【０１０６】
図中左側の入力された画像データは、εフィルタ５３により平滑化処理されることによって、図中右側に示されるような画像データにされる。
【０１０７】
すなわち、急峻なエッジの前後で画素値の差の絶対値が所定の閾値εを超える場合、中心画素ｐｎ を中心とし、ローパスフィルタ処理を行う際には画素ｐｍ の画素値Ｘｍ を中心画素ｐｎ の画素値Ｘｎ に置き換えてローパスフィルタ処理を施すことにより、画素値Ｘｎ の近傍で平滑化するのに対して、画素ｐｍ を中心としてローパスフィルタ処理を行う際には画素値Ｘｍ 近傍で平滑化する。
【０１０８】
このとき、εフィルタ５３Ａは、フィルタ処理を行う範囲内に画素値Ｘの近い画素ｐが存在しないエッジ部分の画素ｐでは、当該エッジ部分の画素ｐの画素値Ｘをほぼそのまま出力することにより、エッジの急峻な変化をそのまま保存する（式（１））。因みに、εフィルタ５３Ａは、線形ローパスフィルタ５１と同様に、１次元のεフィルタを画像の水平方向および垂直方向にそれぞれ適用することによって構成される場合と、２次元のεフィルタによって構成される場合とがある。
【０１０９】
図１６は、εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける周波数応答を説明する図である。
【０１１０】
図１６において、横軸はナイキスト周波数を０．５として正規化した周波数であり、縦軸はフィルタ周波数応答を表わしている。５３Ａ，５３Ｂ，５３Ｃは、それぞれ、εフィルタ５３Ａ、εフィルタ５３Ｂ、およびεフィルタ５３Ｃに対応する周波数応答を示す。
【０１１１】
εフィルタ５３（例えば、εフィルタ５３Ａ）のタップ数のインターバル（間引き）を２のべき乗とすることにより、図１６に示されるように、εフィルタ５３Ａ（例えば、インターバル＝４）の周波数応答の波形におけるサイドローブのピークを、εフィルタ５３Ｂ（例えば、インターバル＝２）の周波数応答により打ち消すことができる（εフィルタ５３Ａの周波数応答の波形におけるサイドローブのピーク点と、εフィルタ５３Ｂの周波数応答の波形における０点が同じであるため、打ち消している）。すなわち、前のεフィルタ５３のサイドローブを、次のεフィルタ５３により抑制することができ、もって、平滑化効果とサイドローブの抑制を両立することができる。
【０１１２】
εフィルタ５３により平滑化された平滑化画像データは、上述したように、演算部２３により演算され、画像処理回路１２の出力として、カメラ信号処理回路１３に供給される。このとき、画像処理回路１２から出力される出力画像データの各画素値ｙ（ｉ，ｊ）は、式（３）により示される。
【数３】

【０１１３】
式（３）において、ゲイン係数ｇ（ｉ，ｊ）を全画面に一様なゲイン係数Ｇに置き換えると共に、平滑化画像データの各画素値ｓ（ｉ，ｊ）を入力画像データのダイナミックレンジの中央値又は全画素値の平均値Ｃに置き換えると、この式（３）は、従来のトーンカーブ調整によるコントラスト強調方法を示す式（４）のように表わされる。
【数４】

【０１１４】
また、式（３）において、ゲイン係数ｇ（ｉ，ｊ）を全画面に一様なゲイン係数Ｇに置き換えると共に、平滑化画像データの各画素値ｓ（ｉ，ｊ）を入力画像データの各画素値ｘ（ｉ，ｊ）に線形ローパスフィルタ処理を施した画像データの各画素値ｆ（ｉ，ｊ）に置き換えると、この式（３）は、従来のアンシャープマスクによる高域成分強調方法を示す式（５）のように表わされる。
【数５】

【０１１５】
このように、従来のコントラスト強調方法は、画像の各画素毎に独立して画像強調処理を行う方法であると共に、従来の高域成分強調方法は、中心画素の周辺に存在する周辺画素との相対的なレベル差を基に画像強調処理を行う方法であるのに対して、本実施の形態による方法は、非線形平滑化フィルタを用いて従来のコントラスト強調方法及び高域成分強調方法を統合し、一段と高品位な画像強調処理を可能にしている。
【０１１６】
また、従来の非線形平滑化フィルタを用いた方法（入力画像データのうち画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を増幅することにより、エッジ以外の部分を強調して表示する方法）に対して、本実施の形態による方法は、εフィルタ５３のタップ数のインターバルを２のべき乗とし、εフィルタ５３の閾値εをフィルタの帯域にあわせて変更することにより、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。
【０１１７】
以上のように、帯域に応じた閾値εを設定し（狭帯域フィルタ（タップ数の多いεフィルタ）の場合は、閾値εを小さくし、広帯域フィルタ（タップ数の少ないεフィルタ）の場合は、閾値εを大きくする）、狭帯域フィルタから広帯域フィルタの順番にεフィルタをカスケード接続し、さらに、εフィルタのインターバル（実質的に処理する画素の抽出間隔）を２のべき乗とすることにより、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。
【０１１８】
また、ノイズが生じにくく、かつ、計算量を抑えた出力画像を得ることができる。
【０１１９】
また、サイドローブを抑えた画像を出力させることができる。
【０１２０】
さらに上述の実施の形態においては、本発明をビデオカメラ１に適用した場合について述べたが、本発明はこれに限らず、例えば電子スチルカメラ、プリンタ、ディスプレイ、コンピュータのような他の種々の画像処理装置に本発明を広く適用し得る。この場合、コンピュータでは、画像コントラストを修正する際、ダイナミックレンジを維持しながら高品位なコントラスト修正画像を得ることができ、また異なる照明条件下で得られた画像同士を合成する際、それぞれのコントラスト成分の違いのみを補正でき、自然な合成画像を生成することができる。
【０１２１】
なお、以上の例においては、非線形平滑化器２２が備えるεフィルタの数を３個（εフィルタ５３Ａ，εフィルタ５３Ｂ、およびεフィルタ５３Ｃ）としたが、任意のＮ個のεフィルタ５３を備えるようにしてもよい。勿論、その場合、Ｎ番目のεフィルタをεフィルタ５３Ｎとすると、εフィルタ５３Ａの閾値ε_１は最小とされ、後段に接続されるεフィルタ５３Ｎの閾値ε_Ｎは最大とされる（ε_１＜ε_Ｎ）。
【０１２２】
また、以上の例においては、εフィルタを用いるようにしたが、εフィルタだけでなく、例えば、メディアンフィルタ等を用いるようにしても良い。
【０１２３】
なお、本明細書において、各フローチャートを記述するステップは、記載された順序に従って時系列的に行われる処理はもちろん、必ずしも時系列的に処理されなくとも、並列的あるいは個別に実行される処理をも含むものである。
【０１２４】
【発明の効果】
以上の如く、本願発明によれば、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。特に、この発明によれば、ノイズが生じにくく、かつ、サイドローブを抑えた画像を出力させることができる。
【図面の簡単な説明】
【図１】本発明を適用したビデオカメラの機能的構成を示すブロック図である。
【図２】図１の画像処理回路における画像処理を説明するフローチャートである。
【図３】図１の非線形平滑化器の機能的構成を示すブロック図である。
【図４】図２のεフィルタの構成例を示すブロック図である。
【図５】図２の他のεフィルタの構成例を示すブロック図である。
【図６】図２のさらに他のεフィルタの構成例を示すブロック図である。
【図７】図４のセレクタの構成例を示すブロック図である。
【図８】図２の非線形平滑化器における非線形平滑化処理を説明するフローチャートである。
【図９】図８のステップＳ３４の処理を説明するフローチャートである。
【図１０】図９のステップＳ７１乃至ステップＳ７３の処理を説明するフローチャートである。
【図１１】図４のレジスタに保持される画素値を説明する図である。
【図１２】図３の各εフィルタにおける閾値の関係を説明する図である。
【図１３】図５のレジスタに保持される画素値を説明する図である。
【図１４】図６のレジスタに保持される画素値を説明する図である。
【図１５】平滑化処理される画像データを説明する図である。
【図１６】タップ数と周波数応答の関係を示す図である。
【符号の説明】
１ ビデオカメラ， １１ 撮像素子， １２ 画像処理回路， ２１ 遅延回路， ２２ 非線形平滑化器， ２３ 演算部， ３１ 減算器， ３２ 乗算器， ３３ 加算器， ５１ 線形ローパスフィルタ， ５２ ルックアップテーブル， ５３ εフィルタ， ５４ ルックアップテーブル， ５５ 線形ローパスフィルタ， １０１乃至１１６ レジスタ， １４１乃至１４４ セレクタ， １６１乃至１６５ 増幅器， １８１ 加算器， １８２ 割算器， １４０ 閾値設定部， ２００ 減算器， ２０１ セレクタ， ２０２ 絶対値変換器， ２０３ 大小比較器[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and method capable of reducing the amount of calculation even when smoothing a part other than the edge while preserving an edge with a sharp change in pixel value. Apparatus and method.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, in a video camera, the contrast (brightness / darkness difference) and sharpness (clearness of a boundary) of an image captured by an imaging device such as a charge coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS) are improved. As a method, a contrast enhancement method by gradation conversion and a high-frequency component enhancement method for enhancing the contrast of a high-frequency component in an image have been considered.
[0003]
As a contrast enhancement method, for each pixel of an image, a tone curve adjustment for converting the pixel level by a function having a predetermined input / output relationship (hereinafter referred to as a level conversion function), a frequency distribution of the pixel level, and the like. A method called histogram equalization in which a level conversion function is adaptively changed according to the so-called contour enhancement, which extracts edges from an image and emphasizes the extracted edges, has been proposed as a high-frequency component enhancement method. A method called “unsharp mask” has been proposed.
[0004]
However, the contrast enhancement method has a problem that the contrast can be improved only in a part of the luminance range of the entire dynamic range (difference between the maximum level and the minimum level) of the image. In contrast, in contrast, in the brightest and darkest parts of an image, and in the case of histogram equalization, there is a problem that the contrast is reduced in the vicinity of a luminance region where the frequency distribution is small. Further, the high-frequency component emphasizing method has a problem that only the contrast of the high-frequency component of the image is enhanced, whereby the vicinity of the edge of the image is unnaturally enhanced, and the image quality cannot be avoided from being deteriorated.
[0005]
Therefore, the present applicant has previously proposed that a part other than the edge is amplified and displayed by emphasizing a part other than the edge while preserving the edge of the input image data in which the pixel value changes sharply. (For example, Patent Document 1).
[0006]
[Patent Document 1]
JP 2001-298621 A
[0007]
[Problems to be solved by the invention]
However, in the method described in Patent Literature 1, when a portion other than the edge is smoothed while preserving an edge with a sharp change in pixel value, it is extremely difficult to obtain a sufficient smoothing effect. It is necessary to provide a non-linear filter having a large number of taps, and the problem is that the amount of calculation increases.
[0008]
The present invention has been made in view of such a situation, and it is possible to reduce the amount of calculation even in a case where a portion other than the edge is smoothed while an edge having a sharp change in pixel value is stored. Is to do so.
[0009]
[Means for Solving the Problems]
The image processing apparatus of the present invention includes smoothing means for smoothing input image data and generating smoothed image data, wherein the smoothing means is a cascade-connected parameter and band for controlling the degree of smoothing. It is characterized by comprising a plurality of non-linear filters having different intervals with respect to each other.
[0010]
The smoothing means may adaptively change the degree of smoothing based on a difference value between a central pixel and its neighboring pixels.
[0011]
The value of the parameter of the nonlinear filter can be set to be larger in the latter stage than in the former stage.
[0012]
The value of the interval of the non-linear filter can be set to be smaller in the latter stage than in the former stage.
[0013]
The interval of the nonlinear filter may be a power of two.
[0014]
The nonlinear filter may be an ε filter or a median filter.
[0015]
The image processing method of the present invention includes a smoothing step of smoothing input image data and generating smoothed image data, wherein the processing of the smoothing step is a cascade-connected parameter for controlling the degree of smoothing. And a plurality of nonlinear filters having different intervals with respect to the band.
[0016]
In the present invention, the input image data is smoothed. The smoothing of the input image data is performed by a plurality of non-linear filters connected in tandem and having different parameters for controlling the degree of smoothing and intervals related to bands.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 1 shows a configuration example of a video camera 1 to which the present invention is applied.
[0018]
An image sensor 11 including an image sensor such as a CCD or a CMOS acquires a light image from a subject as input image data, and inputs the input image data to a delay circuit 21 and a non-linear smoother 22 of an image processing circuit 12. The input image data is a two-dimensional digital image, and a pixel value corresponding to a position (i, j) on the image is represented by x (i, j).
[0019]
The nonlinear smoother 22 smoothes the input image data to generate smoothed data.
[0020]
The delay circuit 21 delays the input image data captured by the image sensor 11 by a predetermined amount, and sends the delayed image data to the subtracter 31 of the arithmetic unit 23 as delayed image data. The arithmetic unit 23 constituting the image enhancing means is configured by a multiplier 32 and an adder 33 in addition to the subtracter 31, and generates output image data. The subtractor 31 subtracts each pixel value s (i, j) of the smoothed image data from each pixel value x (i, j) of the delayed image data, and outputs the resulting difference image data to the multiplier 32. Output.
[0021]
The multiplier 32 multiplies and amplifies each pixel value (x (i, j) -s (i, j)) of the difference image data by a gain coefficient g (i, j), and obtains the result. The difference image data is output to the adder 33. As the gain coefficient g (i, j), a uniform value for the entire screen or a value set for each pixel (i, j) is used.
[0022]
The adder 33 calculates each pixel value of the differential image data supplied from the multiplier 32.
g (i, j) × (x (i, j) −s (i, j))
Is added to each pixel value s (i, j) of the smoothed image data, which is the offset amount subtracted by the subtractor 31, and the output image data obtained as a result is output to the camera signal processing circuit 13. .
[0023]
The camera signal processing circuit 13 performs predetermined data processing on the output image data, and sends the resulting output image data to a VTR (Video Tape Recorder) 14 for recording.
[0024]
Next, the image processing in the image processing circuit 12 in FIG. 1 will be described with reference to the flowchart in FIG. This process is started when input image data is supplied from the image sensor 11 in accordance with a user's command.
[0025]
In step S1, the delay circuit 21 acquires input image data supplied from the image sensor 11.
[0026]
In step S2, the non-linear smoother 22 acquires the input image data supplied from the image sensor 11.
[0027]
In step S3, the non-linear smoother 22 extracts an edge component whose pixel value changes sharply from the input image data acquired in the process of step S2, and does not smooth the edge component without smoothing. On the other hand, the small amplitude components other than the edge components are smoothed, so that the input image data is smoothed while the edge components of the input image data are preserved to generate smoothed image data. I do. Details of the processing will be described later with reference to FIGS. The non-linear smoother 22 supplies the generated smoothed image data to the subtractor 31 and the adder 33.
[0028]
In step S4, the delay circuit 21 delays the input image data acquired by the processing in step S1 by a predetermined amount, and sends the input image data to the subtracter 31 as delayed image data.
[0029]
In step S5, the subtractor 31 is supplied from the non-linear smoother 22 from each pixel value x (i, j) of the delayed image data (the processing in step S4) supplied from the delay circuit 21 (the processing in step S3). ) Subtract each pixel value s (i, j) of the smoothed image data to generate difference image data. The subtracter 31 supplies the generated difference image data to the multiplier 32.
[0030]
In step S6, the multiplier 32 adds a gain coefficient g to each pixel value (x (i, j) -s (i, j)) of the difference image data supplied from the subtractor 31 (processing in step S5). The signal is amplified by multiplying by (i, j), and the resulting difference image data is sent to the adder 33. As the gain coefficient g (i, j), a uniform value for the entire screen or a value set for each pixel (i, j) is used.
[0031]
In step S7, the adder 33 determines each pixel value of the differential image data supplied from the multiplier 32 by the processing in step S6.
g (i, j) × (x (i, j) −s (i, j))
Is added to each pixel value s (i, j) of the smoothed image data (smoothed image data supplied from the non-linear smoother 22 by the process of step S3), and the resulting output image data To the camera signal processing circuit 13.
[0032]
By the processing of FIG. 2, the image processing circuit 12 can smooth the input image data and obtain output image data subjected to image enhancement processing.
[0033]
FIG. 3 is a block diagram illustrating a configuration example of the nonlinear smoother 22.
[0034]
The linear low-pass filter 51 attenuates the signal level of an extremely high frequency component of the input image data, and sends out the resulting image data to the look-up table 52. The look-up table 52 performs a gradation conversion such as a logarithmic conversion on the image data supplied from the linear low-pass filter 51 (a non-linear conversion is performed on the input image data to generate the non-linear image data). ), And sends the resulting image data to the ε-filter 53.
[0035]
The ε filter 53 is composed of n (three in the example of FIG. 3) ε filters 53A to 53C connected in cascade (cascade connection). The ε filter 53A is a non-linear smoothing filter, that is, a digital filter effective when smoothing a pixel value without impairing a sharp change in the pixel value. Is stored, and the image data is smoothed, and the resulting smoothed image data is sent to the ε filter 53B.
[0036]
The ε filter 53B connected to the subsequent stage of the ε filter 53A and the ε filter 53C connected to the subsequent stage have the same configuration as the ε filter 53A, but differ in the threshold ε and the band (the number of taps). The details of the ε filters 53A to 53C will be described later with reference to FIGS.
[0037]
The ε filter 53B smoothes the image data, and sends the resulting smoothed image data to the ε filter 53C. The ε filter 53C further smoothes the smoothed image data supplied from the ε filter 53B, and supplies the smoothed image data to the lookup table 54.
[0038]
The look-up table 54 performs gradation conversion (inverse of the logarithmic conversion performed by the lookup table 52) on the smoothed image data supplied from the ε filter 53C, and obtains the resulting smoothed image data. To the linear low-pass filter 55.
[0039]
The linear low-pass filter 55 generates smoothed image data by slightly dulling the edges of the smoothed image data that has been smoothed while preserving the edges, and sends this to the subtractor 31 and the adder 33. This makes it possible to maintain the smoothness of the image in the vicinity of the edge in the subsequent image enhancement processing (processing performed by the arithmetic unit 23).
[0040]
FIG. 4 is a block diagram illustrating a detailed configuration example of the ε filter 53A.
[0041]
In the ε filter 53A, the pixels for the filtering process are one-dimensional with 17 taps, and the threshold set by the threshold setting unit 140 is ε. ₁ It is said. The ε filter 53A includes registers 101 to 116, a threshold setting unit 140, selectors 141 to 144, amplifiers 161 to 165, an adder 181, and a divider 182. Each of the registers 101 to 116 is a register that holds a pixel value for one pixel.
[0042]
The selector 141 determines the value Xm supplied from the register 116, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold ε supplied from the threshold setting unit 140. ₁ , And supplies the result of the operation to the amplifier 161. The selector 142 determines the value Xm supplied from the register 112, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold ε supplied from the threshold setting unit 140. ₁ , And supplies the operation result to the amplifier 161.
[0043]
The selector 143 calculates the value Xm supplied from the register 104, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold ε supplied from the threshold setting unit 140. ₁ , And supplies the result of the operation to the amplifier 164. The selector 144 receives the input value Xm, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold ε supplied from the threshold setting unit 140. ₁ , And supplies the operation result to the amplifier 165.
[0044]
The register 191 including the registers 101 to 103, the register 192 including the registers 105 to 107, the register 193 including the registers 109 to 111, and the register 194 including the registers 113 to 115 each set an interval. It is provided for.
[0045]
The pixel value Xn of the center pixel pn is supplied from the register 108 to the amplifier 163.
[0046]
Each of the amplifiers 161 to 165 multiplies the input pixel value X by a tap coefficient ak, and sends the calculation result to the adder 181. In this case, the tap coefficients of the amplifiers 161 and 165 are set to 1, and the tap coefficients of the amplifiers 162 to 164 are set to 2. The adder 181 adds the supplied values (the values supplied from the amplifiers 161 to 165) and supplies the result to the dividing unit 182. The division unit 182 divides the supplied value (in the present example, divides by 8) and sends this to the ε filter 53B. Note that the divider 182 may be omitted, the tap coefficients of the amplifiers 161 and 165 may be set to １ ／, and the tap coefficients of the amplifiers 162 to 164 may be set to ／.
[0047]
FIG. 5 is a block diagram illustrating a detailed configuration example of the ε filter 53B, and FIG. 6 is a block diagram illustrating a detailed configuration example of the ε filter 53C.
[0048]
As is apparent from a comparison of FIG. 4 with FIGS. 5 and 6, the ε filters 53B and 53C of FIGS. 5 and 6 have basically the same configuration as the ε filter 53A of FIG.
[0049]
That is, the ε filter 53B of FIG. 5 corresponds to the registers 101 to 116, the threshold setting unit 140, the selectors 141 to 144, the amplifiers 161 to 165, the adder 181, and the divider 182 of the ε filter 53A of FIG. , Registers 301 to 308, a threshold setting unit 340, selectors 341 to 344, amplifiers 361 to 365, an adder 381, and a divider 382. Those with corresponding names have corresponding functions.
[0050]
However, the ε filter 53B is set by the threshold setting unit 340 and supplied to the selectors 341 to 344. ₂ And the number of registers (in the case of the ε filter 53A of FIG. 6, the number of registers is 16, and in the case of the ε filter 53B of FIG. 8, the number of registers is 8). Filter 53A. In the ε filter 53B, the registers 391 to 394 for setting the interval are each constituted by one register 301, 303, 305, or 307.
[0051]
6 correspond to the registers 101 to 116, the threshold setting unit 140, the selectors 141 to 144, the amplifiers 161 to 165, the adder 181, and the divider 182 of the ε filter 53A in FIG. , Registers 401 to 404, a threshold setting unit 440, selectors 441 to 444, amplifiers 461 to 465, an adder 481, and a divider 482. Those with corresponding names have corresponding functions.
[0052]
However, as in FIG. 5 (ε filter 53B) described above, the ε filter 53C is set by the threshold setting unit 440 and supplied to the selectors 401 to 404. ₃ And the number of registers (the number of registers of the ε filter 53A is 16 and the number of registers of the ε filter 53C is 4) are different from the ε filter 53A of FIG. The ε filter 53C is not provided with a register for setting an interval.
[0053]
FIG. 7 is a block diagram showing a configuration example of the selector 141 of FIG.
[0054]
The pixel value Xm of the pixel pm (the pixel value supplied from the register 116 in this example) and the pixel value Xn of the central pixel pn supplied from the register 108 are input to the subtractor 200 and the selector 201. The subtracter 200 calculates the difference between the pixel value Xn and the pixel value Xm, and outputs the result to the absolute value converter 202. The absolute value converter 202 obtains the absolute value | Xn−Xm | of the difference between the pixel value Xn and the pixel value Xm, and outputs this to the magnitude comparator 203.
[0055]
The magnitude comparator 203 calculates the absolute value | Xn−Xm | and a predetermined threshold value ε (in this example, the threshold value ε ₁ ) And sends the comparison result to the selector 201. When the absolute value | Xn−Xm | is smaller than the predetermined threshold ε, the selector 201 selects the pixel value Xm and sends it to the amplifier 161 at the subsequent stage, and the absolute value | Xn−Xm | ε ₁ If it is larger, the pixel value Xn is selected and sent to the subsequent amplifier 161.
[0056]
Each of the selectors 142 to 144 is configured similarly to the selector 141, and sends the pixel value X of the selected pixel p to the corresponding subsequent amplifier 162, 164, or 165, respectively.
[0057]
The configurations of the selectors 341 to 344 in FIG. 5 and the selectors 441 to 444 in FIG. 6 are the same as those of the selector 141, and a description thereof will be omitted.
[0058]
Next, the non-linear smoothing process in the non-linear smoother 22 in FIG. 3 will be described with reference to the flowchart in FIG. This flowchart explains in detail the processing of steps S2 and S3 in FIG. This process is started when the input image data is supplied from the image sensor 11 to the linear low-pass filter 51 of the nonlinear smoother 22 according to a user's command.
[0059]
In step S31, the linear low-pass filter 51 acquires the input image data supplied from the image sensor 11.
[0060]
In step S32, the linear low-pass filter 51 attenuates the signal level of an extremely high frequency component of the input image data, and transmits the resulting image data to the lookup table 52.
[0061]
As described above, the linear low-pass filter 51 attenuates the signal level of an extremely high frequency component of the input image data, so that when the ε filter 53 (ε filters 53A to 53C) at the subsequent stage performs the smoothing processing, the amplitude of the amplitude is reduced. This prevents large high-frequency components from remaining on the image as point-like noise without being sufficiently smoothed. Note that the linear low-pass filter 51 is configured by applying a one-dimensional linear low-pass filter in the horizontal and vertical directions of an image, respectively, or is configured by a two-dimensional linear low-pass filter.
[0062]
In step S33, the look-up table 52 acquires the image data supplied from the linear low-pass filter 51 by the processing in step S32, and performs, for example, tone conversion such as logarithmic conversion on the obtained image data, The image data obtained as a result is sent to the ε filter 53 (ε filter 53A).
[0063]
By the way, the ε filters 53A to 53C at the subsequent stage have the threshold ε ₁ Or ε ₃ By performing the smoothing process adaptively in accordance with the amplitude of the image data, the magnitude of the amplitude of the image data generally depends on the intensity of the illumination light illuminating the imaging target. It increases in proportion to that. Therefore, the look-up table 52 performs logarithmic conversion on the image data in advance, so that the same smoothing effect can be obtained regardless of the lighting conditions when performing the smoothing processing in the subsequent ε filters 53A to 53C. I do. At the same time, the look-up table 52 can control the smoothing effect according to the pixel value, for example, by increasing the degree of smoothing in a dark area or a bright area of the image to increase the degree of enhancement in the subsequent image enhancement processing. enable.
[0064]
In step S34, the ε filter 53 (the ε filters 53A to 53C) performs a smoothing process. The details of the smoothing process will be described later with reference to FIG.
[0065]
In step S35, the look-up table 54 converts the smoothed image data supplied from the ε filter 53 (ε filter 53C) into a gradation conversion (inverse of the logarithmic conversion performed by the lookup table 52 by the processing in step S33). Conversion), and sends the resulting smoothed image data to the linear low-pass filter 55.
[0066]
In step S36, the linear low-pass filter 55 slightly smoothes the edges of the smoothed image data (the smoothed image data supplied from the lookup table 54 by the processing in step S35) while maintaining the edges. To generate the smoothed image data and send it to the subtractor 31 and the adder 33 (FIG. 1). This makes it possible to maintain the smoothness of the image in the vicinity of the edge in the image enhancement processing by the arithmetic unit 23 in the subsequent stage.
[0067]
As described above, the processing in steps S4 to S7 in FIG. 2 is further performed on the smoothed data on which the non-linear smoothing processing has been performed by the processing in FIG.
[0068]
FIG. 9 is a flowchart illustrating the smoothing process in step S34 of FIG. This process is started when image data is supplied to the first stage ε filter of the ε filter 53 (that is, in this case, the ε filter 53A) (after the process of step S33 in FIG. 8). You.
[0069]
In step S71, the ε filter 53A sets the first threshold ε ₁ And the first interval, the first-stage smoothing process is executed. The details of this processing will be described later with reference to FIG. The ε filter 53A supplies the smoothed data that has been subjected to the first smoothing process to the subsequent ε filter 53B.
[0070]
In step S72, the ε filter 53B sets the second threshold ε ₂ And the second interval, the second-stage smoothing process is executed. The ε filter 53B supplies the smoothed data that has been subjected to the second smoothing process to the subsequent ε filter 53C.
[0071]
In step S73, the ε filter 53C sets the third threshold ε ₂ And the third interval, a third-stage smoothing process is executed. The ε filter 53C supplies the lookup table 54 with the smoothed data that has been subjected to the third smoothing process as an output of the ε filter 53. As described above, the look-up table 54 performs gradation conversion on the smoothed data (the processing of step S35 in FIG. 8).
[0072]
By the processing of FIG. 9, smoothing processing is performed by the ε filters 53A to 53C provided in the ε filter 53, and smoothed data is generated.
[0073]
Specifically, the ε filter 53A applies ｛1,0,0,0,2,0,0,0,2,0,0,0,2,0,0, A tap coefficient of 0,1} is calculated, a tap coefficient of {1,0,2,0,2,0,2,0,1} is calculated by the ε filter 53B, and a {1, 0} is calculated by the ε filter 53C. Calculation of tap coefficients of 2, 2, 2, 1} is performed. As a result, the input image data substantially includes {1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15, 16, 16, 16, 16, 16, 15, 15, 14}. , 13, 12, 10, 8, 6, 6, 4, 3, 2, 1}.
[0074]
That is, by cascading three ε filters (17 tap, 9 tap, and 5 tap ε filters), it is possible to obtain substantially the same effect as the result calculated using the 27 tap nonlinear filter. At this time, the number of comparison operations can be reduced from 26 times to 12 times, and the number of multiplication operations can be reduced from 27 times to 15 times as compared with the case where the calculation is performed using a 27-tap nonlinear filter.
[0075]
In this smoothing process (the process in step S34 in FIG. 8), it is desirable to generate an image with as little change as possible in portions other than the edges. For this purpose, an extremely large filter is used in the ε filter 53A. There is a need. However, when a filter to be used is increased, multiplying all pixels by the tap coefficient ak requires increasing the number of selectors 141 to 144 and amplifiers 161 to 165 at the subsequent stage, which increases the circuit scale.
[0076]
Therefore, in the present embodiment, the register columns (registers 101 to 104, registers 105 to 108, registers 109 to 112, and registers 113 to 116) are each configured by a register for four pixels, and the following arithmetic processing is performed. On the other hand, by applying to only the first pixel (pixels of the register 104, the register 108, the register 112, and the register 116) of each register row, spatially without increasing the circuit scale of the subsequent stage. An ε filter covering a wide range is realized. However, a filter having a significant coefficient only every few pixels can expect a large smoothing effect, but generally tends to have a large side lobe, and unnecessary high frequency components remain in the smoothed image data. Easier to do.
[0077]
Therefore, the ε filter 53B and the ε filter 53C are connected in cascade (cascade) to achieve both the smoothing effect and the suppression of the side lobe.
[0078]
FIG. 10 is a flowchart for explaining the processing in each of steps S71 to S73 in FIG. 9, that is, the smoothing processing (smoothing processing in an arbitrary ε filter) of the n-th stage (n = 1, 2, or 3). It is. This processing is started when input image data is supplied to an arbitrary ε filter 53 that executes the processing.
[0079]
In step S101, each register holds data. For example, when this processing is executed in the ε filter 53A of FIG. 4, the registers 101 to 116 hold the input data. At this time, the held data is supplied to the selectors 141 to 144 as the pixel value Xm of the pixel pm. That is, the pixel values Xm held in the registers 104, 108, 112, and 116 at that time are supplied to the selectors 141 to 144, respectively. The pixel value Xn of the central pixel pn supplied from the register 108 is input to the selectors 141 to 144 and the multiplier 163.
[0080]
In step S102, the subtracters 200 of the selectors 141 to 144 calculate a difference (Xn-Xm) between the pixel value Xm of the pixel pm and the pixel value Xn of the center pixel pn.
[0081]
In step S103, the absolute value converter 202 of each of the selectors 141 to 144 obtains the absolute value | Xn−Xm | of the difference (Xn−Xm) between the difference pixel value Xn and the pixel value Xm calculated by the processing in step S102. Is sent to the magnitude comparator 203.
[0082]
In step S104, the magnitude comparators 203 of the selectors 141 to 144 determine the absolute value | Xn−Xm | and a predetermined threshold ε (in this example, the threshold ε ₁ ) And sends the comparison result to the selector 201.
[0083]
In step S105, when the absolute value | Xn−Xm | is smaller than the predetermined threshold value ε, the selector 201 of the selectors 141 to 144 selects the pixel value Xm and sends it to the amplifier 161 at the subsequent stage, where the absolute value | Xn−Xm | is a predetermined threshold ε ₁ If it is larger, the pixel value Xn is selected and sent to the amplifier 161. At this time, the pixel value supplied to the amplifier 161 is set as a pixel value X.
[0084]
In step S106, the amplifiers 161 to 165 multiply the input pixel value X by the tap coefficient ak, and send the operation result to the adder 181. That is, the amplifiers 161 to 165 weight the pixel value X. In the case of the present example, the tap coefficients of the amplifiers 161 and 165 are 1, and the tap coefficients of the amplifiers 162 to 164 are 2, so that the pixel value X is multiplied by 1 or 2.
[0085]
In step S107, the adder 181 adds the respective values supplied from the amplifiers 161 to 165, and supplies the result to the divider 182.
[0086]
In step S108, the divider 182 divides the value supplied from the adder 181 (divides by 8 in the present example), and sends this to the ε filter 53B. Then, the process returns to FIG. 9, and thereafter (step S72 in this case) is performed.
[0087]
By the processing in FIG. 10, the ε filter 53 (ε filters 53A to 53C), which is a non-linear smoothing filter, smoothes the pixel value without impairing the sharp change of the pixel value, and the image supplied from the lookup table 52. The image data can be smoothed while retaining the edges of the data (the image data supplied from the lookup table 52 by the processing in step S32 in FIG. 8). When the smoothing process (the process of FIG. 10) in the ε filter 53 is summarized, when the pixel for the filter process is one-dimensional and has 2N + 1 taps, it is expressed by Expression (1).
(Equation 1)

[0088]
That is, the ε filter 53 compares the absolute value | Xn−Xn−k | of the difference between the pixel value Xn of the center pixel pn of the filtering process and the pixel value Xn−k of the pixel pn−k with the predetermined threshold value ε. As a result, when the ε filter 53A determines that the absolute value | Xn−Xn−k | is smaller than the predetermined threshold ε, the ε filter 53A substitutes the pixel value Xn−k into wn−k and sets ak to each tap coefficient. By performing the same processing as that of the ordinary linear low-pass filter described above, the image is uniformly smoothed around the center pixel pn.
[0089]
On the other hand, when the ε filter 53 determines that the absolute value | Xn−Xn−k | is larger than the predetermined threshold ε, the ε filter 53 substitutes the pixel value Xn into wn−k, and By substituting the pixel value Xn-k with the pixel value Xn of the central pixel pn and then performing a low-pass filter process around the central pixel pn, the pixel value Xn-k is ignored and only the pixel values near the pixel value Xn are used. Perform smoothing.
[0090]
FIG. 11 is a diagram illustrating pixel values held in registers 101 to 116 of the ε filter 53A (a 17-tap ε filter).
[0091]
In FIG. 11, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are X1, X2, X3, L1, X4, X5, X6, and L2. The pixel values on the right side of the center pixel are Y1, Y2, Y3, R1, Y4, Y5, Y6, and R2.
[0092]
Usually, the scanning of the line is performed from the upper left to the lower right, so that the leftmost pixel value L2 in the figure is scanned first and the rightmost pixel value R2 is scanned last, so that the pixel value L2 Are stored in the register 116, the pixel value X6 is stored in the register 115, and the pixel value X5 is stored in the register 114. .., The pixel value Y6 is stored in the register 101, and the pixel value R2 is directly supplied to the selector 144 (see FIG. 11). A register for latching the pixel value R2 may be provided).
[0093]
As described above, the tap coefficients of the ε filter 53A are {1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 1}. The coefficients of L2 and R2 are set to 1, the coefficients of L1, C1 and R1 are set to 2, and X1 to X6 (respectively held in registers 115, 114, 113, 111, 110 and 109), and Y1 to X1 The coefficient of Y6 (respectively held in registers 107, 106, 105, 103, 102, and 101) is set to 0 (the output of each register is not supplied to adder 181).
[0094]
When the above equation (1) is applied, for example, the absolute value | C−L1 | of the difference between the pixel value C and the pixel value L1 becomes a predetermined threshold ε ₁ Is compared with a predetermined threshold ε ₁ If the pixel value is larger than the pixel value L, the pixel value L1 is replaced with the pixel value C. The pixel value of the result calculated for each of R2, Y6, Y5, Y4, R1, Y3, Y2, Y1, X1, X2, X3, L1, X4, X5, X6, and L2 by equation (1) is R2 ', Y6', Y5 ', Y4', R1 ', Y3', Y2 ', Y1', X1 ', X2', X3 ', L1', X4 ', X5', X6 ', and L2' , The pixel value C ′ output from the ε filter 53A is obtained by calculating Expression (2) (Expression (2) is an expression obtained by summarizing the processing of Steps S106 to S108 in FIG. 10). ).
(Equation 2)

[0095]
FIG. 12 is a diagram illustrating the relationship between the thresholds ε in the ε filters 53A to 53C.
[0096]
The threshold value ε in the ε filters 53A to 53C increases as the ε filter 53 is located at a later stage. That is, in this case, the threshold ε ₁ <Ε ₂ <Ε ₃ It is said.
[0097]
The larger the threshold value ε is, the more likely a summarization occurs (that is, blurring occurs and noise is removed), and the smaller the threshold value ε is, the less likely the summarization occurs (namely, blurring occurs and noise remains).
[0098]
For example, the number of taps is 3, the tap coefficients are {1, 2, 1}, L1, C, and R1 are L1 = 30, C = 100, and R1 = 80, respectively, and the threshold ε is 50. In this case, L1 ′ = 100 (R1 ′ is left as it is (R1)) according to the equation (1). In addition, C ′ = {1 × 100 + 2 × 100 + 1 × 80} / 4 is calculated by Expression (2), and the output pixel C ′ = 95. On the other hand, for example, when the number of taps is 3, the tap coefficient is {1, 2, 1}, L1 = 30, C = 100, R1 = 80, and the threshold ε is 80 (that is, In the case where only the value of the threshold ε is different from the example described above), L1 ′ and R1 ′ are set to the same values (L1, R1) from Expression (1). In addition, C ′ = {1 × 30 + 2 × 100 + 1 × 80} / 4 is calculated by equation (2), and the output pixel C ′ is set to 77.5. As described above, the larger the threshold ε is (the more the threshold ε is 80 than 50 in the present example), the more the output image is blurred (the more blur is generated).
[0099]
FIG. 13 is a diagram for explaining the pixel values held in the registers 301 to 308 of the ε filter 53B (9-tap ε filter). FIG. 14 is a diagram illustrating the pixels held in the registers 301 to 308 of the ε filter 53C. It is a figure explaining a value.
[0100]
In FIG. 13, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are X1, L1, X2, and L2. The pixel values on the right side of the center pixel are Y1, R1, Y2, and R2. As described above, since the tap coefficients of the ε filter 53B are {1, 0, 2, 0, 2, 0, 2, 0, 1}, the coefficients of L2 and R2 are 1, and the coefficients of L1, C1, and R1 are The coefficient is 2, and the coefficients of X1, X2, Y1, and Y2 (stored in registers 307, 305, 303, and 301, respectively) are 0.
[0101]
In the ε filter 53C of FIG. 14, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are L1 and L2. The pixel values on the right side of the center pixel are R1 and R2. As described above, since the tap coefficients of the ε filter 53C are {1, 2, 2, 2, 1}, the coefficients of L2 and R2 are set to 1, and the coefficients of L1, C1 and R1 are set to 2.
[0102]
Thus, the interval of the ε filter 53A is 4 (= 2 ² ) (The number of registers for setting the interval is three), and the interval of the ε filter 53B is 2 (= 2 ¹ ) (The number of registers for setting the interval is one), and the interval of the ε filter 53C is 1 (= 2 ⁰ ) (The number of registers for setting the interval is 0).
[0103]
Here, the output pixel value C ′ is obtained by applying the above-described equations (1) and (2) to each pixel value. At this time, the output pixel value C ′ calculated by the ε filter 53B has more noise than the ε filter 53A having a large number of taps. The output pixel value C ′ calculated by the ε filter 53C has more noise than the ε filter 53B having a large number of taps. That is, the magnitude of the noise is small in the ε filter 53A having a large number of taps, and increases as the ε filter 53C (the number of taps) in the subsequent stage is small.
[0104]
Therefore, as shown in FIG. 12, the threshold value ε of the ε filter 53 is changed by the number of taps of the ε filter (in the ε filter 53A having a large number of taps, the threshold value ε ₁ In the ε filter 53C with a small number of taps, the threshold ε ₃ Is increased), it is possible to suppress noise of all the output pixel values C ′ of the ε filters 53A to 53C.
[0105]
FIG. 15 is a diagram illustrating image data to be subjected to smoothing processing.
[0106]
The input image data on the left side in the figure is subjected to a smoothing process by the ε-filter 53 to be image data as shown on the right side in the figure.
[0107]
That is, when the absolute value of the difference between the pixel values before and after the steep edge exceeds a predetermined threshold value ε, the center pixel pn is set as the center, and the pixel value Xm of the pixel pm is set to the center pixel pn when performing the low-pass filtering. By performing low-pass filter processing in place of the pixel value Xn, smoothing is performed in the vicinity of the pixel value Xn. On the other hand, when performing low-pass filter processing centering on the pixel pm, smoothing is performed in the vicinity of the pixel value Xm.
[0108]
At this time, the ε filter 53A outputs the pixel value X of the pixel p in the edge portion substantially as it is, for the pixel p in the edge portion where the pixel p close to the pixel value X does not exist within the range for performing the filtering process, The sharp change of the edge is stored as it is (Equation (1)). Incidentally, the ε filter 53A is, similarly to the linear low-pass filter 51, configured by applying a one-dimensional ε filter in the horizontal direction and the vertical direction of the image, respectively, and configured by a two-dimensional ε filter. There is.
[0109]
FIG. 16 is a diagram illustrating the frequency response of the ε filters 53A to 53C.
[0110]
In FIG. 16, the horizontal axis represents the frequency normalized with the Nyquist frequency being 0.5, and the vertical axis represents the filter frequency response. 53A, 53B, and 53C indicate frequency responses corresponding to the ε filter 53A, the ε filter 53B, and the ε filter 53C, respectively.
[0111]
By setting the interval (decimation) of the number of taps of the ε filter 53 (for example, ε filter 53A) to a power of 2, as shown in FIG. 16, the frequency response waveform of the ε filter 53A (for example, interval = 4) Can be canceled by the frequency response of the ε filter 53B (for example, interval = 2) (the peak point of the side lobe in the frequency response waveform of the ε filter 53A and the frequency response waveform of the ε filter 53B). Are canceled because the 0 points in are the same). That is, the side lobe of the previous ε-filter 53 can be suppressed by the next ε-filter 53, so that both the smoothing effect and the suppression of the side lobe can be achieved.
[0112]
As described above, the smoothed image data smoothed by the ε filter 53 is calculated by the calculation unit 23 and supplied to the camera signal processing circuit 13 as an output of the image processing circuit 12. At this time, each pixel value y (i, j) of the output image data output from the image processing circuit 12 is represented by Expression (3).
[Equation 3]

[0113]
In equation (3), the gain coefficient g (i, j) is replaced with a uniform gain coefficient G for the entire screen, and each pixel value s (i, j) of the smoothed image data is replaced with the dynamic range of the input image data. When replaced by the median value or the average value C of all pixel values, this equation (3) is expressed as equation (4) showing a conventional contrast enhancement method by tone curve adjustment.
(Equation 4)

[0114]
In the equation (3), the gain coefficient g (i, j) is replaced with a uniform gain coefficient G for the entire screen, and each pixel value s (i, j) of the smoothed image data is replaced with each of the input image data. When the pixel values x (i, j) are replaced with the pixel values f (i, j) of the image data obtained by performing the linear low-pass filter processing, the expression (3) is expressed by the conventional high-frequency component enhancement method using an unsharp mask. Is expressed as shown in Expression (5).
(Equation 5)

[0115]
As described above, the conventional contrast enhancement method is a method in which image enhancement processing is performed independently for each pixel of an image, and the conventional high-frequency component enhancement method is a method in which a peripheral pixel existing around a central pixel is compared with a peripheral pixel existing around the central pixel. In contrast to the method of performing image enhancement processing based on a relative level difference, the method according to the present embodiment integrates a conventional contrast enhancement method and a high-frequency component enhancement method using a nonlinear smoothing filter. This enables a higher quality image enhancement process.
[0116]
Also, a method using a conventional nonlinear smoothing filter (amplifying and displaying a portion other than the edge by amplifying a portion other than the edge of the input image data while preserving an edge having a sharp change in pixel value). On the other hand, the method according to the present embodiment sets the interval of the number of taps of the ε filter 53 to a power of 2 and changes the threshold ε of the ε filter 53 in accordance with the band of the filter. The amount of calculation can be reduced even when a portion other than the sharp edge is smoothed while the sharply changing edge is preserved.
[0117]
As described above, the threshold ε is set according to the band (in the case of a narrow band filter (ε filter with a large number of taps), the threshold ε is reduced, and in the case of a wide band filter (an ε filter with a small number of taps), The threshold value ε is increased), the ε filters are cascaded in order from the narrow band filter to the wide band filter, and the interval of the ε filter (substantially, the extraction interval of the pixel to be processed) is set to a power of two, whereby the pixel The amount of calculation can be reduced even when a portion other than the edge is smoothed while an edge having a sharp change in value is preserved.
[0118]
Further, it is possible to obtain an output image in which noise hardly occurs and the amount of calculation is suppressed.
[0119]
Further, an image in which side lobes are suppressed can be output.
[0120]
Further, in the above-described embodiment, the case where the present invention is applied to the video camera 1 has been described. However, the present invention is not limited to this, and various other images such as an electronic still camera, a printer, a display, and a computer may be used. The present invention can be widely applied to a processing apparatus. In this case, when correcting the image contrast, the computer can obtain a high-quality contrast-corrected image while maintaining the dynamic range, and when synthesizing images obtained under different lighting conditions, the respective contrasts can be improved. Only the difference between the components can be corrected, and a natural synthesized image can be generated.
[0121]
In the above example, the number of ε filters included in the non-linear smoother 22 is three (ε filter 53A, ε filter 53B, and ε filter 53C), but any number of ε filters 53 are provided. You may do so. Of course, in this case, if the N-th ε filter is the ε filter 53N, the threshold ε of the ε filter 53A ₁ Is minimized, and the threshold ε of the ε filter 53N connected at the subsequent stage is _N Is the maximum (ε ₁ <Ε _N ).
[0122]
In the above example, the ε filter is used. However, not only the ε filter but also a median filter or the like may be used.
[0123]
Note that, in this specification, the steps describing each flowchart refer to not only processes performed in chronological order according to the described order, but also processes performed in parallel or individually even if not necessarily performed in chronological order. Is also included.
[0124]
【The invention's effect】
As described above, according to the present invention, the amount of calculation can be suppressed even when a portion other than the edge is smoothed while an edge having a sharp change in pixel value is stored. In particular, according to the present invention, it is possible to output an image in which noise is less likely to occur and side lobes are suppressed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a video camera to which the present invention is applied.
FIG. 2 is a flowchart illustrating image processing in the image processing circuit of FIG. 1;
FIG. 3 is a block diagram illustrating a functional configuration of the nonlinear smoother in FIG. 1;
FIG. 4 is a block diagram illustrating a configuration example of an ε filter of FIG. 2;
FIG. 5 is a block diagram showing a configuration example of another ε filter of FIG. 2;
FIG. 6 is a block diagram illustrating a configuration example of still another ε filter of FIG. 2;
FIG. 7 is a block diagram illustrating a configuration example of a selector of FIG. 4;
FIG. 8 is a flowchart illustrating a nonlinear smoothing process in the nonlinear smoother of FIG. 2;
FIG. 9 is a flowchart illustrating a process in step S34 of FIG. 8;
FIG. 10 is a flowchart illustrating processing in steps S71 to S73 of FIG. 9;
FIG. 11 is a diagram illustrating pixel values held in the register of FIG. 4;
FIG. 12 is a diagram illustrating a relationship between threshold values in each ε filter of FIG. 3;
FIG. 13 is a diagram illustrating pixel values held in the register of FIG. 5;
FIG. 14 is a diagram illustrating pixel values held in the register of FIG. 6;
FIG. 15 is a diagram illustrating image data to be subjected to smoothing processing.
FIG. 16 is a diagram illustrating a relationship between the number of taps and a frequency response.
[Explanation of symbols]
Reference Signs List 1 video camera, 11 image pickup device, 12 image processing circuit, 21 delay circuit, 22 nonlinear smoother, 23 operation unit, 31 subtractor, 32 multiplier, 33 adder, 51 linear low-pass filter, 52 lookup table, 53 ε filter, 54 look-up table, 55 linear low-pass filter, 101 to 116 registers, 141 to 144 selector, 161 to 165 amplifier, 181 adder, 182 divider, 140 threshold setting unit, 200 subtractor, 201 selector, 202 Absolute value converter, 203 large / small comparator

Claims (7)

In an image processing device that processes input image data,
Smoothing the input image data, comprising a smoothing means to generate smoothed image data,
The image processing apparatus according to claim 1, wherein the smoothing means includes a plurality of non-linear filters connected in tandem and having different parameters for controlling a degree of smoothing and intervals related to bands. 2. The image processing apparatus according to claim 1, wherein the smoothing unit adaptively changes a degree of smoothing based on a difference value between a central pixel and a neighboring pixel. 2. The image processing apparatus according to claim 1, wherein parameter values of the non-linear filter are larger in a latter stage than in a preceding stage. 3. 2. The image processing apparatus according to claim 1, wherein the value of the interval of the nonlinear filter is smaller in a subsequent stage than in a preceding stage. The image processing apparatus according to claim 4, wherein an interval of the nonlinear filter is a power of two. The image processing device according to claim 1, wherein the nonlinear filter is an ε filter or a median filter. In an image processing method of an image processing device that processes input image data,
Smoothing the input image data, including a smoothing step of generating smoothed image data,
An image processing method, wherein the processing of the smoothing step is performed by a plurality of non-linear filters connected in tandem and having different parameters for controlling the degree of smoothing and intervals related to bands.

Images