JP4556319B2 - Image processing apparatus and method, and recording medium - Google Patents

Description

【００６０】
なお、図１８は、広ダイナミックレンジ画像の輝度のダイナミックレンジと、変換後の狭ダイナミックレンジ画像の輝度のダイナミックレンジの関係を説明するための図であり、同図の横軸は、入力される広ダイナミックレンジ画像の輝度Ｙの対数値logＹを示し、縦軸は変換後の狭ダイナミックレンジ画像の輝度Ｙｍを示している。図１９乃至図２１も同様である。また、Ｈｍ，Ｌｍは、それぞれ狭ダイナミックレンジの最大輝度または最小輝度を表しており、予め設定されている値である。
【００６１】
ステップＳ４４において、高域関数決定部９２は、図１８に示す２点（logＭ，Ｍｍ），（logＨ，Ｈｍ）を通る直線Ｃに相当する高域対数関数ＹｍＨ（Ｙ）を生成して関数合成部９４に出力する。
ＹｍＨ（Ｙ）＝αＨlogＹ＋βＨ ・・・（４）
【数２】

【００６２】
ステップＳ４５において、低域関数決定部９３は、図１８に示す２点（logＬ，Ｌｍ），（logＭ，Ｍｍ）を通る直線Ｂに相当する低域対数関数ＹｍＬ（Ｙ）を生成して関数合成部９４に出力する。
ＹｍＬ（Ｙ）＝αＬlogＹ＋βＬ ・・・（５）
【数３】

【００６３】
なお、ステップＳ４３，Ｓ４４，Ｓ４５の処理の順序は、適宜入れ替えてもよいし、平行して同時に実行するようにしてもよい。
【００６４】
ステップＳ４６において、関数合成部９４は、 図１８の直線Ａに相当する対数関数Ｙｍ（Ｙ）のＹに注目輝度Ｍを代入し、得られたＹｍ（Ｍ）と最適注目輝度Ｍｍを比較して、その比較結果に基づき、注目輝度・最適注目輝度対に対応する点（logＭ，Ｍｍ）が、直線Ａよりも上に位置するか、直線Ａよりも下に位置するか、または直線Ａに一致するかを判定する。
【００６５】
比較結果が、ＭｍがＹｍ（Ｍ）よりも大きい場合には、図１８に示すように、注目輝度・最適注目輝度対に対応する点（logＭ，Ｍｍ）は直線Ａよりも上に位置すると判定されて、処理はステップＳ４７に進む。ステップＳ４７において、関数合成部９４は、図１９に示すような上に凸のマッピング関数となるように、図１９の横軸上のパラメータＣＬ，ＣＨを次式（６）のように設定する。
【数４】

・・・（６）
【００６６】
ステップＳ５０において、関数合成部９４は、高域対数関数ＹｍＨ（Ｙ）と低域対数関数ＹｍＬ（Ｙ）を滑らかに連結してマッピング関数を生成する。具体的には、入力輝度Ｙ、すなわち、広ダイナミックレンジ画像の輝度Ｙを、最小輝度Ｌよりも小さい領域、最小輝度Ｌ以上であってＣＬよりも小さい領域、ＣＬ以上であってＣＨよりも小さい領域、ＣＨ以上で最大輝度Ｈよりも小さい領域、または、最大輝度Ｈ以上である領域に分け、各領域に対応するマッピング関数ｆ（Ｙ）を生成する（詳細は図２２を参照して後述する）。
【００６７】
ステップＳ４６での比較結果が、ＭｍがＹｍ（Ｍ）と等しい場合、注目輝度・最適注目輝度対に対応する点（logＭ，Ｍｍ）は直線Ａに一致すると判定されて、処理はステップＳ４８に進む。ステップＳ４８において、関数合成部９４は、マッピング関数のパラメータＣＬ，ＣＨを次式（７）のように設定する。
ＣＬ＝Ｌ
ＣＨ＝Ｈ ・・・（７）
【００６８】
ステップＳ４６での比較結果が、ＭｍがＹｍ（Ｍ）よりも小さい場合には、図２０に示すように、注目輝度・最適注目輝度対に対応する点（logＭ，Ｍｍ）は直線Ａよりも下に位置すると判定されて、処理はステップＳ４９に進む。ステップＳ４９において、関数合成部９４は、図２１に示すような下に凸のマッピング関数となるようにパラメータＣＬ，ＣＨを次式（８）のように設定する。
【数５】

・・・（８）
【００６９】
ステップＳ５０における関数合成部９４の処理の詳細について、図２２のフローチャートを参照して説明する。ステップＳ６１，Ｓ６３，Ｓ６５，Ｓ７２における大小比較判定により、マッピング関数ｆ（Ｙ）の変数である入力輝度Ｙが５つの領域のうちのいずれかに分類される。
【００７０】
ステップＳ６１の処理を経て、最小輝度Ｌよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ６２において、次式（９）に示すようにマッピング関数ｆ（Ｙ）が定義される。
ｆ（Ｙ）＝Ｌｍ Ｙ＜Ｌ ・・・（９）
【００７１】
ステップＳ６１，Ｓ６３の処理を経て、最小輝度Ｌ以上であってＣＬよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ６４において、次式（１０）に示すように、マッピング関数ｆ（Ｙ）が低域対数関数ＹｍＬ（Ｙ）によって定義される。
ｆ（Ｙ）＝ＹｍＬ（Ｙ）＝αＬlogＹ＋βＬ Ｌ≦Ｙ＜Ｌ・・・（１０）
【００７２】
ステップＳ６１，Ｓ６３，Ｓ６５の処理を経て、ＣＬ以上であってＣＨよりも小さい領域に分類された場合、処理はステップＳ６６に進む。
【００７３】
ステップＳ６６において、ＣＬ−２Ｍ＋ＣＨが０よりも大きいか否かが判定される。ＣＬ−２Ｍ＋ＣＨが０よりも大きいと判定された場合、処理はステップＳ６７に進む。ステップＳ６７において、媒介変数ｔが次式（１１）によって定義される。
【数６】

・・・（１１）
【００７４】
ステップＳ６８において、入力輝度Ｙに対し、次式（１２）に示すように、マッピング関数ｆ（Ｙ）が定義される。
【数７】

ＣＬ≦Ｙ＜ＣＨ・・・（１２）
【００７５】
ステップＳ６６において、ＣＬ−２Ｍ＋ＣＨが０よりも大きくないと判定された場合、処理はステップＳ６９に進む。ステップＳ６９において、ＣＬ−２Ｍ＋ＣＨが０よりも小さいか否かが判定される。ＣＬ−２Ｍ＋ＣＨが０よりも小さいと判定された場合、処理はステップＳ７０に進む。ステップＳ７０において、媒介変数ｔが次式（１３）によって定義される。
【数８】

・・・（１３）
【００７６】
ステップＳ６９において、ＣＬ−２Ｍ＋ＣＨが０よりも小さくない、すなわち、ＣＬ−２Ｍ＋ＣＨ＝０であると判定された場合、処理はステップＳ７１に進む。ステップＳ７１において、媒介変数ｔが次式（１４）によって定義される。
ｔ＝（Ｙ−ＣＬ）／２（Ｍ−ＣＬ） ・・・（１４）
【００７７】
ステップＳ６１，Ｓ６３，Ｓ６５，Ｓ７２の処理を経て、ＣＨ以上であって最大輝度Ｈよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ７３において、次式（１５）に示すように、マッピング関数ｆ（Ｙ）が高域対数関数ＹｍＨ（Ｙ）によって定義される。
ｆ（Ｙ）＝ＹｍＨ（Ｙ）＝αＨlogＹ＋βＨ ＣＨ≦Ｙ＜Ｈ・・・（１５）
【００７８】
ステップＳ６１，Ｓ６３，Ｓ６５，Ｓ７２の処理を経て、最大輝度Ｈ以上である領域に分類された入力輝度Ｙに対しては、ステップＳ７４において、次式（１６）に示すように、マッピング関数ｆ（Ｙ）が高域対数関数ＹｍＨ（Ｙ）によって定義される。
ｆ（Ｙ）＝Ｌｍ Ｈ＜Ｙ ・・・（１６）
【００７９】
以上説明したように、マッピング関数生成部８２の一連の処理によって、マッピング関数ｆ（Ｙ）が生成される。
【００８０】
次に、生成されたマッピング関数ｆ（Ｙ）に広ダイナミックレンジ画像信号を適用して狭ダイナミックレンジ画像信号を生成するマッピング部８３の処理について、図２３のフローチャートを参照して説明する。
【００８１】
ステップＳ８１において、マッピング部８３は、記憶部５０から広ダイナミックレンジ画像信号を取得する。また、マッピング部８３は、マッピング関数生成部８２からマッピング関数ｆ（Ｙ）を取得する。
【００８２】
ステップＳ８２において、マッピング部８３は、広ダイナミックレンジ画像の全て画素の輝度信号Ｙ(x,y)を、マッピング関数ｆ（Ｙ）に順次代入して狭ダイナミックレンジ画像信号を生成する。
【００８３】
以上説明したように画像処理部１１の一連の処理によれば、簡易生成されてた複数の狭ダイナミックレンジの画像がユーザに提示され、そのうちの被写体の階調が最も適切に表示されている画像がユーザによって選択されて、選択された画像に基づいて注目輝度・最適注目輝度対（Ｍ，Ｍｍ）が導き出される。さらに、注目輝度・最適注目輝度対（Ｍ，Ｍｍ）に基づいてマッピング関数ｆ（Ｙ）が生成されて、広ダイナミックレンジ画像信号がマッピング関数ｆ（Ｙ）によって狭ダイナミックレンジ画像信号に変換されるので、画像中の被写体が適切な階調で表現される狭ダイナミックレンジの画像を得ることができる。
【００８４】
なお、上述した説明においては、注目輝度・最適注目輝度対（Ｍ，Ｍｍ）を１つだけ決定して、以降の処理を実行するようにしたが、２つ以上の注目輝度・最適注目輝度対を決定してマッピング関数を生成するようにしてもよい。
【００８５】
次に、画像処理部１１が最適露出情報を設定する第２の動作例について、図２４および図２５を参照して説明する。図８乃至図１１を参照して上述した第１の動作例では、簡易生成された複数の狭ダイナミックレンジ画像のうちの１つをユーザに選択させるようにしたが、第２の動作例においては、所定の露出比率ｒを適用して簡易生成した狭ダイナミックレンジ画像を１枚だけ表示し、その画像の内で注目すべき範囲（例えば、被写体が表示されている範囲）をユーザに指定させるようにし、その指定された範囲（図２４（Ｂ）のマスク１０３）を示す情報を最適露出情報に設定して輝度対検出部８１に供給するようにする。
【００８６】
図２４は、簡易生成された１枚の狭ダイナミックレンジ画像を表示する表示部１３の表示例を示している。同図（Ａ）に示すように、表示部１３の画像表示・指定範囲描画領域１０１には、簡易生成された狭ダイナミックレンジ画像１０２が１枚だけ表示される。画像表示・指定範囲描画領域１０１に表示された狭ダイナミックレンジ画像１０２には、同図（Ｂ）に示すような、ユーザによって指定された範囲を表すマスク１０３が重畳して表示される。
【００８７】
図２５は、狭ダイナミックレンジ画像１０２に重畳して表示されるマスク１０３をユーザが設定するとき操作するGUIの表示例を示している。GUIとしての範囲指定パネル１１１には、マスク１０３の設定を開始するときクリックされるペンボタン１１２、設定されたマスク１０３の設定を解除する消しゴムボタン１１３、および、設定したマスク１０３を確定する完了ボタン１１４が設けられている。
【００８８】
ユーザは、ペンボタン１１２をクリックした後、マウス等の入力デバイスを用いてマスク１０３の範囲を描画し、完了ボタン１１４をクリックすることにより、描画したマスク１０３の範囲を確定させる。これに対応して、確定されたマスク１０３の範囲を示す情報が、最適露出情報として輝度対検出部８１に供給される。
【００８９】
第２の動作例において、輝度対検出部８１は、最適注目輝度Ｍｍとして、上述したステップＳ３１（図１３）の処理と同様に、予め設定されている狭ダイナミックレンジ画像の中間輝度Ｙｎｍを、記憶部５０に記憶されている画像処理用アプリロケーションプログラムから取得する。
【００９０】
一方の注目輝度Ｍを求めるために輝度対検出部８１は、広ダイナミックレンジ画像のうち、マスク１０３の範囲に含まれる画素の輝度信号を取得して画素集合Ｇ１とし、画素集合Ｇ１のうち、所定の飽和レベルを上回る輝度の画素、および所定のノイズレベルを下回る輝度の画素を除外して画素集合Ｇ２とする。さらに、輝度対検出部８１は、画素集合Ｇ２の輝度のヒストグラムを生成して、最も頻度が高い輝度を注目輝度Ｍに設定する。
【００９１】
このようにして設定された注目輝度・最適注目輝度対（Ｍ，Ｍｍ）は、マッピング関数生成部８２に供給されて、上述した一連の処理と同様に処理される。
【００９２】
以上のように、最適露出情報を設定する第２の動作例によれば、ユーザが最適な輝度で表示してほしい被写体の範囲を指示するという直感的な操作によって、所望の狭ダイナミックレンジ画像を作成することが可能となる。
【００９３】
次に、撮像装置１から画像処理装置２に対して、カラーの広ダイナミックレンジ画像信号（３原色信号Ｒ，Ｇ，Ｂ）が供給される場合に対応する画像処理部１１の構成例（以下、画像処理部１１の第２の構成例と記述する）および動作について、図２６を参照して説明する。画像処理部１１の第２の構成例においては、カラー画像の色バランスが損なわれないように階調変換が行われる。
【００９４】
図２６は、画像処理部１１の第２の構成例を示している。輝度信号生成部１２１は、次式（１７）を用いて、入力される広ダイナミックレンジ画像の各画素の３原色信号Ｒ，Ｇ，Ｂを用いて輝度Ｙを生成し、輝度対検出部１２２、色補正関数生成部１２３、マッピング部１２４、マッピング関数生成部１２５、マッピング部１２６、および、べき乗演算部１２７に出力する。
輝度Ｙ＝ｋｒ・Ｒ＋ｋｇ・Ｇ＋ｋｂ・Ｂ ・・・（１７）
【００９５】
ここで、ｋｒ，ｋｇ，ｋｂは定数であり、例えば、ｋｒ＝０．３、ｋｇ＝０．６、ｋｂ＝０．１とする。
【００９６】
輝度対検出部１２１は、輝度信号生成部１２１からの広ダイナミックレンジ画像の各画素の輝度Ｙ、および、上述した画像処理部１１の第１または第２の動作例によって取得した最適露出情報に基づき、注目輝度・最適注目輝度対（Ｍ，Ｍｍ）を検出して色補正関数生成部１２３およびマッピング関数生成部８２に出力する。
【００９７】
色補正関数生成部１２３は、輝度信号生成部１２１からの広ダイナミックレンジ画像の各画素の輝度Ｙ、および、輝度対検出部８１からの注目輝度・最適注目輝度対（Ｍ，Ｍｍ）に基づき、輝度Ｙに対応する色補正量γを示す色補正関数ｆＣ（Ｙ）を生成し、マッピング部１２４に出力する。
【００９８】
色補正関数生成部１２３が色補正関数ｆＣ（Ｙ）を生成する処理について、より詳細に説明する。当該色補正関数生成処理は、図１６乃至図２２を参照して上述したマッピング関数生成部８２のマッピング関数生成処理とほぼ同様であり、図２２を用いて説明したステップＳ５０（図１６）における処理だけが若干異なる。よって、当該色補正関数生成処理のうち、マッピング関数生成処理のうちのステップＳ５０における処理に相当する処理についてのみ、図２７を参照して説明する。
【００９９】
色補正関数生成部１２３は、ステップＳ９１，Ｓ９３，Ｓ９５，Ｓ１０２における大小比較判定により、色補正関数ｆＣ（Ｙ）の変数である入力輝度Ｙを５つの領域のうちのいずれかに分類する。
【０１００】
ステップＳ９１の処理を経て、最小輝度Ｌよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ９２において、次式（１７）に示すように色補正関数ｆＣ（Ｙ）が定義される。
ｆＣ（Ｙ）＝αＬ Ｙ＜Ｌ ・・・（１７）
【０１０１】
ステップＳ９１，Ｓ９３の処理を経て、最小輝度Ｌ以上であってＣＬよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ９４において、次式（１８）に示すように、色補正関数ｆＣ（Ｙ）が定義される。
ｆＣ（Ｙ）＝αＬ Ｌ≦Ｙ＜Ｌ・・・（１８）
【０１０２】
ステップＳ９１，Ｓ９３，Ｓ９５の処理を経て、ＣＬ以上であってＣＨよりも小さい領域に分類された場合、処理はステップＳ９６に進む。
【０１０３】
ステップＳ９６において、ＣＬ−２Ｍ＋ＣＨが０よりも大きいか否かが判定される。ＣＬ−２Ｍ＋ＣＨが０よりも大きいと判定された場合、処理はステップＳ９７に進む。ステップＳ９７において、媒介変数ｔが式（１１）によって定義される。
【０１０４】
ステップＳ９８において、入力輝度Ｙに対し、次式（１９）に示すように、色補正関数ｆＣ（Ｙ）が定義される。
ｆＣ（Ｙ）＝（１−ｔ）αＬ＋ｔαＨ ＣＬ≦Ｙ＜ＣＨ ・・・（１９）
【０１０５】
ステップＳ９６において、ＣＬ−２Ｍ＋ＣＨが０よりも大きくないと判定された場合、処理はステップＳ９９に進む。ステップＳ９９において、ＣＬ−２Ｍ＋ＣＨが０よりも小さいか否かが判定される。ＣＬ−２Ｍ＋ＣＨが０よりも小さいと判定された場合、処理はステップＳ１００に進む。ステップＳ１００において、媒介変数ｔが式（１３）によって定義される。
【０１０６】
ステップＳ９９において、ＣＬ−２Ｍ＋ＣＨが０よりも小さくない、すなわち、ＣＬ−２Ｍ＋ＣＨ＝０であると判定された場合、処理はステップＳ１０１に進む。ステップＳ１０１において、媒介変数ｔが式（１４）によって定義される。
【０１０７】
ステップＳ９１，Ｓ９３，Ｓ９５，Ｓ１０２の処理を経て、ＣＨ以上であって最大輝度Ｈよりも小さい領域に分類された入力輝度Ｙに対しては、ステップＳ１０３において、次式（２０）に示すように、色補正関数ｆＣ（Ｙ）が定義される。
ｆＣ（Ｙ）＝αＨ ＣＨ≦Ｙ＜Ｈ・・・（２０）
【０１０８】
ステップＳ９１，Ｓ９３，Ｓ９５，Ｓ１０２の処理を経て、最大輝度Ｈ以上である領域に分類された入力輝度Ｙに対しては、ステップＳ１０４において、次式（２１）に示すように、色補正関数ｆＣ（Ｙ）が定義される。
ｆＣ（Ｙ）＝αＨ Ｈ＜Ｙ ・・・（２１）
【０１０９】
以上説明したような色補正関数生成部１２３の処理によって、色補正関数ｆＣ（Ｙ）が生成される。
【０１１０】
マッピング部１２４は、輝度信号生成部１２１からの各画素の輝度Ｙを、色補正関数ｆＣ（Ｙ）に適用して色補正量γを演算し、べき乗演算部１２７、１２８Ｒ，１２８Ｇ，１２８Ｂに出力する。
【０１１１】
マッピング関数生成部１２５は、上述したマッピング関数生成部８２と同様に、広ダイナミックレンジ画像の各画素の輝度Ｙおよび注目輝度・最適注目輝度対（Ｍ，Ｍｍ）に基づき、マッピング関数ｆ（Ｙ）を生成してマッピング部１２６に出力する。マッピング部１２６は、広ダイナミックレンジ画像の各画素の輝度Ｙを、マッピング関数生成部１２５からのマッピング関数ｆ（Ｙ）に適用して狭ダイナミックレンジ画像の各画素の輝度Ｙｍを生成し、スケーリング部１２９Ｒ，１２９Ｇ，１２９Ｂに出力する。
【０１１２】
べき乗演算部１２７は、広ダイナミックレンジ画像の各画素の輝度Ｙをγ乗して、得られた補正輝度Ｙγをスケーリング部１２９Ｒ，１２９Ｇ，１２９Ｂに出力する。べき乗演算部１２８Ｒは、広ダイナミックレンジ画像の各画素の赤色信号Ｒをγ乗して、得られた補正赤色信号Ｒγをスケーリング部１２９Ｒに出力する。べき乗演算部１２８Ｇは、広ダイナミックレンジ画像の各画素の緑色信号Ｇをγ乗して、得られた補正緑色信号Ｇγをスケーリング部１２９Ｇに出力する。
べき乗演算部１２８Ｂは、広ダイナミックレンジ画像の各画素の青色信号Ｂをγ乗して、得られた補正青色信号Ｂγをスケーリング部１２９Ｂに出力する。
【０１１３】
スケーリング部１２９Ｒ乃至１２９Ｂは、それぞれ、次式（２２）乃至（２４）を用い、狭ダイナミックレンジ画像の各画素の赤色信号Ｒｍ、緑色信号Ｇｍ、または青色信号Ｂｍを演算する。
Ｒｍ＝Ｒγ・Ｙｍ／Ｙγ ・・・（２２）
Ｇｍ＝Ｇγ・Ｙｍ／Ｙγ ・・・（２３）
Ｂｍ＝Ｂγ・Ｙｍ／Ｙγ ・・・（２４）
【０１１４】
以上説明したように、画像処理部１１の第２の構成例よれば、入力された広ダイナミックレンジ画像の３原色信号Ｒ，Ｇ，Ｂに対し、ダイナミックレンジを圧縮する度合いに応じて自然な色バランスとなるように補正を施すので、広ダイナミックレンジのカラー画像を、色バランスが不自然ではない狭ダイナミックレンジのカラー画像に変換することが可能となる。
【０１１５】
次に、図２８は、本発明を適用したディジタルカメラの構成例を示している。
このディジタルカメラ１４０は、言わば、図１に示した画像処理システムを１つの筐体に納めたものであり、被写体を広ダイナミックレンジ画像信号として撮像し、適宜、狭ダイナミックレンジ画像信号に変換して内蔵するメモリ１４８に記録する。
【０１１６】
ディジタルカメラ１４０は、被写体の光画像を集光するレンズ１４１、光画像の光量を調整する絞り１４２、集光された光画像を光電変換して広ダイナミックレンジの電気信号に変換するCCDイメージセンサ１４３、CCDイメージセンサ１４３からの電気信号をサンプリングすることによってノイズを低減させるCDS(Corelated Double Sampling)１４４、および、アナログの電気信号をディジタル化するＡ／Ｄコンバータ１４５、ディジタル化された電気信号を画像信号に変換したり、ダイナミックレンジを圧縮したりする画像信号処理用プロセッサと画像用RAMよりなるDSP(Digital Signal Processor)１４６から構成される。
【０１１７】
また、ディジタルカメラ１４０は、DSP１４６が処理した画像信号を圧縮符号化してメモリ１４８に記録し、また、読み出して伸張し、DSP１４６に供給するCODEC(Compression/Decompression)１４７、DSP１４６が処理した画像信号をアナログ化するＤ／Ａコンバータ１４９、アナログ化された画像信号を後段の表示部１５１に適合する形式のビデオ信号にエンコードするビデオエンコーダ１５０、および、ビデオ信号に対応する画像を表示することによりファインダとして機能するLCD(Liquid Crystal Display)等よりなる表示部１５１から構成される。
【０１１８】
さらに、ディジタルカメラ１４０は、ドライブ１５３を制御して、磁気ディスク１５４、光ディスク１５５、光磁気ディスク１５６、または半導体メモリ１５７に記憶されている制御用プログラムを読み出して、読み出した制御用プログラム、操作部１５８から入力されるユーザからのコマンド等に基づいて、ディジタルカメラ１４０の全体を制御するCPUなどよりなる制御部１５２、ユーザがシャッタタイミングやその他のコマンドを入力する操作部１５８、および、CCDイメージセンサ１４３乃至DSP１４６の動作タイミングを制御するタイミングジェネレータ１５９から構成される。
【０１１９】
ディジタルカメラ１４０においては、DSP１４６が、上述した画像処理システムの画像処理部１１に相当し、広ダイナミックレンジ画像信号を狭ダイナミックレンジ画像信号に変換する処理を実行する。
【０１２０】
なお、ディジタルカメラ１４０を構成する操作部１５８および表示部１５１は、図１の画像処理システムの画像処理装置２（パーソナルコンピュータ等よりなる）を構成する操作入力部１２および表示部１３に比較して、小型のものが用いられるので、より簡単にユーザが各種の操作入力を実行できるようにする必要がある。
【０１２１】
そこで、ディジタルカメラ１４０では、全体的な処理の順序を、図７のフローチャートにより説明した図１の画像処理システムの処理の順序とは異なり、広ダイナミックレンジ画像信号を取得する前に、最適露出情報を設定するようになされている。
【０１２２】
ディジタルカメラ１４０の第１の動作例について、図２９のフローチャートを参照して説明する。ステップＳ１１１において、撮像する画像の構図がユーザによって選択される。選択された画像の構図（後述するガイド１７２に相当する）が最適露出情報として設定される。
【０１２３】
当該最適露出情報を設定する処理について、図３０乃至図３２を参照して説明する。図３０は、ファインダとして機能する表示部１５１の画像表示エリア１７１の表示例を示している。同図に示すように、画像表示エリア１７１には、被写体の画像に重畳して太線等によって示されるガイド１７２や破線によって示される補助線が重畳して表示される。なお、ガイド１７２の形状としては、複数のパターンが用意されており、操作部１５８に設けられたガイド選択パネル１８１（図３２）がユーザによって操作される毎、その形状が切り替わるようになされている。
【０１２４】
図３１（Ａ）乃至（Ｑ）は、予め用意されているガイド１７２の形状の例を示している。ガイド１７２の各形状には、所定のインデックスが付与されている。
なお、ガイド１７２の形状は、同図に示すような矩形の他、例えば、円形や多角形であってもよい。
【０１２５】
図３２は、操作部１５８に設けられたガイド選択パネル１８１を示している。
ガイド切替ボタン１８２は、ガイド１７２の形状を１つ前のインデックスに対応するものに切り替えるとき押下される。切替ボタン１８３は、ガイド１７２の形状を１つ先のインデックスに対応するものに切り替えるとき押下される。選択ボタン１８４は、表示されているガイド１７２の形状を確定させるとき押下される。
【０１２６】
例えば、図３０は、ガイド１７２の形状として図３１（Ｊ）の例に切り替えられている状態を示しているが、この状態において、切替ボタン１８２が押下された場合には、ガイド１７２の形状が同図（Ｉ）の例に切り替えられ、切替ボタン１８３が押下された場合には、ガイド１７２の形状が同図（Ｋ）の例に切り替えられる。
【０１２７】
ユーザは、ファインダとして機能する表示部１５１を見ながら被写体（例えば、人物）がガイド１７２の中に収まるように、ガイド選択パネル１８１の切替ボタン１８２，１８３を操作してガイド１７２の切り替えた後、選択ボタン１８４を押下してガイド１７２の形状を確定させる。選択ボタン１８４の押下に対応して、現在表示されているガイド１７２の形状に対応するインデックスが最適露出情報として設定される。
【０１２８】
図２９に戻る。ステップＳ１１２において、ユーザが操作部１５８に設けられたシャッタを操作した場合、それに対応して、広ダイナミックレンジ画像信号が取得され、DSP１４６が内蔵する画像用RAMに格納される。
【０１２９】
ステップＳ１１３において、ステップＳ１１１で設定された最適露出情報（ガイド１７２の形状を示すインデックス）に基づき、広ダイナミックレンジ画像信号が狭ダイナミックレンジ画像信号に変換される。
【０１３０】
具体的には、注目輝度・最適注目輝度対については、最適露出情報（ガイド１７２の形状を示すインデックス）に基づいてガイド１７２の内部領域の画像信号が取得され、その輝度のヒストグラムが生成されて、最も高い頻度を示す輝度が注目輝度Ｍに設定される。最適注目輝度Ｍｍには、予め設定された狭ダイナミックレンジの中間輝度が設定される。それ以降の処理は、図２４および図２５を用いて上述した、画像処理システムの画像処理部１１が最適露出情報を設定する際の第２の動作例と同様である。
【０１３１】
変換された狭ダイナミックレンジ画像信号は、メモリ１４８に格納される。
【０１３２】
なお、ディジタルカメラ１４０のDSP１４６においても、図２６に示した画像処理システムの画像処理部１１の第２の構成例と同様に、カラーの広ダイナミックレンジ画像を、色バランスを損なうことなく、カラーの狭ダイナミックレンジ画像に変換する処理を実行するようにしてもよい。
【０１３３】
次に、ディジタルカメラ１４０の第２の動作例について、図３３のフローチャートを参照して説明する。ステップＳ１２１において、撮像する画像の構図がユーザによって選択される。選択された画像の構図（ガイド１７２に相当する）が第１の最適露出情報として設定される。
【０１３４】
当該第１の最適露出情報を設定する処理については、図３０乃至図３２を参照して上述した第１の動作例の最適露出情報を設定する処理と同様である。
【０１３５】
ステップＳ１２２において、ユーザが操作部１５８に設けられたシャッタを操作した場合、それに対応して、広ダイナミックレンジ画像信号が取得され、DSP１４６が内蔵する画像用RAMに格納される。
【０１３６】
ステップＳ１２３において、ユーザの操作に対応し、第２の最適露出情報として露出比率ｒが設定される。第２の最適露出情報として露出比率ｒを設定する処理について、図３４および図３５を参照して説明する。
【０１３７】
図３４は、ステップＳ１２３における表示部１５１の画像表示エリア１７１の表示例を示している。いまの場合、画像表示エリア１７１には、ステップＳ１２２で取得された広ダイナミックレンジ画像が表示され、それに重畳してステップＳ１２１で設定されたガイド１７２や補助線（破線）が表示される。ただし、画像表示エリア１７１のガイド１７２の内部領域には、操作部１５８に設けられた輝度補正パネル１９１（図３５）に対するユーザの操作に対応して変更される露出比率ｒを用いて簡易生成された狭ダイナミックレンジ画像が表示される。
【０１３８】
図３５は、操作部１５８に設けられた輝度補正パネル１９１を示している。明補正(BRIGHTER)ボタン１９２は、ガイド１７２の内部領域に表示される狭ダイナミックレンジ画像の輝度を、現状よりも１段階だけ明るくする（輝度を上げる）とき押下される。暗補正(DARKER)ボタン１９３は、ガイド１７２の内部領域に表示される狭ダイナミックレンジ画像の輝度を、現状よりも１段階だけ暗くする（輝度を下げる）とき押下される。ＯＫボタン１９４は、現状の輝度を確定するとき押下される。
【０１３９】
例えば、明補正ボタン１９２が押下された場合、狭ダイナミックレンジ画像の簡易生成に用いられる露出比率ｒが所定の値だけ増加される。よって、ガイド１７２の内部領域の狭ダイナミックレンジ画像の輝度が増して表示される。反対に、暗補正ボタン１９３が押下された場合、狭ダイナミックレンジ画像の簡易生成に用いられる露出比率ｒが所定の値だけ減少される。よって、ガイド１７２の内部領域の狭ダイナミックレンジ画像の輝度が減じて表示される。ＯＫボタン１９４が押下された場合、現状の露出比率ｒが第２の最適露出情報として設定される。
【０１４０】
図３３に戻る。ステップＳ１２４において、ステップＳ１２１で設定された第１の最適露出情報（ガイド１７２の形状を示すインデックス）、およびステップＳ１２３で設定された第２の最適露出情報に基づき、広ダイナミックレンジ画像信号が狭ダイナミックレンジ画像信号に変換される。
【０１４１】
具体的には、注目輝度・最適注目輝度対については、第１の最適露出情報（ガイド１７２の形状を示すインデックス）に基づいてガイド１７２の内部領域の広ダイナミックレンジ画像の輝度が取得され、その輝度のヒストグラムが生成されて、最も高い頻度を示す輝度が注目輝度Ｍに設定される。
【０１４２】
最適注目輝度Ｍｍとしては、広ダイナミックレンジ画像の輝度に第２の最適露出情報としての露出比率ｒが乗算されて狭ダイナミックレンジ画像が簡易生成され、そのうちのガイド１７２の内部領域の輝度が抽出されて、ヒストグラムが生成され、最も高い頻度を示す輝度が最適注目輝度Ｍｍに設定される。それ以降の処理は、図２４および図２５を用いて上述した、画像処理システムの画像処理部１１が最適露出情報を設定する際の第２の動作例と同様である。
【０１４３】
変換された狭ダイナミックレンジ画像信号は、メモリ１４８に格納される。
【０１４４】
以上説明したように、ディジタルカメラ１４０の第２の動作例によれば、操作部１５８に対する簡易な操作により、ユーザが注目したい領域をガイド１７２で指定することができ、さらに、ガイド１７２の内部領域に対して所望する輝度を指定することができるので、得られる狭ダイナミックレンジ画像の階調がユーザが所望する結果により近いものとなる。
【０１４５】
なお、ディジタルカメラ１４０のDSP１４６に、上述した画像処理システムの画像処理部１１と同様の処理を実行させるようにしてもよい。反対に、画像処理システムの画像処理部１１に、ディジタルカメラ１４０のDSP１４６と同様の処理を実行させるようにしてもよい。
【０１４６】
ところで、本発明は、画像のダイナミックレンジを変更せずに輝度を変更させる場合に適用することも可能である。
【０１４７】
また、本発明は、本実施の形態のような画像変換システムやディジタルカメラのみならず、例えば、スキャナ、ファクシミリ、コピー機など、画像信号を処理する電子機器に適用することが可能である。
【０１４８】
なお、本明細書において、記録媒体に記録されるプログラムを記述するステップは、記載された順序に従って時系列的に行われる処理はもちろん、必ずしも時系列的に処理されなくとも、並列的あるいは個別に実行される処理をも含むものである。
【０１４９】
また、本明細書において、システムとは、複数の装置により構成される装置全体を表すものである。
【０１５０】
【発明の効果】
以上のように、本発明によれば、広ダイナミックレンジ画像を狭ダイナミックレンジ画像に変換する際、画像中の任意の位置の輝度を最適化することが可能となる。
【図面の簡単な説明】
【図１】本発明の一実施の形態である画像処理システムの構成例を示すブロック図である。
【図２】図１の撮像装置１の構成例を示すブロック図である。
【図３】図２のCCDイメージセンサ２３の構成例を示すブロック図である。
【図４】 CCDイメージセンサ２３の感度特性を説明するための図である。
【図５】前置増幅器２４の処理を説明するための図である。
【図６】図１の画像処理装置２を実現するパーソナルコンピュータの構成例を示すブロック図である。
【図７】画像処理システムの動作を説明するフローチャートである。
【図８】画像処理装置２の最適露出情報設定処理を説明するフローチャートである。
【図９】画像処理装置２が狭ダイナミックレンジ画像を簡易生成する処理を説明するフローチャートである。
【図１０】最適露出情報を設定する第１の動作例を説明するための表示部１３の表示例を示す図である。
【図１１】最適露出情報を設定する第１の動作例におけるGUIの表示例を示す図である。
【図１２】画像処理部１１の第１の構成例を示すブロック図である。
【図１３】図１２の輝度対検出部８１の処理を説明するフローチャートである。
【図１４】輝度対検出部８１の処理を説明するための図である。
【図１５】図１２のマッピング関数生成部８２の構成例を示すブロック図である。
【図１６】マッピング関数生成部８２の処理を説明するフローチャートである。
【図１７】図１５の最大・最小輝度取得部９１の処理を説明するための図である。
【図１８】マッピング関数生成部８２の処理を説明するための図である。
【図１９】マッピング関数生成部８２の処理を説明するための図である。
【図２０】マッピング関数生成部８２の処理を説明するための図である。
【図２１】マッピング関数生成部８２の処理を説明するための図である。
【図２２】図１６のステップＳ５０の処理の詳細を説明するフローチャートである。
【図２３】図１２のマッピング部８３の処理を説明するフローチャートである。
【図２４】最適露出情報を設定する第２の動作例を説明するための表示部１３の表示例を示す図である。
【図２５】最適露出情報を設定する第２の動作例におけるGUIの表示例を示す図である。
【図２６】画像処理部１１の第２の構成例を示すブロック図である。
【図２７】画像処理部１１の第２の構成例の処理を説明するフローチャートである。
【図２８】本発明の一実施の形態であるディジタルカメラ１４０の構成例を示すブロック図である。
【図２９】ディジタルカメラ１４０の第１の動作例を説明するフローチャートである。
【図３０】ディジタルカメラ１４０の第１の動作例において最適露出情報を設定する処理を説明するための表示部１５１の表示例を示す図である。
【図３１】表示部１５１に表示されるガイド１７２の形状の種類を示す図である。
【図３２】ディジタルカメラ１４０の操作部１５８に設けられるガイド選択パネル１８１を示す図である。
【図３３】ディジタルカメラ１４０の第２の動作例を説明するフローチャートである。
【図３４】ディジタルカメラ１４０の第１の動作例において最適露出情報を設定する処理を説明するための表示部１５１の表示例を示す図である。
【図３５】ディジタルカメラ１４０の操作部１５８に設けられる輝度補正パネル１９１を示す図である。
【符号の説明】
１ 撮像装置， ２ 画像処理装置， １１ 画像処理部， １２ 操作入力部， １３ 表示部， ４１ CPU， ５２ 磁気ディスク， ５３ 光ディスク， ５４ 光磁気ディスク， ５５ 半導体メモリ， ７１画像選択ボタンパネル， ８１ 輝度対検出部， ８２ マッピング関数生成部， ８３ マッピング部， ９１ 最大・最小輝度取得部， ９２ 高域関数決定部， ９３ 低域関数決定部， ９４ 関数合成部， １１１ 範囲指定パネル， １２１ 輝度信号生成部， １２２ 輝度対検出部， １２３ 色補正関数生成部， １２４ マッピング部， １２５ マッピング関数生成部， １２６ マッピング部， １２７，１２８ べき乗演算部， １２９ スケーリング部， １４０ ディジタルカメラ， １４６ DSP， １５２ 制御部， １５４ 磁気ディスク， １５５ 光ディスク， １５６ 光磁気ディスク， １５７ 半導体メモリ， １５８ 操作部， １７２ ガイド， １８１ ガイド選択パネル， １９１ 輝度補正パネル[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, and a recording medium. For example, the present invention relates to an image processing apparatus and method suitable for use in changing the dynamic range of luminance of an image signal, and a recording medium.
[0002]
[Prior art]
Solid-state imaging devices such as CCD (Charge Coupled Device) image sensors and CMOS (Complementary Mental-Oxide Semiconductor) image sensors are imaging devices such as video cameras and digital still cameras, and component inspection devices in the field of FA (Factory Automation). And widely used in optical measuring devices such as electronic endoscopes in the field of ME (Medical Electronics).
[0003]
In recent years, many techniques have been developed that can obtain an image signal having a dynamic range comparable to that of an optical film photograph using these solid-state imaging devices.
[0004]
On the other hand, conventional devices such as a display device such as a CRT (Cathode Ray tube) that displays captured moving images and still images, printers for printing, projectors for projection, etc. currently have brightness gradations that they can represent. There is a limit. That is, the dynamic range of an image that can be expressed is narrow. Therefore, even if an image signal of an image having a dynamic range comparable to that of an optical film photograph (hereinafter referred to as a wide dynamic range image) can be obtained, it cannot be expressed (displayed or printed). There's a problem.
[0005]
Therefore, there is a dynamic range compression processing technology that compresses the luminance gradation of a wide dynamic range image to narrow the dynamic range and converts it into an image that can be expressed by a conventional device such as a display device (hereinafter referred to as a narrow dynamic range image). is needed.
[0006]
Hereinafter, four conventionally proposed dynamic range compression techniques will be described.
[0007]
As a first technique, there is a method of scaling the luminance by adjusting the luminance gain and offset of the image signal. This first technique is very simple and can be easily applied. However, since the luminance gradation is not compressed, all luminance values exceeding the dynamic range of the display device or the like are clipped, and the information of the original wide dynamic range image cannot be fully utilized.
[0008]
As a second technique, there is a technique of compressing the dynamic range by diverting a gamma correction process originally performed to correct the gamma characteristic of a display device or the like.
Since the gamma correction curve used for the gamma correction process is a function of a power, the correction characteristic can be easily adjusted by changing the exponent which is the gamma value. However, due to the diversion to the dynamic range compression, the original gamma correction process is affected, and the color balance may be lost or the contrast may deteriorate.
[0009]
As a third technique, there is a histogram equalization method using an accumulated histogram curve of an image as a gradation correction curve. In the histogram equalization method, gradation conversion is performed so as to give more gradations to the luminance occupying a wider area in the image, corresponding to the luminance distribution (histogram). Therefore, the entire image acts in the direction of enhancing the contrast, and an image with high visibility in which details are clarified can be obtained even in a narrow dynamic range. However, since the image obtained as a result of the gradation conversion is greatly influenced by the luminance distribution of the original wide dynamic range image, a desired image may not be obtained.
[0010]
As a fourth technique, there is a technique of emphasizing details using a local operator. In the fourth technique, images are separated for each spatial frequency band by a local operator, and after adjusting the gain for each of the separated spatial frequency bands, they are integrated again. When adjusting the gain, it is possible to compress the gradation by attenuating the low frequency band, and to reduce the attenuation amount of the high frequency band so that the detail contrast is not impaired.
[0011]
Therefore, even a conventional device such as a display device with few gradations can express an image with high visibility of details. However, as a result of changing the balance of each band, gradation inversion occurs in a high-contrast region such as an edge portion in an image, and this portion is recognized as a very annoying artifact in some cases. Sometimes.
[0012]
[Problems to be solved by the invention]
As described above, with the conventional dynamic range compression technique, the obtained narrow dynamic range image may not result in reflecting the user's intention.
Specifically, when converting a wide dynamic range image to a narrow dynamic range image, for example, even if an area occupied by a subject in the image is expressed with an optimum luminance, there is a problem that it is difficult to execute.
[0013]
The present invention has been made in view of such circumstances, and an object thereof is to optimize the luminance of an arbitrary region in an image when converting a wide dynamic range image into a narrow dynamic range image. .
[0014]
[Means for Solving the Problems]
  The image processing apparatus according to the present invention includes a setting unit that sets a second luminance value in the second dynamic range, and an arithmetic unit that calculates the first luminance value in the first dynamic range corresponding to the second luminance value. And a first luminance value and a second luminance valueFirstMapping function generating means for generating a mapping function based on the luminance pair;LivingCompletionIsConversion means for converting the first image signal into the second image signal using the mapping function.The mapping function generation means includes a point corresponding to the first luminance pair, a third luminance value near the minimum luminance value in the first dynamic range, and a fourth luminance value near the minimum luminance value in the second dynamic range. And a third brightness point consisting of a fifth brightness value near the maximum brightness value in the first dynamic range and a sixth brightness value near the maximum brightness value in the second dynamic range. A smooth monotonically increasing function passing through the vicinity of the three points corresponding to the luminance pairs of.
[0015]
  The mapping function generation meansIs,Generate a monotonically increasing function as a first function passing through a point corresponding to the first luminance pair and a point corresponding to the second luminance pairFirst function generating means;A monotonically increasing function passing through a point corresponding to the first luminance pair and a point corresponding to the third luminance pair is generated as the second function.A second function generating means;FirstIn the vicinity of the point indicated by the luminance pair, the firstIs a monotonically increasing function through a point on the function and a point on the second functionThe third functionAs a mapping functionThird function generation means for generatingMube able to.
[0016]
  Said third function generating meansIsA predetermined point indicated by the first function,FirstA quadratic curve function defined by three points of one point indicated by the luminance pair and a predetermined one point indicated by the second function is generated as a third function.Dobe able to.
[0017]
  The image processing apparatus according to the present invention uses the calculation information to calculate correction information for correcting the color balance of the three primary color signals constituting the second image signal based on the mapping function, and the correction information. And a correction means for correcting the three primary color signals constituting the image signal.The calculation means acquires the gamma characteristic of the mapping function, and corrects the three primary color signals so that the ratio between the corrected three primary color signals and the luminance value after applying the mapping function becomes a ratio calculated by the gamma characteristics. A coefficient can be determined.
[0018]
  The image processing method of the present invention includes a setting step for setting a second luminance value in the second dynamic range, and a calculation step for calculating the first luminance value in the first dynamic range corresponding to the second luminance value. And a first luminance value and a second luminance valueFirstA mapping function generation step for generating a mapping function based on the luminance pair;LivingConverting the first image signal into the second image signal using the generated mapping function.In the mapping function generation step, the point corresponding to the first luminance pair, the third luminance value near the minimum luminance value in the first dynamic range, and the fourth luminance near the minimum luminance value in the second dynamic range. A point corresponding to the second luminance pair consisting of values, and a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range. A smooth monotonically increasing function passing through the vicinity of the three points corresponding to the three luminance pairs is generated as a mapping function.
[0019]
  Recording medium of the present inventionIsAn image processing program for converting a first image signal having a first dynamic range into a second image signal having a second dynamic range with respect to a luminance value, wherein the first image signal has a second dynamic range. A setting step for setting a luminance value of 2, a calculation step for calculating a first luminance value in a first dynamic range corresponding to the second luminance value, and a first luminance value and a second luminance valueFirstA mapping function generation step for generating a mapping function based on the luminance pair;LivingConverting the first image signal into the second image signal using the generated mapping function.In the mapping function generation step, the point corresponding to the first luminance pair, the third luminance value near the minimum luminance value in the first dynamic range, and the fourth luminance near the minimum luminance value in the second dynamic range. A point corresponding to the second luminance pair consisting of values, and a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range. A process for generating a smooth monotonically increasing function passing through the vicinity of the three points corresponding to the three luminance pairs as a mapping functionComputerTo runThe program is recorded.
[0020]
  The present inventionInIn this case, the second luminance value in the second dynamic range is set, and the first luminance value in the first dynamic range corresponding to the second luminance value is calculated. Furthermore, it consists of a first luminance value and a second luminance valueFirstA mapping function is generated based on the luminance pair, and the first image signal is converted into a second image signal using the generated mapping function. In addition,The mapping function includes a point corresponding to the first luminance pair, a third luminance value near the minimum luminance value in the first dynamic range, and a fourth luminance value near the minimum luminance value in the second dynamic range. A third luminance composed of a point corresponding to the second luminance pair and a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range A smooth monotonically increasing function is generated that passes in the vicinity of the three points corresponding to the pair.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a configuration example of an image processing system to which the present invention is applied. This image processing system includes an imaging device 1 including a digital camera that images a subject as a wide dynamic range image signal, and a personal computer that converts a wide dynamic range image signal supplied from the imaging device 1 into a narrow dynamic range image signal. The image processing apparatus 2 is composed of the above.
[0022]
The image processing apparatus 2 receives an image processing unit 11 that processes an image signal, an operation command received from a user, and the operation input unit 12 that notifies the image processing unit 11 of information on the operation command. It comprises a corresponding GUI (Graphical User Interface) and a display unit 13 for displaying the processing result of the image processing unit 11. The image processing unit 11 of the image processing apparatus 2 outputs a narrow dynamic range image signal as a processing result to the outside of the image processing apparatus 2 as appropriate.
[0023]
FIG. 2 shows a configuration example of the imaging apparatus 1. The imaging apparatus 1 includes a lens 21 that collects an optical image of a subject, a diaphragm 22 that adjusts the amount of light transmitted, a lens 21 and a CCD image sensor 23 that converts an optical image input through the diaphragm 22 into an electrical signal, and a correlation. A preamplifier 24 including a double sampling circuit, a gamma correction circuit, a knee characteristic circuit, and the like, a video encoder 25 that encodes an electric signal input from the preamplifier 24 into a wide dynamic range image signal, and a wide dynamic range image signal The output unit 26 is configured to output to the image processing apparatus 2.
[0024]
The CCD image sensor 23 and the preamplifier 24 that are components of the imaging apparatus 1 will be described. FIG. 3 shows a configuration example of the CCD image sensor 23. The CCD image sensor 23 has the same configuration as an interline CCD that performs interlaced scanning. That is, photodiodes (PD) 31 that accumulate charges according to the amount of incident light are two-dimensionally arranged, and vertical registers (V registers) 32 are provided between the columns of the photodiodes 31. A horizontal register (H register) 33 is provided at the end (lower end in FIG. 3).
[0025]
Note that the high sensitivity photodiode 31H is used for the 2i (i = 0, 1, 2,...) Line constituting the even field, and the low sensitivity photodiode is used for the 2i + 1 line constituting the odd field. 31L is used.
[0026]
The sensitivity characteristics of the photodiodes 31H and 31L will be described with reference to FIG. In FIG. 4, the horizontal axis Ei indicates the intensity of light input to the photodiodes 31H and 31L, and the vertical axis Eo indicates the amount of charge accumulated in the photodiodes 31H and 31L.
[0027]
The low-sensitivity photodiode 31L has a sensitivity characteristic indicated by a straight line A in FIG.
That is, a charge proportional to the intensity of incident light is accumulated over the entire range of the intensity of input light. The high-sensitivity photodiode 31H has a sensitivity characteristic indicated by a straight line B in FIG. That is, it corresponds to light with low intensity, and accumulates more charge than the low-sensitivity photodiode 31L in proportion to the intensity of the incident light.
[0028]
Returning to FIG. The charges accumulated in the photodiodes 31H and 31L are read to the corresponding vertical register 32 at every predetermined timing and then transferred to the horizontal register 33. The horizontal register 33 sequentially outputs the charges transferred from the vertical register 32 for one horizontal line.
[0029]
The timing for reading out the charges accumulated in the photodiodes 31H and 31L will be described. The CCD image sensor 23 performs an interlaced scan. When reading out charges, the charges of the photodiodes 31H and 31L adjacent to each other are read and added as charges corresponding to one pixel.
[0030]
For example, as the charge of the second i line in the even field, the charge accumulated in the photodiode 31H of the second i line and the charge accumulated in the photodiode 31L of the second i + 1 line are read and added. As the charges on the second i + 1 line in the odd field, the charges accumulated in the photodiode 31L on the second i + 1 line and the charges accumulated on the photodiode 31H on the second (i + 1) line are read and added.
[0031]
Therefore, the CCD image sensor 23 outputs the sum of the charge accumulated in the low-sensitivity photodiode 31L and the charge accumulated in the high-sensitivity photodiode 31H as an electrical signal corresponding to each pixel. . Therefore, the CCD image sensor 23 has a sensitivity characteristic indicated by a line C obtained by adding the straight line A (corresponding to the photodiode 31L) and the straight line B (corresponding to the photodiode 31H) in FIG. That is, in the CCD image sensor 23, the high-sensitivity photodiode 31H accumulates electric charge in a region where light intensity is low, and the low-sensitivity photodiode 31L accumulates electric charge in a region where light intensity is high. A wide dynamic range electric signal with little saturation can be obtained.
[0032]
The preamplifier 24 applies the function shown in FIG. 5 (inverse function of the curve C in FIG. 4) to the wide dynamic range electric signal Eo output from the CCD image sensor 23 to estimate the intensity of the original optical signal. The value Ei ′ is obtained.
[0033]
Next, FIG. 6 shows a configuration example of a personal computer that operates as the image processing apparatus 2 by executing an application program for image processing. This personal computer includes a CPU (Central Processing Unit) 41. An input / output interface 45 is connected to the CPU 41 via the bus 44.
[0034]
The input / output interface 45 includes an input unit 46 including an input device such as a keyboard and a mouse corresponding to the operation input unit 12, a display control unit 47 that outputs an image signal as a GUI or a processing result to the display unit 13, and the imaging apparatus 1. A video decoder 48 for decoding a wide dynamic range image signal inputted from the A, a A / D converter 49 for quantizing a voltage corresponding to the luminance of each pixel and converting it into a digital image signal, a hard disk, a frame memory, etc. Storage unit 50 for storing programs and digital image signals, etc., and magnetic disk 52 (including floppy disk), optical disk 53 (including CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc)) ), Magneto-optical disk 54 (including MD (Mini Disc)), and recording medium such as semiconductor memory 55 Drive 51 for reading and writing data are connected.
[0035]
In addition to the input / output interface 45, a ROM (Read Only Memory) 42 and a RAM (Random Access Memory) 43 are connected to the bus 44.
[0036]
An image processing program for causing the personal computer to execute the operation as the image processing apparatus 2 is supplied to the personal computer in a state stored in the magnetic disk 52 to the semiconductor memory 55, read by the drive 51, and stored in the storage unit 50. Installed in the hard disk drive built in The image processing program installed in the storage unit 50 is loaded from the storage unit 50 to the RAM 43 and executed by a command of the CPU 41 corresponding to a command from the user input to the input unit 46.
[0037]
Next, the operation of the image processing system will be described with reference to the flowchart of FIG. In step S <b> 1, the imaging device 1 captures a light image of a subject and outputs the obtained wide dynamic range image signal to the image processing device 2.
[0038]
In step S <b> 2, the image processing unit 11 of the image processing apparatus 2 sets optimum exposure information corresponding to the user's selection operation. A first operation example for setting optimum exposure information will be described with reference to the flowchart of FIG.
[0039]
In step S <b> 11, the image processing unit 11 of the image processing device 2 simply generates a predetermined number of narrow dynamic range image signals from the wide dynamic range image signal input from the imaging device 1. A process of simply generating a narrow dynamic range image signal will be described with reference to the flowchart of FIG. Hereinafter, it is assumed that the image signal to be processed is a luminance signal.
[0040]
In step S <b> 21, the image processing unit 11 stores the image signal Y (x, y) of each pixel of the wide dynamic range image input and stored from the imaging device 1 and a plurality of exposure ratios r prepared in advance. One of them is read from the storage unit 50. Here, x and y are vertical or horizontal coordinate values of the image, respectively.
[0041]
In step S22, the image processing unit 11 generates a converted image signal Yn (x, y) by multiplying the image signal Y (x, y) of all pixels by the exposure ratio r. In step S23, the image processing unit 11 compares the converted image signal Yn (x, y) with the predetermined threshold values Ynmax and Ynmin, and determines the converted image signal Yn (x, y) larger than the threshold value Ynmax as the threshold value Ynmax. And the converted image signal Yn (x, y) smaller than the threshold Ynmin is replaced using the threshold Ynmin. The converted image signal Yn (x, y) generated in this way is a narrow dynamic range image signal that is simply generated.
[0042]
The processes in steps S21 to S23 described above are repeatedly executed as many times as the number of types of exposure ratio r prepared in advance.
[0043]
The process returns to step S12 in FIG. In step S12, the image processing unit 11 causes the display unit 13 to display the narrow dynamic range image signal simply generated by the processing in step S11, and also displays one of the displayed narrow dynamic range images as a user. The display unit 13 is caused to display a GUI for causing the user to make a selection.
[0044]
FIG. 10 shows a display example of the display unit 13 that displays a simply generated narrow dynamic range image. In this display example, 9 (= 3 × 3) easily generated narrow dynamic range images 62 can be displayed in the image display area 61 of the display unit 13, and 4 simply generated narrow dynamic range images can be displayed. A range image 62 is displayed. A frame 63 is provided around the displayed narrow dynamic range image 62.
[0045]
FIG. 11 shows a display example of a GUI that is operated when the user selects a plurality of simply generated narrow dynamic range images 62 displayed on the display unit 13. On the image selection button panel 71 as a GUI, the left move button 72 that is clicked when designating the image on the left side of the currently designated image, is clicked when the image on the right side of the currently designated image is designated. A right movement button 73 and a select button 74 that is clicked when a designated image is selected are provided.
[0046]
The frame 63 displayed in the image display area 61 of the display unit 13 is highlighted, for example, so that the user can recognize when the corresponding image is designated by the user. In the display example of FIG. 10, the upper left image of the 3 × 3 images is designated, and the frame 63 is highlighted.
[0047]
The user uses the image selection button panel 71 to select an image in which the gradation of the subject in the image is most appropriately displayed among the plurality of images displayed in the image display area 61 of the display unit 13. To.
[0048]
Returning to FIG. 8, in step S13, the image processing unit 11 waits until the user selects one of the simply generated narrow dynamic range images displayed in the image display area 61 of the display unit 13, If it is determined that the user has selected one of the simply generated narrow dynamic range images, the process proceeds to step S14. In step S14, the image processing unit 11 sets the exposure ratio r corresponding to the image selected by the user as optimum exposure information. The process returns to step S3 in FIG.
[0049]
In step S3, the image processing unit 11 converts the wide dynamic range image signal input from the imaging device 1 into a narrow dynamic range image signal according to the optimum exposure information set in step S2, and outputs the narrow dynamic range image signal to the display unit 13.
[0050]
Details of processing for converting a wide dynamic range image signal into a narrow dynamic range image signal according to the optimum exposure information will be described.
[0051]
FIG. 12 shows a first configuration example of the image processing unit 11 related to the processing. The luminance pair detection unit 81 detects a target luminance / optimum target luminance pair (M, Mm) based on the wide dynamic range image signal and the optimal exposure information, and outputs the detected target luminance / optimum target luminance pair (M, Mm) to the mapping function generation unit 82. The mapping function generation unit 82 generates a mapping function based on the wide dynamic range image signal and the target luminance / optimal target luminance pair (M, Mm) from the luminance pair detection unit 81 and outputs the mapping function to the mapping unit 83. The mapping unit 83 applies the wide dynamic range image signal to the mapping function from the mapping function generation unit 82 to generate a narrow dynamic range image signal.
[0052]
Processing for detecting the target luminance / optimal target luminance pair (M, Mm) of the luminance pair detection unit 81 will be described with reference to the flowchart of FIG. 13 and FIG. FIG. 14 is a diagram for explaining the relationship between the luminance dynamic range of a wide dynamic range image and the luminance dynamic range of a simply generated narrow dynamic range image. The horizontal axis of FIG. The logarithmic value logY of the luminance Y of the dynamic range image is shown, and the vertical axis shows the logarithmic value logYn of the luminance Yn of the narrow dynamic range image simply generated.
[0053]
In step S31, the luminance pair detection unit 81 acquires preset saturation luminance Ys of the wide dynamic range image, saturation luminance Yns of the narrow dynamic range image, and intermediate luminance Ynm of the narrow dynamic range image from the storage unit 50. . Here, the intermediate luminance Ynm of the narrow dynamic range image is calculated in advance using the saturation luminance Yns and noise level luminance Ynn of a predetermined narrow dynamic range image, for example, as shown in the following equation (1). And
Intermediate luminance Ynm = (Yns + Ynn) / 2 (1)
[0054]
In step S32, the luminance pair detection unit 81 acquires the optimal exposure information (exposure ratio r) set in step S14 of FIG. In step S33, the luminance pair detection unit 81 calculates the luminance of the wide dynamic range image corresponding to the intermediate luminance Ynm of the narrow dynamic range image using the following equation (2). This luminance is set to the target luminance M.
Attention luminance M = r · Ys (Ynm / Yns) (2)
[0055]
In step S34, the luminance pair detection unit 81 sets the intermediate luminance Ynm of the narrow dynamic range image to the optimum attention luminance Mm. In step S <b> 35, the luminance pair detection unit 81 outputs the target luminance / optimal target luminance pair (M, Mm) to the mapping function generation unit 82.
[0056]
Next, FIG. 15 shows a detailed configuration example of the mapping function generation unit 82. The mapping function generation unit 82 is a maximum / minimum luminance acquisition unit 91 that acquires the maximum luminance H and minimum luminance L of the wide dynamic range image, and the maximum luminance H and minimum luminance of the wide dynamic range image from the maximum / minimum luminance acquisition unit 91. L and a wide dynamic from the maximum / minimum luminance acquisition unit 91 and a high frequency function determination unit 92 that generates a high frequency logarithmic function based on the target luminance / optimal target luminance pair (M, Mm) from the luminance pair detection unit 81 A low-frequency function determining unit 93 that generates a low-frequency logarithmic function based on the maximum luminance H and the minimum luminance L of the range image and the target luminance / optimum target luminance pair (M, Mm) from the luminance pair detection unit 81; The high-frequency logarithmic function generated by the high-frequency function determining unit 92 and the low-frequency logarithmic function generated by the low-frequency function determining unit 93 are combined to generate a mapping function.
[0057]
The process in which the mapping function generation unit 82 generates the mapping function will be described with reference to the flowchart of FIG.
[0058]
In step S41, the maximum / minimum luminance acquisition unit 91 generates a histogram of the luminance signal Y of the wide dynamic range image as shown in FIG. In step S42, the maximum / minimum luminance acquisition unit 91 considers noise included in the luminance signal Y and sets a value smaller than the maximum luminance value by a predetermined ratio (for example, 1%) to the maximum luminance H. And determining a value larger than the minimum value of luminance having a frequency by a predetermined ratio (for example, 1%) as the minimum luminance L of luminance, and setting the maximum luminance H and the minimum luminance L to the high frequency function determining unit 92, The result is output to the low frequency function determination unit 93 and the function synthesis unit 94.
[0059]
In step S43, the function synthesis unit 94 generates a logarithmic function Ym (Y) corresponding to the straight line A passing through the two points (logL, Lm) and (logH, Hm) shown in FIG.
Ym (Y) = αlogY + β (3)
[Expression 1]

[0060]
FIG. 18 is a diagram for explaining the relationship between the dynamic range of the luminance of the wide dynamic range image and the dynamic range of the luminance of the narrow dynamic range image after conversion, and the horizontal axis in FIG. The logarithmic value logY of the brightness Y of the wide dynamic range image is shown, and the vertical axis shows the brightness Ym of the narrow dynamic range image after conversion. The same applies to FIGS. 19 to 21. Hm and Lm represent the maximum brightness or the minimum brightness in a narrow dynamic range, and are preset values.
[0061]
In step S44, the high frequency function determining unit 92 generates a high frequency logarithmic function YmH (Y) corresponding to the straight line C passing through the two points (logM, Mm) and (logH, Hm) shown in FIG. Output to the unit 94.
YmH (Y) = αHlogY + βH (4)
[Expression 2]

[0062]
In step S45, the low frequency function determining unit 93 generates a low frequency logarithmic function YmL (Y) corresponding to the straight line B passing through the two points (logL, Lm) and (logM, Mm) shown in FIG. Output to the unit 94.
YmL (Y) = αLlogY + βL (5)
[Equation 3]

[0063]
Note that the order of the processes of steps S43, S44, and S45 may be changed as appropriate, or may be executed simultaneously in parallel.
[0064]
In step S46, the function synthesizer 94 substitutes the noticed luminance M for Y of the logarithmic function Ym (Y) corresponding to the straight line A in FIG. 18, and compares the obtained Ym (M) with the optimum noticed luminance Mm. Based on the comparison result, the point (logM, Mm) corresponding to the target luminance / optimal target luminance pair is located above the straight line A, below the straight line A, or coincides with the straight line A Judge whether to do.
[0065]
When the comparison result indicates that Mm is larger than Ym (M), it is determined that the point (logM, Mm) corresponding to the target luminance / optimal target luminance pair is located above the straight line A as shown in FIG. Then, the process proceeds to step S47. In step S47, the function synthesizer 94 sets the parameters CL and CH on the horizontal axis in FIG. 19 as in the following equation (6) so as to obtain an upward convex mapping function as shown in FIG.
[Expression 4]

... (6)
[0066]
In step S50, the function synthesizer 94 smoothly connects the high-frequency logarithm function YmH (Y) and the low-frequency logarithm function YmL (Y) to generate a mapping function. Specifically, the input luminance Y, that is, the luminance Y of the wide dynamic range image is an area smaller than the minimum luminance L, an area that is equal to or larger than the minimum luminance L and smaller than CL, and is equal to or larger than CL and smaller than CH. The area is divided into areas that are greater than or equal to CH and smaller than the maximum brightness H, or areas that are greater than or equal to the maximum brightness H, and a mapping function f (Y) corresponding to each area is generated (details will be described later with reference to FIG. 22). ).
[0067]
If the comparison result in step S46 is that Mm is equal to Ym (M), it is determined that the point (logM, Mm) corresponding to the target luminance / optimal target luminance pair matches the straight line A, and the process proceeds to step S48. . In step S48, the function synthesis unit 94 sets the parameters CL and CH of the mapping function as shown in the following equation (7).
CL = L
CH = H (7)
[0068]
When the comparison result in step S46 is smaller than Ym (M), the point (logM, Mm) corresponding to the target luminance / optimal target luminance pair is lower than the straight line A as shown in FIG. If it is determined that the current position is located in step S49, the process proceeds to step S49. In step S49, the function synthesizer 94 sets the parameters CL and CH as shown in the following equation (8) so as to obtain a downwardly convex mapping function as shown in FIG.
[Equation 5]

... (8)
[0069]
Details of the processing of the function synthesis unit 94 in step S50 will be described with reference to the flowchart of FIG. Based on the magnitude comparison determination in steps S61, S63, S65, and S72, the input luminance Y that is a variable of the mapping function f (Y) is classified into one of the five regions.
[0070]
For the input luminance Y classified in the region smaller than the minimum luminance L through the process of step S61, a mapping function f (Y) is defined in step S62 as shown in the following equation (9).
f (Y) = Lm Y <L (9)
[0071]
For the input luminance Y classified into the region that is equal to or greater than the minimum luminance L and smaller than CL through the processing of steps S61 and S63, in step S64, as shown in the following equation (10), the mapping function f (Y) is defined by the low-frequency logarithm function YmL (Y).
f (Y) = YmL (Y) = αLlogY + βL L ≦ Y <L (10)
[0072]
If the process is classified into an area that is greater than or equal to CL and smaller than CH through the processes of steps S61, S63, and S65, the process proceeds to step S66.
[0073]
In step S66, it is determined whether CL-2M + CH is greater than zero. If it is determined that CL-2M + CH is greater than 0, the process proceeds to step S67. In step S67, the parameter t is defined by the following equation (11).
[Formula 6]

(11)
[0074]
In step S68, a mapping function f (Y) is defined for the input luminance Y as shown in the following equation (12).
[Expression 7]

CL ≦ Y <CH (12)
[0075]
If it is determined in step S66 that CL-2M + CH is not greater than 0, the process proceeds to step S69. In step S69, it is determined whether CL-2M + CH is smaller than zero. If it is determined that CL-2M + CH is smaller than 0, the process proceeds to step S70. In step S70, the parameter t is defined by the following equation (13).
[Equation 8]

(13)
[0076]
If it is determined in step S69 that CL-2M + CH is not smaller than 0, that is, CL-2M + CH = 0, the process proceeds to step S71. In step S71, the parameter t is defined by the following equation (14).
t = (Y-CL) / 2 (M-CL) (14)
[0077]
As shown in the following equation (15), in step S73, for the input luminance Y classified into the region that is higher than CH and smaller than the maximum luminance H through the processing of steps S61, S63, S65, and S72. , The mapping function f (Y) is defined by the high-frequency logarithm function YmH (Y).
f (Y) = YmH (Y) = αHlogY + βH CH ≦ Y <H (15)
[0078]
For the input luminance Y classified into the region having the maximum luminance H or higher through the processing of steps S61, S63, S65, and S72, in step S74, as shown in the following equation (16), the mapping function f ( Y) is defined by the high-frequency log function YmH (Y).
f (Y) = Lm H <Y (16)
[0079]
As described above, the mapping function f (Y) is generated by a series of processes of the mapping function generation unit 82.
[0080]
Next, processing of the mapping unit 83 that generates a narrow dynamic range image signal by applying a wide dynamic range image signal to the generated mapping function f (Y) will be described with reference to the flowchart of FIG.
[0081]
In step S <b> 81, the mapping unit 83 acquires a wide dynamic range image signal from the storage unit 50. The mapping unit 83 acquires the mapping function f (Y) from the mapping function generation unit 82.
[0082]
In step S82, the mapping unit 83 generates a narrow dynamic range image signal by sequentially substituting the luminance signals Y (x, y) of all the pixels of the wide dynamic range image into the mapping function f (Y).
[0083]
As described above, according to the series of processes of the image processing unit 11, a plurality of narrow dynamic range images that are simply generated are presented to the user, and the gradation of the subject is most appropriately displayed. Is selected by the user, and the target brightness / optimum target brightness pair (M, Mm) is derived based on the selected image. Further, a mapping function f (Y) is generated based on the target luminance / optimal target luminance pair (M, Mm), and the wide dynamic range image signal is converted into a narrow dynamic range image signal by the mapping function f (Y). Therefore, it is possible to obtain an image with a narrow dynamic range in which a subject in the image is expressed with an appropriate gradation.
[0084]
In the above description, only one target brightness / optimum target brightness pair (M, Mm) is determined and the subsequent processing is executed. And a mapping function may be generated.
[0085]
Next, a second operation example in which the image processing unit 11 sets optimum exposure information will be described with reference to FIGS. 24 and 25. FIG. In the first operation example described above with reference to FIGS. 8 to 11, the user is allowed to select one of a plurality of narrow dynamic range images that are simply generated. In the second operation example, Only one narrow dynamic range image that is simply generated by applying a predetermined exposure ratio r is displayed, and a range to be noted (for example, a range in which the subject is displayed) in the image is specified by the user. Then, information indicating the designated range (mask 103 in FIG. 24B) is set as the optimum exposure information and supplied to the luminance pair detection unit 81.
[0086]
FIG. 24 illustrates a display example of the display unit 13 that displays a single narrow dynamic range image that is simply generated. As shown in FIG. 4A, only one narrow dynamic range image 102 that is simply generated is displayed in the image display / designated range drawing area 101 of the display unit 13. On the narrow dynamic range image 102 displayed in the image display / designated range drawing area 101, a mask 103 representing the range designated by the user as shown in FIG.
[0087]
FIG. 25 shows a display example of a GUI operated when the user sets the mask 103 displayed to be superimposed on the narrow dynamic range image 102. The range designation panel 111 as a GUI includes a pen button 112 that is clicked when setting of the mask 103 is started, an eraser button 113 that cancels the setting of the set mask 103, and a completion button that confirms the set mask 103 114 is provided.
[0088]
After the user clicks the pen button 112, the user draws the range of the mask 103 using an input device such as a mouse, and clicks the completion button 114 to confirm the drawn range of the mask 103. In response to this, information indicating the determined range of the mask 103 is supplied to the luminance pair detection unit 81 as optimum exposure information.
[0089]
In the second operation example, the luminance pair detection unit 81 stores the intermediate luminance Ynm of the preset narrow dynamic range image as the optimum target luminance Mm, as in the above-described process of step S31 (FIG. 13). Obtained from the image processing application location program stored in the unit 50.
[0090]
In order to obtain one target luminance M, the luminance pair detection unit 81 acquires a luminance signal of pixels included in the range of the mask 103 in the wide dynamic range image to obtain a pixel set G1, and among the pixel set G1, a predetermined value is obtained. The pixel set G2 is excluded by excluding pixels with luminance exceeding the saturation level of the pixel and pixels with luminance lower than the predetermined noise level. Further, the luminance pair detection unit 81 generates a luminance histogram of the pixel set G2, and sets the luminance with the highest frequency as the attention luminance M.
[0091]
The attention brightness / optimum attention brightness pair (M, Mm) set in this way is supplied to the mapping function generation unit 82 and processed in the same manner as the series of processes described above.
[0092]
As described above, according to the second operation example in which the optimum exposure information is set, a desired narrow dynamic range image can be obtained by an intuitive operation in which the user designates the range of the subject that the user wants to display at the optimum brightness. It becomes possible to create.
[0093]
Next, a configuration example of the image processing unit 11 corresponding to a case where a color wide dynamic range image signal (three primary color signals R, G, and B) is supplied from the imaging device 1 to the image processing device 2 (hereinafter, referred to as “image processing device 2”). The operation and description of the image processing unit 11 will be described with reference to FIG. In the second configuration example of the image processing unit 11, gradation conversion is performed so that the color balance of the color image is not impaired.
[0094]
FIG. 26 shows a second configuration example of the image processing unit 11. The luminance signal generation unit 121 generates luminance Y using the three primary color signals R, G, and B of each pixel of the input wide dynamic range image using the following equation (17), and the luminance pair detection unit 122, The color correction function generation unit 123, the mapping unit 124, the mapping function generation unit 125, the mapping unit 126, and the power calculation unit 127 are output.
Luminance Y = kr · R + kg · G + kb · B (17)
[0095]
Here, kr, kg, and kb are constants. For example, kr = 0.3, kg = 0.6, and kb = 0.1.
[0096]
The luminance pair detection unit 121 is based on the luminance Y of each pixel of the wide dynamic range image from the luminance signal generation unit 121 and the optimum exposure information acquired by the first or second operation example of the image processing unit 11 described above. The target luminance / optimal target luminance pair (M, Mm) is detected and output to the color correction function generation unit 123 and the mapping function generation unit 82.
[0097]
The color correction function generation unit 123 is based on the luminance Y of each pixel of the wide dynamic range image from the luminance signal generation unit 121 and the target luminance / optimal target luminance pair (M, Mm) from the luminance pair detection unit 81. A color correction function fC (Y) indicating a color correction amount γ corresponding to the luminance Y is generated and output to the mapping unit 124.
[0098]
A process in which the color correction function generation unit 123 generates the color correction function fC (Y) will be described in more detail. The color correction function generation process is substantially the same as the mapping function generation process of the mapping function generation unit 82 described above with reference to FIGS. 16 to 22, and the process in step S50 (FIG. 16) described with reference to FIG. Only slightly different. Therefore, only the process corresponding to the process in step S50 of the mapping function generation process in the color correction function generation process will be described with reference to FIG.
[0099]
The color correction function generation unit 123 classifies the input luminance Y, which is a variable of the color correction function fC (Y), into one of the five regions based on the magnitude comparison determination in steps S91, S93, S95, and S102.
[0100]
For the input luminance Y classified in the area smaller than the minimum luminance L through the processing of step S91, a color correction function fC (Y) is defined in step S92 as shown in the following equation (17). .
fC (Y) = αL Y <L (17)
[0101]
For the input luminance Y classified into the area that is equal to or higher than the minimum luminance L and smaller than CL through the processing of steps S91 and S93, in step S94, as shown in the following equation (18), the color correction function fC (Y) is defined.
fC (Y) = αL L ≦ Y <L (18)
[0102]
If the process is classified into an area that is greater than or equal to CL and smaller than CH through the processes of steps S91, S93, and S95, the process proceeds to step S96.
[0103]
In step S96, it is determined whether CL-2M + CH is greater than zero. If it is determined that CL-2M + CH is greater than 0, the process proceeds to step S97. In step S97, the parameter t is defined by equation (11).
[0104]
In step S98, the color correction function fC (Y) is defined for the input luminance Y as shown in the following equation (19).
fC (Y) = (1-t) αL + tαH CL ≦ Y <CH (19)
[0105]
If it is determined in step S96 that CL-2M + CH is not greater than 0, the process proceeds to step S99. In step S99, it is determined whether CL-2M + CH is smaller than zero. If it is determined that CL-2M + CH is smaller than 0, the process proceeds to step S100. In step S100, the parameter t is defined by equation (13).
[0106]
If it is determined in step S99 that CL-2M + CH is not smaller than 0, that is, CL-2M + CH = 0, the process proceeds to step S101. In step S101, the parameter t is defined by equation (14).
[0107]
As shown in the following equation (20), in step S103, for the input luminance Y classified into the region that is higher than CH and smaller than the maximum luminance H through the processing of steps S91, S93, S95, and S102. A color correction function fC (Y) is defined.
fC (Y) = αH CH ≦ Y <H (20)
[0108]
For the input luminance Y classified into the region having the maximum luminance H or higher through the processing of steps S91, S93, S95, and S102, in step S104, as shown in the following equation (21), the color correction function fC (Y) is defined.
fC (Y) = αH H <Y (21)
[0109]
The color correction function fC (Y) is generated by the processing of the color correction function generation unit 123 as described above.
[0110]
The mapping unit 124 calculates the color correction amount γ by applying the luminance Y of each pixel from the luminance signal generation unit 121 to the color correction function fC (Y), and outputs it to the power calculation units 127, 128R, 128G, and 128B. To do.
[0111]
Similar to the mapping function generation unit 82 described above, the mapping function generation unit 125 is based on the luminance Y of each pixel of the wide dynamic range image and the target luminance / optimal target luminance pair (M, Mm), and the mapping function f (Y). Is generated and output to the mapping unit 126. The mapping unit 126 applies the luminance Y of each pixel of the wide dynamic range image to the mapping function f (Y) from the mapping function generation unit 125 to generate the luminance Ym of each pixel of the narrow dynamic range image, and the scaling unit Output to 129R, 129G, and 129B.
[0112]
The power calculation unit 127 raises the luminance Y of each pixel of the wide dynamic range image to the γ power, and outputs the obtained corrected luminance Yγ to the scaling units 129R, 129G, and 129B. The power calculation unit 128R multiplies the red signal R of each pixel of the wide dynamic range image by γ and outputs the obtained corrected red signal Rγ to the scaling unit 129R. The power calculation unit 128G multiplies the green signal G of each pixel of the wide dynamic range image by γ and outputs the obtained corrected green signal Gγ to the scaling unit 129G.
The power calculation unit 128B multiplies the blue signal B of each pixel of the wide dynamic range image by γ and outputs the obtained corrected blue signal Bγ to the scaling unit 129B.
[0113]
The scaling units 129R to 129B calculate the red signal Rm, the green signal Gm, or the blue signal Bm of each pixel of the narrow dynamic range image using the following equations (22) to (24), respectively.
Rm = Rγ · Ym / Yγ (22)
Gm = Gγ · Ym / Yγ (23)
Bm = Bγ · Ym / Yγ (24)
[0114]
As described above, according to the second configuration example of the image processing unit 11, natural colors corresponding to the degree to which the dynamic range is compressed with respect to the three primary color signals R, G, and B of the input wide dynamic range image. Since correction is performed so as to achieve balance, a color image with a wide dynamic range can be converted into a color image with a narrow dynamic range in which color balance is not unnatural.
[0115]
Next, FIG. 28 shows a configuration example of a digital camera to which the present invention is applied.
In other words, the digital camera 140 has the image processing system shown in FIG. 1 housed in a single housing. The digital camera 140 captures a subject as a wide dynamic range image signal and appropriately converts it into a narrow dynamic range image signal. Record in the built-in memory 148.
[0116]
The digital camera 140 includes a lens 141 that collects a light image of a subject, a diaphragm 142 that adjusts the light amount of the light image, and a CCD image sensor 143 that photoelectrically converts the collected light image into an electric signal with a wide dynamic range. , CDS (Corelated Double Sampling) 144 that reduces noise by sampling the electrical signal from the CCD image sensor 143, A / D converter 145 that digitizes the analog electrical signal, and images of the digitized electrical signal It comprises a DSP (Digital Signal Processor) 146 comprising an image signal processor for converting into a signal or compressing the dynamic range and an image RAM.
[0117]
Further, the digital camera 140 compresses and encodes the image signal processed by the DSP 146 and records it in the memory 148, reads out, decompresses, and supplies the codec (Compression / Decompression) 147 and the image signal processed by the DSP 146 supplied to the DSP 146. A D / A converter 149 for analogization, a video encoder 150 for encoding the analogized image signal into a video signal in a format suitable for the display unit 151 in the subsequent stage, and an image corresponding to the video signal as a finder The display unit 151 includes a functioning LCD (Liquid Crystal Display) or the like.
[0118]
Further, the digital camera 140 controls the drive 153 to read out the control program stored in the magnetic disk 154, the optical disk 155, the magneto-optical disk 156, or the semiconductor memory 157, and read out the control program and operation unit A control unit 152 including a CPU that controls the entire digital camera 140 based on a command from the user input from 158, an operation unit 158 for the user to input shutter timing and other commands, and a CCD image sensor The timing generator 159 controls the operation timings of the 143 to DSP 146.
[0119]
In the digital camera 140, the DSP 146 corresponds to the image processing unit 11 of the image processing system described above, and executes processing for converting a wide dynamic range image signal into a narrow dynamic range image signal.
[0120]
The operation unit 158 and the display unit 151 constituting the digital camera 140 are compared with the operation input unit 12 and the display unit 13 constituting the image processing apparatus 2 (comprising a personal computer or the like) of the image processing system of FIG. Since a small one is used, it is necessary to enable the user to execute various operation inputs more easily.
[0121]
Therefore, in the digital camera 140, the overall processing order is different from the processing order of the image processing system of FIG. 1 described with reference to the flowchart of FIG. 7, and the optimum exposure information is obtained before acquiring the wide dynamic range image signal. Has been made to set.
[0122]
A first operation example of the digital camera 140 will be described with reference to a flowchart of FIG. In step S111, the composition of the image to be captured is selected by the user. The composition of the selected image (corresponding to a guide 172 described later) is set as optimum exposure information.
[0123]
Processing for setting the optimum exposure information will be described with reference to FIGS. FIG. 30 shows a display example of the image display area 171 of the display unit 151 that functions as a viewfinder. As shown in the figure, in the image display area 171, a guide 172 indicated by a thick line or the like and an auxiliary line indicated by a broken line are superimposed and displayed on the image of the subject. A plurality of patterns are prepared as the shape of the guide 172, and the shape is switched every time the guide selection panel 181 (FIG. 32) provided in the operation unit 158 is operated by the user. .
[0124]
31A to 31Q show examples of the shape of the guide 172 prepared in advance. Each shape of the guide 172 is given a predetermined index.
The shape of the guide 172 may be, for example, a circle or a polygon other than the rectangle shown in FIG.
[0125]
FIG. 32 shows a guide selection panel 181 provided in the operation unit 158.
The guide switching button 182 is pressed when switching the shape of the guide 172 to that corresponding to the previous index. The switch button 183 is pressed when the shape of the guide 172 is switched to one corresponding to the next index. The selection button 184 is pressed when the shape of the displayed guide 172 is confirmed.
[0126]
For example, FIG. 30 shows a state in which the shape of the guide 172 is switched to the example of FIG. 31 (J). In this state, when the switch button 182 is pressed, the shape of the guide 172 is changed. When the example is switched to the example in FIG. 10I and the switch button 183 is pressed, the shape of the guide 172 is switched to the example in FIG.
[0127]
The user switches the guide 172 by operating the switch buttons 182 and 183 of the guide selection panel 181 so that the subject (for example, a person) fits in the guide 172 while looking at the display unit 151 functioning as a finder. A selection button 184 is pressed to determine the shape of the guide 172. In response to pressing of the selection button 184, an index corresponding to the shape of the currently displayed guide 172 is set as optimum exposure information.
[0128]
Returning to FIG. In step S112, when the user operates the shutter provided on the operation unit 158, a wide dynamic range image signal is acquired and stored in the image RAM built in the DSP 146.
[0129]
In step S113, the wide dynamic range image signal is converted into the narrow dynamic range image signal based on the optimum exposure information (index indicating the shape of the guide 172) set in step S111.
[0130]
Specifically, for the target luminance / optimal target luminance pair, an image signal of the internal region of the guide 172 is acquired based on the optimal exposure information (index indicating the shape of the guide 172), and a histogram of the luminance is generated. The luminance indicating the highest frequency is set as the target luminance M. As the optimum target brightness Mm, an intermediate brightness with a preset narrow dynamic range is set. The subsequent processing is the same as the second operation example described above with reference to FIGS. 24 and 25 when the image processing unit 11 of the image processing system sets the optimum exposure information.
[0131]
The converted narrow dynamic range image signal is stored in the memory 148.
[0132]
Note that, in the DSP 146 of the digital camera 140, similarly to the second configuration example of the image processing unit 11 of the image processing system shown in FIG. 26, a color wide dynamic range image can be obtained without impairing the color balance. You may make it perform the process converted into a narrow dynamic range image.
[0133]
Next, a second operation example of the digital camera 140 will be described with reference to the flowchart of FIG. In step S121, the composition of the image to be captured is selected by the user. The composition of the selected image (corresponding to the guide 172) is set as the first optimum exposure information.
[0134]
The process for setting the first optimum exposure information is the same as the process for setting the optimum exposure information in the first operation example described above with reference to FIGS. 30 to 32.
[0135]
In step S122, when the user operates the shutter provided in the operation unit 158, a wide dynamic range image signal is acquired and stored in the image RAM built in the DSP 146.
[0136]
In step S123, the exposure ratio r is set as the second optimum exposure information corresponding to the user's operation. Processing for setting the exposure ratio r as the second optimum exposure information will be described with reference to FIGS. 34 and 35. FIG.
[0137]
FIG. 34 shows a display example of the image display area 171 of the display unit 151 in step S123. In this case, the image display area 171 displays the wide dynamic range image acquired in step S122, and the guide 172 and the auxiliary line (broken line) set in step S121 are superimposed on the image. However, in the internal area of the guide 172 in the image display area 171, it is simply generated using an exposure ratio r that is changed in response to a user operation on the brightness correction panel 191 (FIG. 35) provided in the operation unit 158. A narrow dynamic range image is displayed.
[0138]
FIG. 35 shows the brightness correction panel 191 provided in the operation unit 158. The bright correction (BRIGHTER) button 192 is pressed to increase the brightness of the narrow dynamic range image displayed in the inner area of the guide 172 by one level (increase the brightness) from the current level. The dark correction (DARKER) button 193 is pressed when the brightness of the narrow dynamic range image displayed in the internal area of the guide 172 is darkened by one level (decreasing brightness) from the current level. An OK button 194 is pressed to determine the current brightness.
[0139]
For example, when the bright correction button 192 is pressed, the exposure ratio r used for simple generation of a narrow dynamic range image is increased by a predetermined value. Therefore, the brightness of the narrow dynamic range image in the inner area of the guide 172 is increased and displayed. On the other hand, when the dark correction button 193 is pressed, the exposure ratio r used for simple generation of a narrow dynamic range image is decreased by a predetermined value. Therefore, the brightness of the narrow dynamic range image in the inner area of the guide 172 is reduced and displayed. When the OK button 194 is pressed, the current exposure ratio r is set as the second optimum exposure information.
[0140]
Returning to FIG. In step S124, based on the first optimum exposure information (index indicating the shape of the guide 172) set in step S121 and the second optimum exposure information set in step S123, the wide dynamic range image signal is narrowly dynamic. It is converted into a range image signal.
[0141]
Specifically, for the target luminance / optimal target luminance pair, the luminance of the wide dynamic range image in the inner region of the guide 172 is acquired based on the first optimal exposure information (index indicating the shape of the guide 172), A luminance histogram is generated, and the luminance indicating the highest frequency is set as the target luminance M.
[0142]
As the optimum attention brightness Mm, the brightness of the wide dynamic range image is multiplied by the exposure ratio r as the second optimum exposure information to easily generate the narrow dynamic range image, and the brightness of the inner region of the guide 172 is extracted. Thus, a histogram is generated, and the luminance indicating the highest frequency is set as the optimum attention luminance Mm. The subsequent processing is the same as the second operation example described above with reference to FIGS. 24 and 25 when the image processing unit 11 of the image processing system sets the optimum exposure information.
[0143]
The converted narrow dynamic range image signal is stored in the memory 148.
[0144]
As described above, according to the second operation example of the digital camera 140, an area that the user wants to pay attention to can be specified with the guide 172 by a simple operation on the operation unit 158. Since the desired luminance can be designated for the image, the gradation of the obtained narrow dynamic range image is closer to the result desired by the user.
[0145]
Note that the DSP 146 of the digital camera 140 may be caused to execute processing similar to that of the image processing unit 11 of the image processing system described above. Conversely, the image processing unit 11 of the image processing system may be caused to execute the same processing as the DSP 146 of the digital camera 140.
[0146]
By the way, the present invention can also be applied to the case where the luminance is changed without changing the dynamic range of the image.
[0147]
Further, the present invention can be applied not only to the image conversion system and the digital camera as in the present embodiment, but also to electronic devices that process image signals such as scanners, facsimiles, and copiers.
[0148]
In the present specification, the step of describing the program recorded in the recording medium is not limited to the processing performed in time series according to the described order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.
[0149]
Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.
[0150]
【The invention's effect】
  As described above, the present inventionInAccording toWideWhen converting a dynamic range image into a narrow dynamic range image, it is possible to optimize the luminance at an arbitrary position in the image.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an image processing system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a configuration example of the imaging apparatus 1 in FIG.
3 is a block diagram illustrating a configuration example of a CCD image sensor 23 in FIG. 2;
FIG. 4 is a diagram for explaining sensitivity characteristics of the CCD image sensor 23;
FIG. 5 is a diagram for explaining processing of the preamplifier 24;
6 is a block diagram illustrating a configuration example of a personal computer that implements the image processing apparatus 2 of FIG. 1;
FIG. 7 is a flowchart illustrating the operation of the image processing system.
FIG. 8 is a flowchart for explaining optimum exposure information setting processing of the image processing apparatus 2;
FIG. 9 is a flowchart illustrating a process for the image processing apparatus 2 to easily generate a narrow dynamic range image.
FIG. 10 is a diagram showing a display example of the display unit 13 for explaining a first operation example for setting optimum exposure information.
FIG. 11 is a diagram showing a GUI display example in the first operation example for setting optimum exposure information.
12 is a block diagram illustrating a first configuration example of an image processing unit 11. FIG.
13 is a flowchart for explaining processing of a luminance pair detection unit 81 in FIG.
FIG. 14 is a diagram for explaining processing of a luminance pair detection unit 81;
15 is a block diagram illustrating a configuration example of a mapping function generation unit 82 in FIG.
FIG. 16 is a flowchart illustrating processing of a mapping function generation unit.
FIG. 17 is a diagram for explaining processing of a maximum / minimum luminance acquisition unit 91 in FIG. 15;
FIG. 18 is a diagram for explaining processing of a mapping function generation unit 82;
FIG. 19 is a diagram for explaining the processing of the mapping function generation unit 82;
FIG. 20 is a diagram for explaining processing of a mapping function generation unit 82;
FIG. 21 is a diagram for explaining processing of the mapping function generation unit 82;
FIG. 22 is a flowchart illustrating details of the process in step S50 of FIG.
FIG. 23 is a flowchart for explaining processing of the mapping unit 83 in FIG. 12;
FIG. 24 is a diagram showing a display example of the display unit 13 for explaining a second operation example for setting optimum exposure information.
FIG. 25 is a diagram showing a GUI display example in the second operation example for setting optimum exposure information.
26 is a block diagram illustrating a second configuration example of the image processing unit 11. FIG.
FIG. 27 is a flowchart illustrating processing of a second configuration example of the image processing unit.
FIG. 28 is a block diagram illustrating a configuration example of a digital camera 140 according to an embodiment of the present invention.
29 is a flowchart for explaining a first operation example of the digital camera 140. FIG.
30 is a diagram showing a display example of the display unit 151 for explaining processing for setting optimum exposure information in the first operation example of the digital camera 140. FIG.
31 is a diagram showing types of shapes of guides 172 displayed on the display unit 151. FIG.
32 is a diagram showing a guide selection panel 181 provided in the operation unit 158 of the digital camera 140. FIG.
FIG. 33 is a flowchart for explaining a second operation example of the digital camera.
34 is a diagram showing a display example of the display unit 151 for explaining processing for setting optimum exposure information in the first operation example of the digital camera 140. FIG.
35 is a diagram showing a luminance correction panel 191 provided in the operation unit 158 of the digital camera 140. FIG.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Imaging device, 2 Image processing device, 11 Image processing part, 12 Operation input part, 13 Display part, 41 CPU, 52 Magnetic disk, 53 Optical disk, 54 Magneto-optical disk, 55 Semiconductor memory, 71 Image selection button panel, 81 Brightness Pair detection unit, 82 mapping function generation unit, 83 mapping unit, 91 maximum / minimum luminance acquisition unit, 92 high frequency function determination unit, 93 low frequency function determination unit, 94 function synthesis unit, 111 range specification panel, 121 luminance signal generation Unit, 122 luminance pair detection unit, 123 color correction function generation unit, 124 mapping unit, 125 mapping function generation unit, 126 mapping unit, 127, 128 exponentiation operation unit, 129 scaling unit, 140 digital camera, 146 DSP, 152 control unit , 154 Magnetic Disc, 155 disc, 156 a magneto-optical disk, 157 a semiconductor memory, 158 operation unit, 172 guide 181 guides selection panel, 191 luminance correction panel

Claims (6)

In an image processing apparatus for converting a first image signal having a first dynamic range into a second image signal having a second dynamic range with respect to a luminance value,
Setting means for setting a second luminance value in the second dynamic range;
Computing means for computing a first luminance value in the first dynamic range corresponding to the second luminance value;
Mapping function generating means for generating a mapping function based on a first luminance pair consisting of the first luminance value and the second luminance value;
And a converting means for converting the first image signal into the second image signal using the mapping function that is generate,
The mapping function generation means includes
A point corresponding to the first luminance pair;
A point corresponding to a second luminance pair consisting of a third luminance value near the minimum luminance value in the first dynamic range and a fourth luminance value near the minimum luminance value in the second dynamic range;
And a point corresponding to a third luminance pair consisting of a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range
An image processing apparatus that generates a smooth monotonically increasing function passing through the vicinity of the three points as the mapping function . The mapping function generation means includes
First function generating means for generating as a first monotonically increasing function a first function passing through a point corresponding to the first luminance pair and a point corresponding to the second luminance pair ;
Second function generating means for generating, as a second function, a monotonically increasing function that passes through a point corresponding to the first luminance pair and a point corresponding to the third luminance pair ;
A third function that generates, as the mapping function , a third function that is a monotonically increasing function passing through a point on the first function and a point on the second function in the vicinity of the point indicated by the first luminance pair. Including function generation means of
The image processing apparatus according to 請 Motomeko 1. The third function generation means includes:
A predetermined point indicated by the first function,
1 point showing the first luminance pair,
And a quadratic curve function defined by three predetermined points indicated by the second function is generated as the third function.
The image processing apparatus according to 請 Motomeko 2. Calculation means for calculating correction information for correcting the color balance of the three primary color signals constituting the second image signal based on the mapping function;
Correction means for correcting the three primary color signals constituting the second image signal using the correction information ; and
The calculation means acquires the gamma characteristic of the mapping function, and the three primary color signal so that a ratio between the corrected three primary color signal and the luminance value after applying the mapping function is a ratio calculated by the gamma characteristic. The image processing apparatus according to claim 1 , wherein a correction coefficient is determined . In an image processing method of an image processing apparatus for converting a first image signal having a first dynamic range into a second image signal having a second dynamic range with respect to a luminance value,
A setting step of setting a second luminance value in the second dynamic range;
A calculation step of calculating a first luminance value in the first dynamic range corresponding to the second luminance value;
A mapping function generating step for generating a mapping function based on a first luminance pair comprising the first luminance value and the second luminance value;
Look including a conversion step of converting the first image signal into the second image signal using the mapping functions generated and
The mapping function generation step includes
A point corresponding to the first luminance pair;
A point corresponding to a second luminance pair consisting of a third luminance value near the minimum luminance value in the first dynamic range and a fourth luminance value near the minimum luminance value in the second dynamic range;
And a point corresponding to a third luminance pair consisting of a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range
An image processing method for generating, as the mapping function, a smooth monotonically increasing function that passes in the vicinity of the three points . A program for image processing that converts a first image signal having a first dynamic range into a second image signal having a second dynamic range with respect to a luminance value,
A setting step of setting a second luminance value in the second dynamic range;
A calculation step of calculating a first luminance value in the first dynamic range corresponding to the second luminance value;
A mapping function generating step for generating a mapping function based on a first luminance pair comprising the first luminance value and the second luminance value;
Look including a conversion step of converting the first image signal into the second image signal using the mapping functions generated and
The mapping function generation step includes
A point corresponding to the first luminance pair;
A point corresponding to a second luminance pair consisting of a third luminance value near the minimum luminance value in the first dynamic range and a fourth luminance value near the minimum luminance value in the second dynamic range;
And a point corresponding to a third luminance pair consisting of a fifth luminance value near the maximum luminance value in the first dynamic range and a sixth luminance value near the maximum luminance value in the second dynamic range
A smooth monotonically increasing function passing through the vicinity of the three points is generated as the mapping function
A recording medium on which a program for causing a computer to execute processing is recorded.

Images