JP2004005303A - Image search system, image searching method, and image search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem of a current image searching method wherein similar images corresponding to similar-sense of a human being can not be obtained. <P>SOLUTION: The described problem can be solved by using the image searching method containing a feature volme extraction process S404 for extracting a plurality of visual feature volume of a corresponding image from each image as a subject of search contained in a reference image and the described image group as a subject, a feature volume selecting process S405 for selecting feature volume of the corresponding image as a subject of search approximate to the feature quantity of the reference image by the volume enough to satisfy the predetermined condition, and a similar image extracting process S409 for extracting an image similar to the reference image from the described image group as a subject of search. <P>COPYRIGHT: (C)2004,JPO

Description

さらに、小領域のうちｉ番目（ｉ＝０，１，・・・Ｎ−１）の小領域をＳ_ｉとすると、小領域Ｓ_ｉにおける平均明度Ｌ_ｉおよび平均色Ｃ_ｉはそれぞれ数式（２）および数式（３）で表される。ここで、数式（２）および数式（３）におけるＰ_ｉは、小領域Ｓ_ｉ内の画素数とする。
【数２】

全ての小領域Ｓ_ｉについて平均明度Ｌ_ｉおよび平均色Ｃ_ｉを求め、それらを要素とする特徴ベクトルを数式（４）および数式（５）によって求め、それぞれを基準画像に対する明度の特徴量Ｌ_ｋｅｙおよび色の特徴量Ｃ_ｋｅｙとする。ここで、数式（５）におけるベクトル表記は、数式（６）のように各要素を羅列したベクトルを意味するものとする。
【数３】

一方、基準画像のエッジに関する特徴量は以下の手順で求められる。本画像検出方法では、Ｓｏｂｅｌオペレータによる水平方向および垂直方向のエッジ検出を利用したエッジに関する特徴量抽出の例を示す。
【００４２】
画像上の位置（ｘ，ｙ）における水平方向のエッジの強度Ｅｘ（ｘ，ｙ）および垂直方向のエッジの強度Ｅｙ（ｘ，ｙ）は、それぞれ数式（７）および数式（８）によって表される。ここで、数式（７）および数式（８）における‖Ａ‖は、Ａのノルムを意味する。
【数４】

ここでは、色の不連続点も画像のエッジとしてｐ（ｘ，ｙ）を用いたが、色を考慮せず画像の明度のみからエッジの強度を求めたい場合には、ｐ（ｘ，ｙ）の代わりにＬ^＊（ｘ，ｙ）を用いれば良い。小領域Ｓ_ｉにおける水平方向のエッジの平均強度Ｅ_ｘｉおよび垂直方向のエッジの平均強度Ｅ_ｙｉは、それぞれ数式（９）および数式（１０）で表される。
【数５】

さらに、数式（９）および数式（１０）によって得られた水平方向および垂直方向のエッジの平均強度Ｅ_ｘｉ，Ｅ_ｙｉを用いて、小領域Ｓ_ｉのエッジの強度Ｅ_ｉを数式（１１）で定義する。
【数６】

全ての小領域Ｓ_ｉに対するエッジの強度Ｅ_ｉを求め、それらを要素とする特徴ベクトルを数式（１２）によって求め、基準画像データに対するエッジの特徴量Ｅ_ｋｅｙとする。ここで、数式（１２）の表記は、数式（６）のように各要素を羅列したベクトルを意味するものとする。
【数７】

以上により、基準画像データから明度、色およびエッジに関する特徴量が、それぞれ特徴ベクトルＬ_ｋｅｙ，Ｃ_ｋｅｙおよびＥ_ｋｅｙとして抽出される。
【００４３】
本画像検索方法では、画像を小領域Ｓ_ｉに分割した上で明度、色およびエッジに関する特徴量を抽出したが、本発明の画像検索方法はこれに限定されるものではない。例えば、画像を小領域Ｓ_ｉに分割することなく、画像全体を１つの領域として特徴量Ｌ_ｋｅｙ，Ｃ_ｋｅｙおよびＥ_ｋｅｙを抽出しても良い。
【００４４】
ステップＳ４０３からＳ４０４では、検索対象画像データから特徴量の抽出を行う。検索対象画像データは、データベース部２１０に予め定められた順序で格納および保持されているものとし、検索対象画像データのいずれか１つを示すポインタ（アドレス）が最初の画像データを示すように初期化されているものとする。ステップＳ４０３では、制御部２００は、データベース部２１０からポインタが示す画像データを読み出す。
【００４５】
ステップＳ４０４では、読み出された画像データから明度、色およびエッジに関する特徴量が抽出される。各特徴量の抽出は、上記の基準画像に対するものと同様に数式（１）〜数式（１２）を用いて行われ、それぞれ特徴ベクトルＬ_ｓｕｂｊ，Ｃ_ｓｕｂｊおよびＥ_ｓｕｂｊとして抽出される。ここで、ｊ＝０，１，・・・は各画像データに固有に割り当てられた識別番号とする。
【００４６】
ステップＳ４０５では、基準画像からの特徴量Ｌ_ｋｅｙ，Ｃ_ｋｅｙおよびＥ_ｋｅｙと、検索対象画像データからの特徴量Ｌ_ｓｕｂｊ，Ｃ_ｓｕｂｊおよびＥ_ｓｕｂｊとを比較し、類似度を算出するために用いられる特徴量を選択する。
【００４７】
基準画像の各特徴量を示す特徴ベクトルと、検索対象画像データの各特徴量を示す特徴ベクトルとの比較は、数式（１３）から数式（１５）に示す相関係数Ｒ_Ｌ，Ｒ_ｃおよびＲ_Ｅを算出することによって行う。ここで、Ａ・ＢはベクトルＡとベクトルＢとの内積を意味する。
【数８】

特徴量の選択は、相関係数Ｒ_Ｌ，Ｒ_ＣおよびＲ_Ｅが予め定めた閾値ＴＲを超える特徴量を選択することによって行う。
【００４８】
以下に、例として、図１における画像１０１を基準画像、画像１０３を検索対象画像データとした場合の特徴量の選択について説明する。画像１０１は、星が黄色、葉が緑、幹が茶色で彩色された画像である。これに対して、画像１０３は、画像１０１と対応する部分の明度が等しい単一色が施された単色画像である。両画像では、各部の明度は等しいため、明度の特徴量Ｌ_ｋｅｙとＬ_ｓｕｂｊは等しくなる。一方、色の特徴量Ｃ_ｋｅｙとＣ_ｓｕｂｊは著しく異なる。エッジの特徴量Ｅ_ｋｅｙとＥ_ｓｕｂｊは、色が異なるため全く等しくはならないが、画像上の同じ位置に同じ方向のエッジを有するためＥ_ｋｅｙとＥ_ｓｕｂｊは高い相関を持つこととなる。従って、各特徴量に対する相関係数は、例えば、Ｒ_Ｌ＝１．０，Ｒ_Ｃ＝０．１およびＲ_Ｅ＝０．８となる。特徴量を選択するための閾値ＴＲを０．５とすると、これを上回るＲ_Ｌ及びＲ_Ｅが選択される。
【００４９】
特徴量の選択方法は上記の相関係数を用いる方法に限られるものではなく、対応する２つの特徴量の類似性を比較できる方法であれば良い。例えば、相関係数の代わりに２つの特徴ベクトル間のユークリッド距離を用いて選択しても良い。この場合、特徴ベクトル間のユークリッド距離は、対象ベクトル間の非類似性を示すので、予め定めた閾値ＴＲより小さいユークリッド距離を有する特徴量を選択する。
【００５０】
さらに、ユーザ自身が特徴の選択を行うことにより、ユーザが注目する画像検索の意図を反映したシステムを構築することも可能である。この場合、出力部２０４にユーザインターフェースを表示し、ユーザが選択する特徴量をコマンド入力部２０６から指定して選択する。
【００５１】
ステップＳ４０６では、類似度の算出を行う。類似度とは、選択された特徴量に対して基準画像と検索対象画像データとの間の近似性を示す評価値をいう。具体的には、例えば、図１の画像１０１を基準画像、画像１０３を検索対象画像データとし、ステップＳ４０５において明度およびエッジに関する特徴量が選択された場合には、数式（１６）を用いて類似度Ｒを算出することができる。
【数９】

数式（１６）では、基準画像と検索対象画像データにおいて選択された特徴量間の相関係数を用いている。一般に画像からＭ種類の特徴量が選択される場合、基準画像の特徴量をＦ_{ｊ，ｋｅｙ}、検索対象画像データの特徴量をＦ_{ｊ，ｓｕｂｊ}とすると、両者の相関係数による類似度Ｒは数式（１７）で表すことができる。ここで、Ｇは、ステップＳ４０５で選択された特徴量の集合を示す。なお、ステップＳ４０５で選択された特徴量の数が０である場合はＲ＝０とする。
【数１０】

類似度の算出方法は相関係数を用いる方法に限られるものではなく、基準画像と検索対象画像データについて選択された特徴量の類似性を評価できる方法であれば良い。例えば、数式（１８）のように、相関係数の代わりに特徴量ベクトル間のユークリッド距離Ｄを用いても良い。この場合、ベクトル間のユークリッド距離Ｄは２つのベクトルの非類似性を示すので、ユークリッド距離Ｄの値が小さいほど類似性は高くなる。
【数１１】

ステップＳ４０７では、基準画像と検索対象画像データとの類似度が、検索対象画像データと関連付けられて標準記憶部２０２又はデータベース部２１０へ格納および保持される。
【００５２】
ステップＳ４０８では、検索対象となる画像データ群に含まれる全ての画像データについて、基準画像との類似度が算出されたか否かが判断される。検索対象画像データは、データベース部２１０に予め定められた順序で保持されているので、上記のＳ４０３からＳ４０７で処理の対象となった検索対象画像データを示すポインタが最後のものであるか否かを判断し、最後のポインタでなければポインタを次の値として処理をステップＳ４０３へ戻す。最後のポインタであればステップＳ４０９へ処理を移行する。
【００５３】
ステップＳ４０９では、類似度を相関係数を用いて算出した場合は、標準記憶部２０２又はデータベース部２１０に保持されている類似度を降順に並び替える。類似度をユークリッド距離などの非類似性を示す評価値として算出した場合には昇順に並び替える。その結果、最大の類似度を有する検索対象画像データを基準画像データに類似する画像の画像データとして抽出する。また、類似する画像を複数抽出したい場合には、所定の閾値以上の類似度を有する検索対象画像データを抽出することも好適である。
【００５４】
ステップＳ４１０では、類似度の並び替えの結果を用いて、検索結果の表示を行う。検索結果は、最大の類似度を有する検索対象画像データを出力部２０４に表示することができる。また、類似度が高いものから順に所定の数だけ選択し、それらに関連付けられた検索対象画像データを出力部２０４に表示することも好適である。さらに、予め閾値を定めておき、この閾値を超える類似度を有する検索対象画像データを出力部２０４に表示することも好適である。この場合、閾値を超える類似度を有する検索対象画像データが存在しないときには「類似画像無し」等の表示を行うこともできる。
【００５５】
以上のように、第１の画像検索方法によれば、基準画像と検索対象画像を比較して適切な特徴量を選択し、検索対象となる画像データ群から基準画像と人間の類似感覚と一致した類似性を有する画像を検索することができる。
【００５６】
＜第２の画像検索方法＞
上記第１の画像検索方法では、基準画像と検索対象画像データとの特徴の一部のみが異なる場合に、基準画像と検索対象画像データとが全く同一である場合と等しい類似度が算出されることがある。例えば、図１において、画像１０１が基準画像、画像１０３が検索対象画像データである場合に、両画像は明度が同一であるため、明度に関する特徴量Ｌ_ｋｅｙとＬ_ｓｕｂｊとは等しく、色に関する特徴量Ｃ_ｋｅｙとＣ_ｓｕｂｊは大きく異なるものとなる。また、エッジに関する特徴量Ｅ_ｋｅｙとＥ_ｓｕｂｊとは高い相関を示すものとなる。このとき、特徴量を選択するための閾値を高く設定すると、明度に関する特徴量のみが選択される。数式（１７）を用いて類似度を算出した場合に類似度Ｒは１となり、基準画像と検索対象画像データが全く同一である場合と等しくなってしまう。
【００５７】
このような問題を回避するために、選択された特徴量に応じて類似度を修正する方法が考えられる。以下に、類似度を修正して画像データを検索する第２の画像検索方法について、図を参照して詳細に説明する。
【００５８】
図６に、本画像検索方法における画像検索方法を概念的に表したブロック図を示す。検索キーとなる基準画像の画像データは、特徴量抽出手段６０１へ入力され、少なくとも１つの特徴量６０３が抽出される。同様に、検索対象となる画像データの各々は、特徴量抽出手段６０２へ入力され、基準画像データと同一の種類の特徴量６０４が抽出される。特徴量６０３，６０４は、特徴量選択手段６０５へ入力され、それぞれ同種の特徴量同士が比較され、類似性の高い特徴量のみが選択される。類似性の高い特徴量が選択され、基準画像データ及び検索対象画像データのそれぞれに対する選択された特徴量６０６，６０７として出力される。選択された特徴量６０６，６０７は、類似画像抽出手段６０８に入力される。一方、特徴量選択手段６０５からは、特徴量を選択した際の情報として特徴量選択情報６１０が出力される。特徴量選択情報６１０は、類似度修正係数算出手段６１１に入力され、類似度修正係数６１２が求められる。類似度修正係数６１２は、類似画像抽出手段６０８に入力され、選択された特徴量６０６，６０７に基づいて基準画像データと検索対象画像データの類似度が算出され、その類似度は類似度修正係数６１２によって修正されて、最終的な類似度として求められる。類似画像抽出手段６０８では、以上の処理を、検索対象となる全ての画像データに対して順次行うことによって、より高い類似度を有する検索対象画像を基準画像に類似する画像として抽出し、出力６０９とする。
【００５９】
上記処理は、具体的には上記画像検索装置において画像検索プログラムを実行することによって行われる。図７に、第２の画像検索方法における処理のフローチャートを示す。第２の画像検索方法の処理は、画像検索プログラムとして標準記憶部２０２に格納および保持され、制御部２００によって適宜読み出されて実行される。
【００６０】
図７のフローチャートにおいて、ステップＳ７０１からＳ７０５の処理は、上記第１の画像検索方法におけるステップＳ４０１からＳ４０５における処理と同一であるため説明を省略する。
【００６１】
ステップＳ７０６では、ステップＳ７０５における特徴量の選択状況に基づいて、類似度修正係数ｑを算出する。類似度修正係数ｑは、例えば、数式（１９）を用いて求めることができる。ここで、Ｄｉｍ（Ａ）はベクトルＡの次元数を示す。また、関数ｆ（ｘ）は引数ｘに対する単調増加関数であり、例えば、図８に示すような関数である。
【数１２】

数式（１９）を用いることによって、ステップＳ７０５で選択された特徴量の種類が多いほど類似度修正係数ｑの値は大きくなる。例えば、図８に示す関数ｆ（ｘ）を用いた場合、全ての特徴量が選択されるとｑ＝１．０となる。
【００６２】
また、図１における画像１０１が基準画像データ、画像１０３が検索対象画像データである場合、ステップＳ７０５において明度に関する特徴量のみが選択されたとすると、Ｄｉｍ（Ｌ_ｋｅｙ）＝Ｎ、Ｄｉｍ（Ｃ_ｋｅｙ）＝２ＮおよびＤｉｍ（Ｅ_ｋｅｙ）＝２Ｎであるから、数式（１９）の引数はＮ／（Ｎ＋２Ｎ＋２Ｎ）＝０．２となる。図８を用いると、類似度修正係数ｑは０．７６となる。
【００６３】
ステップＳ７０７では、類似度修正係数ｑによる修正を加えた類似度Ｒを算出する。類似度の算出方法には、基準画像データと検索対象画像データとについて選択された特徴量の相関係数を算出する数式（１７）に対して、類似度修正係数ｑを用いて修正を行った数式（２０）を用いることができる。
【数１３】

上記例では、類似度修正係数ｑは０．７６となるので、数式（２０）を用いると、類似度Ｒは０．７６となる。このように、選択された特徴量の種類数に応じて類似度Ｒの値が修正され、基準画像データと検索対象画像データとの違いを適切に反映した類似度を算出することができる。
【００６４】
また、相関係数の代わりに２つの選択された特徴量を示す特徴ベクトル間のユークリッド距離Ｄを用いる場合には、ユークリッド距離Ｄを類似度修正係数ｑによって修正した数式（２１）を用いることが好適である。
【数１４】

ステップＳ７０８からＳ７１１の処理は、上記第１の画像検索方法におけるステップＳ４０７からＳ４１０における処理と同一であるため説明を省略する。
【００６５】
以上のように、第２の画像検索方法によれば、選択された特徴量の種類数に応じて、より適切に基準画像と検索対象画像データとの類似性を評価して画像を検索することができる。
【００６６】
＜第３の画像検索方法＞
第１および第２の画像検索方法では、予め定めた選択条件を満足する特徴量のみを選択して類似度を算出した。このとき、特徴量が選択条件の境界付近にある場合には、画像の微妙な差異によってその特徴量が選択されたり、選択されなかったりすることとなる。その結果、画像には大きな差異がないにも関わらず、算出される類似度には大きな差異を生ずる問題が発生する可能性がある。
【００６７】
このような問題を回避するために、特徴量の選択を行わず、特徴量を重み付けして類似度を算出する方法が考えられる。以下に、特徴量を重み付けして画像データを検索する第３の画像検索方法について、図を参照して詳細に説明する。
【００６８】
図９に、本画像検索方法を概念的に表したブロック図を示す。検索キーとなる基準画像の画像データは、特徴量抽出手段９０１へ入力され、少なくとも１つの特徴量９０３が抽出される。同様に、検索対象となる画像データの各々は、特徴量抽出手段９０２へ入力され、基準画像データと同一の種類の特徴量９０４が抽出される。特徴量９０３，９０４は、特徴量重み付け設定手段９０５へ入力され、特徴量の比較が行われ、その類似性に基づいて特徴量重み付け９０６が算出される。特徴量重み付け９０６は、特徴量９０３，９０４と共に類似画像抽出手段９０７に入力され、これらに基づいて基準画像データと検索対象画像データとの類似度が算出される。以上の処理を検索対象となる画像データに対して順次行うことによって、高い類似度を有する画像を基準画像に類似する画像として検索する。
【００６９】
上記処理は、具体的には上記画像検索装置において画像検索プログラムを実行することによって行われる。図１０に、第３の画像検索方法における処理のフローチャートを示す。第３の画像検索方法の処理は、画像検索プログラムとして標準記憶部２０２に格納および保持され、制御部２００によって適宜読み出されて実行される。
【００７０】
ステップＳ１００１からＳ１００４の処理は、上記第１の画像検索方法におけるステップＳ４０１からＳ４０４の処理と同一であるため説明を省略する。
【００７１】
ステップＳ１００５では、ステップＳ１００４で抽出された特徴量に対して特徴量の重み付けを行う。各特徴量の種類ごとに、特徴量重み付けｗ_ｊ（ｊ＝０，１・・・Ｍ−１）は数式（２２）を用いて求めることができる。数式（２２）における関数ｇ（ｘ）は引数ｘに対して単調増加関数であり、例えば、図１１に示すような関数とすることができる。
【数１５】

数式（１９）を図１１に示すような関数とすることによって、基準画像データと検索対象画像データから抽出された特徴量について、それぞれの特徴量間の相関係数が０．５未満の場合には特徴量重み付けｗ_ｊは０となり、相関係数が０．５以上の場合には特徴量重み付けｗ_ｊは相関係数に伴って増加する。
【００７２】
ステップＳ１００５では、さらに、特徴量重み付けｗ_ｊを用いて、数式（２３）及び数式（２４）のように各特徴量の重み付けを行う。
【数１６】

ステップＳ１００６では、数式（２３）及び数式（２４）によって重み付けされた特徴量を用いて、数式（２５）を用いて類似度Ｒを算出する。
【数１７】

ステップＳ１００７からＳ１０１０の処理は、上記第１の画像検索方法におけるステップＳ４０７からＳ４１０の処理と同一であるため説明を省略する。
【００７３】
以上のように、第３の画像検索方法によれば、閾値などの選択条件を用いて特徴量を選択することなく、抽出した全ての特徴量に対して適切な重み付けをすることによって、画像には大きな差異がなく、特徴量が選択条件の境界付近にある場合においても、算出される類似度に大きな差異を生ずる問題を回避することができる。その結果、基準画像データと検索対象画像データとの類似性を適切に評価して画像を検索することができる。
【００７４】
＜第４の画像検索方法＞
第３の画像検索方法において、特徴量重み付けｗ_ｊに基づいて類似度を修正することも好適である。以下に、特徴量重み付けｗ_ｊによって類似度を修正する第４の画像検索方法について、図を参照して詳細に説明する。
【００７５】
図１２に、本画像検索方法を概念的に表したブロック図を示す。検索キーとなる基準画像は、特徴量抽出手段１２０１へ入力され、少なくとも１つの特徴量１２０３が抽出される。同様に、検索対象となる画像データの各々は、特徴量抽出手段１２０２へ入力され、基準画像データと同一の種類の特徴量１２０４が抽出される。特徴量１２０３，１２０４は、特徴量重み付け設定手段１２０５へ入力され、特徴量の比較が行われ、その類似性に基づいて特徴量重み付け１２０６が算出される。特徴量重み付け設定手段１２０５では、さらに、類似度修正係数１２１１を算出する際に用いられる特徴量重み付け情報１２０９を出力する。特徴量重み付け情報１２０９は、類似度修正係数算出手段１２１０に入力され、類似度修正係数１２１１が求められる。特徴量重み付け１２０６及び類似度修正係数１２１１は、抽出された特徴量１２０３，１２０４と共に類似画像抽出手段１２０７へ入力され、これらに基づいて、基準画像データと検索対象画像データとの類似度が算出される。以上の処理を検索対象となる画像データに対して順次行うことによって、高い類似度を有する画像を基準画像に類似するものとして検索する。
【００７６】
上記処理は、具体的には上記画像検索装置において画像検索プログラムを実行することによって行われる。図１３に、第４の画像検索方法における処理のフローチャートを示す。第４の画像検索方法の処理は、画像検索プログラムとして標準記憶部２０２に格納および保持され、制御部２００によって適宜読み出されて実行される。
【００７７】
ステップＳ１３０１からＳ１３０５の処理は、上記第３の画像検索方法におけるステップＳ１００１からＳ１００５の処理と同一であるため説明を省略する。
【００７８】
ステップＳ１３０６では、ステップＳ１３０５で算出された各特徴量に対する重み付けｗ_ｊに基づいて、類似度修正係数ｑを算出する。類似度修正係数ｑは、例えば、数式（２６）を用いて求めることができる。ここで、Ｄｉｍ（Ａ）はベクトルＡの次元数を示す。また、関数ｆ（ｘ）は引数ｘに対する単調増加関数であり、例えば、図８に示すような関数である。
【数１８】

数式（２６）を用いることによって、各特徴量に対する重み付けｗ_ｊが大きくなるに伴って、類似度修正係数ｑも増加させることができる。
【００７９】
ステップＳ１３０７では、類似度修正係数ｑによって修正された類似度Ｒを算出する。類似度Ｒの算出方法には、重み付けされた特徴量から類似度Ｒを求める数式（２５）に対して、類似度修正係数ｑを用いて修正を行った数式（２７）を用いることができる。
【数１９】

ステップＳ１３０８からＳ１３１１の処理は、上記第３の画像検索方法におけるステップＳ１００７からＳ１０１０の処理と同一であるため説明を省略する。
【００８０】
以上のように、第４の画像検索方法によれば、特徴量に対する重み付けｗ_ｊの値に応じて類似度Ｒを修正することによって、より適切に基準画像と検索対象画像データとの類似性を評価して画像を検索することができる。
【００８１】
【発明の効果】
本発明によれば、白黒画像とカラー画像のように彩度や色相に関する特徴のみが異なる画像間、又は、線図と色塗りされた図のようにエッジの強さの特徴は共通する画像間において、従来の画像検索技術では検索が困難であった類似画像を検索することができる。すなわち、基準画像と検索対象画像データとにおいて類似する特徴を適切に選択し、人間の類似感覚と一致した類似画像を検索することを可能とした。
【図面の簡単な説明】
【図１】特徴の一部が類似した画像を例示する図である。
【図２】本発明の実施の形態における画像検索装置の構成図である。
【図３】本発明の第１の画像検索方法における処理を概念的に示すブロック図である。
【図４】本発明の第１の画像検索方法におけるフローチャートを示す図である。
【図５】画像の特徴量を抽出するための画像分割方法を説明する図である。
【図６】本発明の第２の画像検索方法における処理を概念的に示すブロック図である。
【図７】本発明の第２の画像検索方法におけるフローチャートを示す図である。
【図８】本発明の画像検索方法における関数ｆ（ｘ）の一例を示す図である。
【図９】本発明の第３の画像検索方法における処理を概念的に示すブロック図である。
【図１０】本発明の第３の画像検索方法におけるフローチャートを示す図である。
【図１１】本発明の画像検索方法における関数ｇ（ｘ）の一例を示す図である。
【図１２】本発明の第４の画像検索方法における処理を概念的に示すブロック図である。
【図１３】本発明の第４の画像検索方法におけるフローチャートを示す図である。
【符号の説明】
１０１　色分けされた画像、１０２　線図による画像、１０３　画像１０１と等しい明度の単色画像、１０４　画像１０１と同一色で描かれたスケッチ風の画像、２００　制御部、２０２　標準記憶部、２０４　出力部、２０６　コマンド入力部、２０８　画像入力部、２１０　データベース部、２１２　バス、２１４インターフェース部、２１６　ネットワーク、３０１，３０２，６０１，６０２，９０１，９０２，１２０１，１２０２　特徴量抽出手段、３０３，３０４，６０３，６０４，９０３，９０４，１２０３，１２０４　特徴量、３０５，６０５　特徴量選択手段、３０６，３０７，６０６，６０７　選択された特徴量、３０８，６０８，９０７，１２０７　類似画像抽出手段、３０９，６０９，９０８，１２０８　類似度、６１０　特徴量選択情報、６１１，１２１０　類似度修正係数算出手段、６１２，１２１１　類似度修正係数、９０５，１２０５　特徴量重み付け設定手段、９０６，１２０６　特徴量重み付け、１２０９　特徴量重み付け情報。[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image search device, an image search method, and an image search program for searching for an image having a common feature from a plurality of images.
[0002]
[Prior art]
With the development of networks in recent years and the increase in data storage capacity, a large number of image databases for storing a large amount of image information have been constructed. These image databases can be easily used through an information transmission medium such as the Internet. In addition, due to the spread of digital cameras, digital videos, and scanners, individual users often hold a large amount of image data. Furthermore, image data for use in business documents and presentation materials has been marketed as clip art collections and image material collections. Therefore, the necessity of an image search unit capable of appropriately extracting desired image data from a large amount of image data group is increasing.
[0003]
Currently, the most common form of image search means is keyword search. This is means for assigning a keyword indicating a feature of the image data to each of the image data, and performing a search using the keyword as a clue when extracting desired image data.
[0004]
However, unlike document data, in which automatic extraction of keywords is easy, it is necessary to manually assign keywords to image data, and enormous labor is required when constructing a database containing a large amount of image data. Become. Further, in the case of image data, the keyword to be assigned differs depending on which element of the image is focused, and it is difficult to assign a keyword according to a consistent standard.
[0005]
As another image search means, there is a similar image search means for performing a search using the image itself as a clue for the search. The similar image search means extracts an image similar to the key reference image from the image data group to be searched. At this time, a feature amount is extracted from each image data included in the reference image and the image data group, and image data similar to the reference image is extracted by comparing the feature amounts.
[0006]
For example, "Iwanami Lecture on Multimedia Informatics 8 Information Structuring and Retrieval" (by Shojiro Nishio et al., Iwanami Shoten) describes a similar image retrieval method in the trademark design database system TRADEMARK. In this system, each of a reference image and a search target image is divided into a plurality of mesh regions, and feature amounts of four types of images are extracted using image information of each mesh and a relationship between image information between the meshes. . The feature amount is a density distribution (64 dimensions), a run length distribution (256 dimensions), a local correlation (24 dimensions), and a local contrast (24 dimensions), all of which are represented as multidimensional vectors. The reference image and the search target image are mapped as one point in a multidimensional space in which these four types of feature amounts are combined, and the Euclidean distance in the multidimensional space is used to determine the similarity between the images. That is, the smaller the Euclidean distance between the point indicating the feature amount of the reference image and the point indicating the feature amount of the search target image, it is determined that the images are more similar.
[0007]
Also, Japanese Patent Application Laid-Open No. 2000-48181 discloses a method of weighting each of the feature amounts of an image and comparing similarities between images based on the weighted feature amounts.
[0008]
In this method, a plurality of feature amounts are extracted from each of a reference image and image data to be searched. Here, the characteristic amount is exemplified by the average value C0 of the saturation, the peak value P0 of the spectrum image, and the strength E0 of the edge. Next, weighting applied to each feature value is determined based on the feature value extracted from the reference image. For example, the weight Wc of the feature amount related to the color and the weight Wt related to the texture are determined. Here, the larger the average value C0 of the saturation of the reference image is, the larger the color weight Wc is, and the larger the peak value P0 and the edge strength E0 of the spectrum image of the reference image are, the larger the texture weight Wt is set. The similarity between the reference image and each search target image data is obtained based on the weighted color feature Rc represented by C0 × Wc and the weighted texture feature Rt represented by (P0 + E0) × Wt. Can be As described above, by using the weighted feature amount calculated according to the features of the reference image, the periodicity of the texture and the strength of the edge are weak, and the color is emphasized in the reference image with high saturation. In a reference image having a low degree and a strong periodicity or edge of the texture, the image can be searched with emphasis on the texture.
[0009]
[Problems to be solved by the invention]
However, in the above-described conventional technology, there is a problem that a search result that matches human similarity cannot be obtained. For example, as shown in FIG. 1, the reference image 101 is an image in which each part is colored, and the images 102 to 104 to be searched are each a diagram in which only the outline is drawn by a black line, and the brightness is equal to the reference image 101. And a sketch image drawn in a sketch style with the same color as the single-color image and the reference image 101. At this time, when brightness, saturation, hue, and edge strength are selected as the feature amounts of the image, the brightness, saturation, and hue of the image 102 are different from those of the reference image 101, and the saturation and hue of the image 103 are different. In contrast, since the image 104 has a different edge strength, it is determined that the reference image 101 and the images 102 to 104 are not similar.
[0010]
Therefore, even when the feature amount is weighted, both the color weight Wc and the texture weight Wt are set to large values because the saturation and the edge strength of the reference image 101 have large values. As a result, since the image 102 and the image 103 have different color characteristics, and the image 104 has different texture characteristics, it is determined that none of the images is similar to the reference image.
[0011]
This is because it has not been appropriate to select feature amounts of lightness, saturation, hue, and edge strength for the reference image 101. As described above, it is difficult to universally select a feature amount indicating the feature of the reference image, and a means for selecting an appropriate feature amount for each search target image and extracting a similar image is required.
[0012]
The present invention has been made in consideration of the above-described problems of the related art, and has an image search apparatus capable of comparing a reference image and a search target image, selecting an appropriate feature amount, and obtaining a search result that matches a human similarity. A search method and an image search program are provided.
[0013]
[Means for Solving the Problems]
The present invention for solving the above-mentioned problem is an image search device that finds an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images, wherein the reference image and the search A feature amount extracting unit that extracts a plurality of visual feature amounts of the image from each of the search target images included in the target image group; and a predetermined condition for the feature amount of the reference image for each of the search target images. A feature amount selecting unit that selects a feature amount of the search target image that is close enough to satisfy, and extracts an image similar to the reference image from the search target image group based only on the selected feature amount. And a similar image extracting unit.
[0014]
Further, in the above image search device, the similar image extracting means may further include, in addition to the selected feature amount, the number of feature amounts selected for each of the search target images and the number of feature amounts extracted from the search target image. It is preferable to extract an image similar to the reference image from the search target image group based on a difference from the total number of feature amounts.
[0015]
Another embodiment of the present invention for solving the above problem is an image search device that finds an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images, From each of the reference images and the search target images included in the search target image group, a feature amount extracting unit that extracts a plurality of visual feature amounts of the image, and for each of the search target images, A feature amount correction unit that weights the feature amount based on a difference between the feature amounts extracted from the reference image, and an image similar to the reference image from the search target image group based on the weighted feature amount. And a similar image extracting means for extracting.
[0016]
Further, in the image search device, the similar image extracting unit may further include, in addition to the weighted feature amounts, a similarity extraction unit based on a difference between the feature amounts before and after the feature amounts are weighted. It is preferable to extract an image similar to the reference image from the inside.
[0017]
In order to solve the above problem, the present invention is an image search method for finding an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images, wherein the reference image and the search target A feature amount extracting step of extracting a plurality of visual feature amounts of the image from each of the search object images included in the image group, and a predetermined condition for the feature amount of the reference image for each of the search object images. A feature amount selecting step of selecting a feature amount of the search target image that is as close as possible, and similarity extracting an image similar to the reference image from the search target image group based on only the selected feature amount. And an image extracting step.
[0018]
Further, in the image search method, the similar image extraction step further includes, in addition to the selected feature amount, the number of feature amounts selected for each of the search target images and the number of feature amounts extracted from the search target image. It is preferable to extract an image similar to the reference image from the search target image group based on a difference from the total number of feature amounts.
[0019]
Another embodiment of the present invention for solving the above problem is an image search method for finding an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images, From each of the reference images and the search target images included in the search target image group, a feature amount extraction step of extracting a plurality of visual feature amounts of the image, and for each of the search target images, A feature amount correction step of weighting the feature amount based on a difference between the feature amounts extracted from the reference image, and an image similar to the reference image from the search target image group based on the weighted feature amount. And extracting a similar image.
[0020]
Further, in the image search method, the similar image extraction step may further include, in addition to the weighted feature amount, further determining the search target image group based on a difference between the feature amounts before and after the feature amount is weighted. It is preferable to extract an image similar to the reference image from the inside.
[0021]
The present invention for solving the above-mentioned problems is an image search program for finding an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images. And from each of the search target images included in the search target image group, a feature amount extraction step of extracting a plurality of visual feature amounts of the image, and for each of the search target images, a feature amount of the reference image A feature amount selecting step of selecting a feature amount of the search target image that is only approximate to a predetermined condition, and an image similar to the reference image from the search target image group based on only the selected feature amount. And a similar image extracting step of extracting the same.
[0022]
Further, in the image search program, the similar image extraction step further includes, in addition to the selected feature amount, the number of feature amounts selected for each of the search target images and the number of feature amounts extracted from the search target image. It is preferable to extract an image similar to the reference image from the search target image group based on a difference from the total number of feature amounts.
[0023]
Another embodiment of the present invention for solving the above-described problem is an image search program for finding an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images, A computer extracting, from each of the reference images and the search target images included in the search target image group, a feature amount extracting step of extracting a plurality of visual feature amounts of the image; A feature amount correcting step of weighting the feature amount based on a difference between the feature amount extracted from the target image and the reference image; and a similarity to the reference image from the search target image group based on the weighted feature amount. And a similar image extracting step of extracting an image to be processed.
[0024]
Further, in the image search program, the similar image extraction step may further include, in addition to the weighted feature amount, further, based on a difference between the feature amounts before and after the feature amount is weighted, the search target image group. It is preferable to extract an image similar to the reference image from the inside.
[0025]
BEST MODE FOR CARRYING OUT THE INVENTION
An embodiment of the present invention will be described with reference to the drawings.
[0026]
As shown in FIG. 2, the image search device according to the embodiment of the present invention basically includes a control unit 200, a standard storage unit 202, an output unit 204, a command input unit 206, an image input unit 208, a database unit 210, and a bus 212. Is composed of The control unit 200, the standard storage unit 202, the output unit 204, the command input unit 206, the image input unit 208, and the database unit 210 are connected via a bus 212 so that information (data) can be transmitted.
[0027]
The image search device according to the present embodiment may further include an interface unit 214 connected to a bus 212 as shown in FIG. The image search device may be connected to an external network 216 via the interface unit 214 so as to be able to transmit information, and may be capable of receiving image data via the network 216.
[0028]
The control unit 200 executes the image search program stored in the standard storage unit 202, and executes an image similar to the reference image from the image data group stored in the database unit 210 or the image data group received via the network. Search for data. The searched image data is output to the output unit 204 or the standard storage unit 202. The control unit 200 can be a CPU used for a general computer.
[0029]
The standard storage unit 202 stores and holds an image search program executed by the control unit 200, parameters used for search processing, image data as search results, and the like. The standard storage unit 202 can be appropriately referred to from the control unit 200 via the bus 212. As the standard storage unit 202, a general semiconductor memory, a hard disk device, a magneto-optical disk device, or the like can be appropriately selected and used.
[0030]
The output unit 204 outputs the search result output from the control unit 200 as information that can be confirmed by the user. It is also used as a user interface when a user inputs a command or parameter when performing an image search. As the output unit 204, a display device, a printer, or the like can be appropriately selected and used.
[0031]
The command input unit 206 is used by a user to input control commands and parameters for executing the image search program. For example, it is used when a control command for instructing the control unit 200 to start execution of an image search program stored in the standard storage unit 202 is input. As the command input unit 206, a keyboard, a mouse, a touch panel, or the like can be appropriately selected and used.
[0032]
The image input unit 208 is used when a reference image serving as a search key and image data to be searched are input. The image data input from the image input unit 208 is stored and held in the database unit 210 or the standard storage unit 202. As the image input unit 208, a general image input device such as a scanner, a digital camera, or a digital video can be appropriately selected and used.
[0033]
The database unit 210 stores and holds image data captured from the image input unit 208 and the network 216. The image data can be stored and held in various image formats such as a bitmap format, a jpeg format, and a tiff format. The image data held in the database unit 210 can be appropriately referred to from the control unit 200 via the bus 212. Since the database unit 210 needs to store and hold image data having a large data amount, it is preferable to appropriately select and use a large-capacity storage device such as a hard disk device or a magneto-optical disk device.
[0034]
<First image search method>
Hereinafter, a first image search method for searching the image data group to be searched for image data similar to the reference image using the image search apparatus will be described in detail with reference to the drawings.
[0035]
FIG. 3 is a block diagram conceptually showing an image search method in the present image search method. Image data of a reference image serving as a search key is input to a feature amount extracting unit 301, and at least one feature amount 303 is extracted. Similarly, each piece of image data to be searched is input to the feature amount extracting means 302, and a feature amount 304 of the same type as the reference image data is extracted. The feature amounts 303 and 304 of the reference image data and the search target image data are input to the feature amount selection unit 305, and the same type of feature amounts are compared with each other, and only the feature amount with high similarity is selected. A feature amount having a high similarity is selected and output as selected feature amounts 306 and 307 for the reference image data and the search target image data, respectively. The selected feature amounts 306 and 307 are input to the similar image extracting unit 308, and the similarity between the reference image and the search target image is obtained based on only the selected feature amounts 306 and 307. By sequentially performing the above processing on all image data to be searched, a search target image having a higher similarity is extracted as being similar to the reference image.
[0036]
The above processing is specifically performed by executing an image search program in the image search device. FIG. 4 shows a flowchart of a process in the first image search method. The processing of the first image search method is stored and held in the standard storage unit 202 as an image search program, and is appropriately read and executed by the control unit 200.
[0037]
The user inputs a control command for starting an image search from the command input unit 206. The control unit 200 receives the control command and executes the image search program stored in the standard storage unit 202. Upon execution of the pixel search program, the process immediately proceeds to step S401.
[0038]
In step S401, image data of a reference image serving as a search key is read from the image input unit 208. Alternatively, the reference image data can be read from outside the image search device via the network 216 using the interface unit 214. Further, a group of image data stored and held in advance in the database unit 210 is displayed as an image on the output unit 204, and one of them is appropriately selected by the user, and designated by using the command input unit 206. A reference image can also be determined. The reference image data is stored and held in the standard storage unit 202 or the database unit 210, read out by the control unit 200 as appropriate, and provided for processing.
[0039]
In step S402, at least one feature amount is extracted from the reference image. As shown in FIG. 5, the reference image is divided into K in the horizontal direction and L in the vertical direction, and is divided into K × L = N small areas. A feature amount is extracted for each of these small areas. In the first image search method, as an example, it is assumed that feature amounts relating to brightness, color, and edge are extracted.
[0040]
It is assumed that the reference image is digital image data composed of pixels expressed in the CIELAB color space. The image data may be expressed in another color space, but is preferably expressed in a uniform color space such as the CIELAB color space in consideration of consistency with human visual characteristics.
[0041]
The characteristic values relating to brightness and color are obtained by the following procedure. Assuming that the pixel value at the position (x, y) on the image is p (x, y), the image is expressed in the CIELAB color space, so that it can be expressed as Expression (1).
(Equation 1)

Further, the i-th (i = 0, 1,..., N−1) small area of the small areas is represented by S _i Then, the small area S _i Lightness L at _i And average color C _i Are represented by Expression (2) and Expression (3), respectively. Here, P in equations (2) and (3) _i Is a small area S _i Is the number of pixels within.
(Equation 2)

All small areas S _i About average lightness L _i And average color C _i , And a feature vector having these as elements is obtained by Expressions (4) and (5). _key And color feature C _key And Here, the vector notation in Expression (5) means a vector in which each element is listed as in Expression (6).
[Equation 3]

On the other hand, the feature value relating to the edge of the reference image is obtained by the following procedure. In the present image detection method, an example of extraction of a feature amount related to an edge using horizontal and vertical edge detection by a Sobel operator will be described.
[0042]
The intensity Ex (x, y) of the horizontal edge and the intensity Ey (x, y) of the vertical edge at the position (x, y) on the image are represented by Expressions (7) and (8), respectively. You. Here, {A} in Equations (7) and (8) means the norm of A.
(Equation 4)

Here, p (x, y) is used as an image edge also at a color discontinuity point. However, if it is desired to obtain the edge intensity only from the brightness of the image without considering the color, p (x, y) L instead of ^* (X, y) may be used. Small area S _i The average intensity E of the horizontal edges at _xi And the average intensity E of the vertical edges _yi Is represented by Expression (9) and Expression (10), respectively.
(Equation 5)

Furthermore, the average intensities E of the horizontal and vertical edges obtained by Equations (9) and (10) _xi , E _yi , The small area S _i Edge strength E _i Is defined by Expression (11).
(Equation 6)

All small areas S _i Of the edge E against _i Are obtained, and a feature vector having these as elements is obtained by Expression (12). _key And Here, the expression of Expression (12) means a vector in which each element is listed as in Expression (6).
(Equation 7)

As described above, the feature amounts relating to the brightness, the color, and the edge from the reference image data are respectively converted into the feature vectors L _key , C _key And E _key Is extracted as
[0043]
In the present image search method, an image is _i Then, the feature amounts relating to brightness, color and edge are extracted, but the image search method of the present invention is not limited to this. For example, if the image is _i , Without dividing the entire image into one region. _key , C _key And E _key May be extracted.
[0044]
In steps S403 to S404, a feature amount is extracted from the search target image data. The search target image data is stored and held in a predetermined order in the database unit 210, and the pointer (address) indicating any one of the search target image data is initially set so as to indicate the first image data. It is assumed that In step S403, the control unit 200 reads out the image data indicated by the pointer from the database unit 210.
[0045]
In step S404, feature values relating to brightness, color, and edge are extracted from the read image data. The extraction of each feature amount is performed using Expressions (1) to (12) in the same manner as that for the above-described reference image. _subj , C _subj And E _subj Is extracted as Here, j = 0, 1,... Are identification numbers uniquely assigned to the respective image data.
[0046]
In step S405, the feature amount L from the reference image _key , C _key And E _key And the feature amount L from the search target image data _subj , C _subj And E _subj And selects a feature value used for calculating the similarity.
[0047]
The comparison between the feature vector indicating each feature amount of the reference image and the feature vector indicating each feature amount of the search target image data is performed using the correlation coefficient R shown in Expressions (13) to (15). _L , R _c And R _E Is calculated. Here, AB means the inner product of the vector A and the vector B.
(Equation 8)

The selection of the feature amount is based on the correlation coefficient R _L , R _C And R _E By selecting a feature amount exceeding a predetermined threshold value TR.
[0048]
Hereinafter, as an example, a description will be given of selection of a feature amount when the image 101 in FIG. 1 is a reference image and the image 103 is image data to be searched. The image 101 is an image in which stars are colored yellow, leaves are green, and trunks are brown. On the other hand, the image 103 is a single-color image in which a single color having the same brightness as that of the image 101 is applied. In both images, since the brightness of each part is equal, the lightness feature amount L _key And L _subj Are equal. On the other hand, the color feature amount C _key And C _subj Are significantly different. Edge feature value E _key And E _subj Are not exactly equal due to the different colors, but have the same direction edge at the same position on the image, so E _key And E _subj Has a high correlation. Therefore, the correlation coefficient for each feature value is, for example, R _L = 1.0, R _C = 0.1 and R _E = 0.8. Assuming that the threshold value TR for selecting a feature value is 0.5, R _L And R _E Is selected.
[0049]
The method of selecting a feature value is not limited to the method using the above-described correlation coefficient, but may be any method capable of comparing the similarity between two corresponding feature values. For example, the selection may be made using the Euclidean distance between two feature vectors instead of the correlation coefficient. In this case, since the Euclidean distance between the feature vectors indicates dissimilarity between the target vectors, a feature amount having a Euclidean distance smaller than a predetermined threshold TR is selected.
[0050]
Furthermore, by selecting a feature by the user himself, it is possible to construct a system that reflects the intention of the image search that the user focuses on. In this case, a user interface is displayed on the output unit 204, and a feature amount selected by the user is designated and selected from the command input unit 206.
[0051]
In step S406, the similarity is calculated. The similarity refers to an evaluation value indicating the similarity between the reference image and the search target image data with respect to the selected feature amount. More specifically, for example, when the image 101 in FIG. 1 is set as the reference image and the image 103 is set as the search target image data, and the feature amount relating to the brightness and the edge is selected in step S405, the similarity is calculated using Expression (16). The degree R can be calculated.
(Equation 9)

Equation (16) uses the correlation coefficient between the feature amount selected in the reference image and the search target image data. Generally, when M kinds of feature amounts are selected from an image, the feature amount of the reference image is set to F _{j, key} , The feature amount of the search target image data is set to F _{j, subj} Then, the similarity R based on the correlation coefficient between the two can be expressed by Expression (17). Here, G indicates a set of feature amounts selected in step S405. When the number of feature amounts selected in step S405 is 0, R = 0.
(Equation 10)

The method of calculating the similarity is not limited to the method using the correlation coefficient, but may be any method that can evaluate the similarity between the feature amounts selected for the reference image and the search target image data. For example, a Euclidean distance D between feature amount vectors may be used instead of the correlation coefficient as in Expression (18). In this case, since the Euclidean distance D between the vectors indicates the dissimilarity between the two vectors, the smaller the value of the Euclidean distance D, the higher the similarity.
[Equation 11]

In step S407, the similarity between the reference image and the search target image data is stored and held in the standard storage unit 202 or the database unit 210 in association with the search target image data.
[0052]
In step S408, it is determined whether or not the similarity with the reference image has been calculated for all the image data included in the image data group to be searched. Since the search target image data is stored in the database unit 210 in a predetermined order, it is determined whether the pointer indicating the search target image data processed in S403 to S407 is the last one. Is not the last pointer, the process returns to step S403 with the pointer as the next value. If it is the last pointer, the process moves to step S409.
[0053]
In step S409, when the similarity is calculated using the correlation coefficient, the similarity stored in the standard storage unit 202 or the database unit 210 is rearranged in descending order. When the similarity is calculated as an evaluation value indicating dissimilarity such as the Euclidean distance, the similarities are rearranged in ascending order. As a result, the search target image data having the maximum similarity is extracted as image data of an image similar to the reference image data. When it is desired to extract a plurality of similar images, it is also preferable to extract search target image data having a degree of similarity equal to or greater than a predetermined threshold.
[0054]
In step S410, a search result is displayed using the result of sorting the similarities. As the search result, the search target image data having the maximum similarity can be displayed on the output unit 204. It is also preferable to select a predetermined number of images in descending order of similarity and display the search target image data associated with them in the output unit 204. Further, it is also preferable to set a threshold value in advance and display search target image data having a similarity exceeding the threshold value on the output unit 204. In this case, when there is no search target image data having a similarity exceeding the threshold, a display such as “No similar image” can be displayed.
[0055]
As described above, according to the first image search method, the reference image and the search target image are compared, an appropriate feature amount is selected, and the reference image is matched with the human similarity sense from the search target image data group. Images having the similarity can be retrieved.
[0056]
<Second image search method>
In the first image search method, when only a part of the features of the reference image and the search target image data are different, the similarity similar to the case where the reference image and the search target image data are completely the same is calculated. Sometimes. For example, in FIG. 1, when the image 101 is the reference image and the image 103 is the search target image data, since both images have the same brightness, the feature amount L regarding brightness is used. _key And L _subj Is equal to, and the feature amount C regarding the color _key And C _subj Will be very different. In addition, the feature amount E related to the edge _key And E _subj Indicates a high correlation. At this time, if the threshold value for selecting the feature value is set high, only the feature value related to lightness is selected. When the similarity is calculated using Expression (17), the similarity R becomes 1, which is equal to the case where the reference image and the search target image data are completely the same.
[0057]
In order to avoid such a problem, a method of correcting the similarity according to the selected feature amount is considered. Hereinafter, a second image search method for searching for image data by correcting the similarity will be described in detail with reference to the drawings.
[0058]
FIG. 6 is a block diagram conceptually showing an image search method in the present image search method. Image data of a reference image serving as a search key is input to a feature amount extracting unit 601 and at least one feature amount 603 is extracted. Similarly, each of the image data to be searched is input to the feature amount extracting unit 602, and the same type of feature amount 604 as the reference image data is extracted. The feature amounts 603 and 604 are input to the feature amount selection unit 605, and the same type of feature amounts are compared with each other, and only the feature amount having high similarity is selected. A feature amount having high similarity is selected and output as selected feature amounts 606 and 607 for the reference image data and the search target image data, respectively. The selected feature amounts 606 and 607 are input to the similar image extracting unit 608. On the other hand, the feature amount selection means 605 outputs feature amount selection information 610 as information when the feature amount is selected. The feature amount selection information 610 is input to the similarity correction coefficient calculation unit 611, and the similarity correction coefficient 612 is obtained. The similarity correction coefficient 612 is input to the similar image extraction unit 608, and the similarity between the reference image data and the search target image data is calculated based on the selected feature amounts 606 and 607. 612 to obtain the final similarity. The similar image extracting unit 608 sequentially performs the above-described processing on all image data to be searched, thereby extracting a search target image having a higher similarity as an image similar to the reference image, and outputting 609. And
[0059]
The above processing is specifically performed by executing an image search program in the image search device. FIG. 7 shows a flowchart of a process in the second image search method. The processing of the second image search method is stored and held in the standard storage unit 202 as an image search program, and is appropriately read and executed by the control unit 200.
[0060]
In the flowchart of FIG. 7, the processing in steps S701 to S705 is the same as the processing in steps S401 to S405 in the first image search method, and a description thereof will be omitted.
[0061]
In step S706, a similarity correction coefficient q is calculated based on the selection state of the feature amount in step S705. The similarity correction coefficient q can be calculated using, for example, Expression (19). Here, Dim (A) indicates the number of dimensions of the vector A. The function f (x) is a monotonically increasing function for the argument x, and is, for example, a function as shown in FIG.
(Equation 12)

By using Expression (19), the value of the similarity correction coefficient q increases as the number of types of the feature amounts selected in step S705 increases. For example, when the function f (x) shown in FIG. 8 is used, q = 1.0 when all the feature amounts are selected.
[0062]
If the image 101 in FIG. 1 is the reference image data and the image 103 is the search target image data, and if only the feature amount relating to the brightness is selected in step S705, Dim (L _key ) = N, Dim (C _key ) = 2N and Dim (E _key ) = 2N, the argument of equation (19) is N / (N + 2N + 2N) = 0.2. Using FIG. 8, the similarity correction coefficient q is 0.76.
[0063]
In step S707, the similarity R corrected by the similarity correction coefficient q is calculated. In the method of calculating the similarity, the expression (17) for calculating the correlation coefficient of the feature quantity selected for the reference image data and the search target image data was corrected using the similarity correction coefficient q. Equation (20) can be used.
(Equation 13)

In the above example, since the similarity correction coefficient q is 0.76, the similarity R is 0.76 by using Expression (20). As described above, the value of the similarity R is corrected according to the number of types of the selected feature amount, and the similarity that appropriately reflects the difference between the reference image data and the search target image data can be calculated.
[0064]
When the Euclidean distance D between two feature vectors indicating the feature amounts is used instead of the correlation coefficient, the equation (21) in which the Euclidean distance D is corrected by the similarity correction coefficient q is used. It is suitable.
[Equation 14]

The processing in steps S708 to S711 is the same as the processing in steps S407 to S410 in the first image search method, and a description thereof will be omitted.
[0065]
As described above, according to the second image search method, it is possible to more appropriately evaluate the similarity between the reference image and the search target image data and search for the image in accordance with the number of types of the selected feature amount. Can be.
[0066]
<Third image search method>
In the first and second image search methods, similarity is calculated by selecting only feature amounts that satisfy predetermined selection conditions. At this time, if the feature amount is near the boundary of the selection condition, the feature amount may or may not be selected due to a subtle difference in the image. As a result, there is a possibility that a problem occurs in which the calculated similarity has a large difference although there is no large difference in the images.
[0067]
In order to avoid such a problem, a method of calculating the similarity by weighting the feature amounts without selecting the feature amounts can be considered. Hereinafter, a third image search method for searching for image data by weighting feature amounts will be described in detail with reference to the drawings.
[0068]
FIG. 9 is a block diagram conceptually showing the image search method. Image data of a reference image serving as a search key is input to a feature amount extracting unit 901 and at least one feature amount 903 is extracted. Similarly, each of the image data to be searched is input to the feature amount extraction unit 902, and the same type of feature amount 904 as the reference image data is extracted. The feature values 903 and 904 are input to a feature value weight setting unit 905, where the feature values are compared, and a feature value weight 906 is calculated based on the similarity. The feature amount weighting 906 is input to the similar image extracting unit 907 together with the feature amounts 903 and 904, and the similarity between the reference image data and the search target image data is calculated based on these. By sequentially performing the above processing on image data to be searched, an image having a high degree of similarity is searched for as an image similar to the reference image.
[0069]
The above processing is specifically performed by executing an image search program in the image search device. FIG. 10 shows a flowchart of a process in the third image search method. The process of the third image search method is stored and held in the standard storage unit 202 as an image search program, and is appropriately read and executed by the control unit 200.
[0070]
The processing in steps S1001 to S1004 is the same as the processing in steps S401 to S404 in the above-described first image search method, and a description thereof will be omitted.
[0071]
In step S1005, feature amounts are weighted for the feature amounts extracted in step S1004. For each type of feature, feature weight w _j (J = 0, 1,..., M-1) can be obtained using Expression (22). The function g (x) in Expression (22) is a monotonically increasing function with respect to the argument x, and can be, for example, a function as shown in FIG.
[Equation 15]

By using Equation (19) as a function as shown in FIG. 11, when the correlation coefficient between the respective feature amounts is less than 0.5 for the feature amounts extracted from the reference image data and the search target image data, Is the feature weight w _j Is 0, and when the correlation coefficient is 0.5 or more, the feature amount weighting w _j Increases with the correlation coefficient.
[0072]
In step S1005, further, the feature amount weighting w _j Is used to weight each feature amount as in Expressions (23) and (24).
(Equation 16)

In step S1006, the similarity R is calculated using Expression (25) using the feature amounts weighted by Expressions (23) and (24).
[Equation 17]

The processing in steps S1007 to S1010 is the same as the processing in steps S407 to S410 in the first image search method, and a description thereof will be omitted.
[0073]
As described above, according to the third image search method, all the extracted feature amounts are appropriately weighted without selecting feature amounts using selection conditions such as thresholds, so that Has no large difference, and even when the feature amount is near the boundary of the selection condition, it is possible to avoid the problem that the calculated similarity causes a large difference. As a result, an image can be searched by appropriately evaluating the similarity between the reference image data and the search target image data.
[0074]
<Fourth image search method>
In the third image search method, the feature amount weighting w _j It is also preferable to correct the similarity based on The feature weight w _j The fourth image search method for correcting the similarity according to the following will be described in detail with reference to the drawings.
[0075]
FIG. 12 is a block diagram conceptually showing the image search method. A reference image serving as a search key is input to a feature amount extraction unit 1201 and at least one feature amount 1203 is extracted. Similarly, each piece of image data to be searched is input to the feature amount extraction unit 1202, and the same type of feature amount 1204 as the reference image data is extracted. The feature values 1203 and 1204 are input to the feature value weight setting unit 1205, where the feature values are compared, and the feature value weight 1206 is calculated based on the similarity. The feature weight setting unit 1205 further outputs feature weight information 1209 used when calculating the similarity correction coefficient 1211. The feature amount weighting information 1209 is input to the similarity correction coefficient calculation unit 1210, and the similarity correction coefficient 1211 is obtained. The feature amount weight 1206 and the similarity correction coefficient 1211 are input to the similar image extracting unit 1207 together with the extracted feature amounts 1203 and 1204, and based on these, the similarity between the reference image data and the search target image data is calculated. You. By sequentially performing the above processing on image data to be searched, an image having a high similarity is searched for as similar to the reference image.
[0076]
The above processing is specifically performed by executing an image search program in the image search device. FIG. 13 shows a flowchart of a process in the fourth image search method. The processing of the fourth image search method is stored and held in the standard storage unit 202 as an image search program, and is appropriately read and executed by the control unit 200.
[0077]
The processing in steps S1301 to S1305 is the same as the processing in steps S1001 to S1005 in the third image search method, and a description thereof will be omitted.
[0078]
In step S1306, the weight w for each feature amount calculated in step S1305 is set. _j Is used to calculate a similarity correction coefficient q. The similarity correction coefficient q can be obtained using, for example, Expression (26). Here, Dim (A) indicates the number of dimensions of the vector A. The function f (x) is a monotonically increasing function for the argument x, and is, for example, a function as shown in FIG.
(Equation 18)

By using the equation (26), the weight w _j Becomes larger, the similarity correction coefficient q can also be increased.
[0079]
In step S1307, the similarity R corrected by the similarity correction coefficient q is calculated. As a method of calculating the similarity R, a mathematical expression (27) obtained by modifying the mathematical expression (25) for obtaining the similarity R from the weighted feature amount using the similarity modification coefficient q can be used.
[Equation 19]

The processing in steps S1308 to S1311 is the same as the processing in steps S1007 to S1010 in the third image search method, and a description thereof will be omitted.
[0080]
As described above, according to the fourth image search method, the weight w _j By correcting the similarity R in accordance with the value of, it is possible to more appropriately evaluate the similarity between the reference image and the search target image data and search for the image.
[0081]
【The invention's effect】
According to the present invention, between images that differ only in saturation and hue, such as a black-and-white image and a color image, or between images having a common edge strength characteristic, such as a line diagram and a colored diagram. , It is possible to search for similar images, which were difficult to search using the conventional image search technology. That is, it is possible to appropriately select a similar feature between the reference image and the search target image data, and to search for a similar image that matches the similarity of a human.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an example of an image having a part of similar features.
FIG. 2 is a configuration diagram of an image search device according to an embodiment of the present invention.
FIG. 3 is a block diagram conceptually showing processing in the first image search method of the present invention.
FIG. 4 is a diagram showing a flowchart in the first image search method of the present invention.
FIG. 5 is a diagram illustrating an image dividing method for extracting a feature amount of an image.
FIG. 6 is a block diagram conceptually showing processing in a second image search method of the present invention.
FIG. 7 is a diagram showing a flowchart in a second image search method of the present invention.
FIG. 8 is a diagram showing an example of a function f (x) in the image search method of the present invention.
FIG. 9 is a block diagram conceptually showing processing in a third image search method of the present invention.
FIG. 10 is a diagram showing a flowchart in a third image search method of the present invention.
FIG. 11 is a diagram showing an example of a function g (x) in the image search method of the present invention.
FIG. 12 is a block diagram conceptually showing processing in a fourth image search method of the present invention.
FIG. 13 is a diagram showing a flowchart in a fourth image search method of the present invention.
[Explanation of symbols]
101 Color-coded image, 102 diagram-based image, 103 monochromatic image with brightness equal to image 101, 104 sketch-like image drawn in the same color as image 101, 200 control unit, 202 standard storage unit, 204 output unit, 206 command input unit, 208 image input unit, 210 database unit, 212 bus, 214 interface unit, 216 network, 301, 302, 601, 602, 901, 902, 1201, 1202 feature amount extraction means, 303, 304, 603 604, 903, 904, 1203, 1204 feature quantity, 305, 605 feature quantity selection means, 306, 307, 606, 607 selected feature quantity, 308, 608, 907, 1207 similar image extraction means, 309, 609, 908 , 1208 Similarity, 610 Feature selection information, 61 1,1210 similarity correction coefficient calculation means, 612, 1211 similarity correction coefficient, 905, 1205 feature weight setting means, 906, 1206 feature weight, 1209 feature weight information.

Claims (12)

An image search device that finds an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images,
From each of the reference images and the search target images included in the search target image group, a feature amount extraction unit that extracts a plurality of visual feature amounts,
For each of the search target images, a feature amount selection unit that selects a feature amount of the search target image that is similar to the feature amount of the reference image by only a predetermined condition,
A similar image extracting unit configured to extract an image similar to the reference image from the search target image group based on only the selected feature amount;
An image search device comprising: The image search device according to claim 1,
The similar image extracting means, together with the selected feature amount,
An image similar to the reference image is extracted from the search target image group based on a difference between the number of feature amounts selected for each of the search target images and the total number of feature amounts extracted from the search target image. An image search device characterized by the above-mentioned. An image search device that finds an image similar to a reference image serving as a search key from a search target image group including a plurality of search target images,
From each of the reference images and the search target images included in the search target image group, a feature amount extraction unit that extracts a plurality of visual feature amounts,
A feature amount correcting unit that weights the feature amount based on a difference between feature amounts extracted from the search target image and the reference image for each of the search target images;
A similar image extracting unit configured to extract an image similar to the reference image from the search target image group based on the weighted feature amount;
An image search device comprising: The image search device according to claim 3,
The similar image extracting means, together with the weighted feature amount,
An image search device, wherein an image similar to the reference image is extracted from the search target image group based on a difference between the feature amounts before and after the feature amount is weighted. An image search method for finding an image similar to a reference image serving as a search key from a group of search target images including a plurality of search target images,
A feature amount extraction step of extracting a plurality of visual feature amounts from each of the reference images and the search target images included in the search target image group;
A feature amount selecting step of selecting, for each of the search target images, a feature amount of the search target image that is similar to the feature amount of the reference image by only a predetermined condition;
A similar image extracting step of extracting an image similar to the reference image from the search target image group based on only the selected feature amount;
An image search method comprising: The image search method according to claim 5,
The similar image extraction step, together with the selected feature amount, further,
An image similar to the reference image is extracted from the search target image group based on a difference between the number of feature amounts selected for each of the search target images and the total number of feature amounts extracted from the search target image. An image search method, characterized in that: An image search method for finding an image similar to a reference image serving as a search key from a group of search target images including a plurality of search target images,
A feature amount extraction step of extracting a plurality of visual feature amounts from each of the reference images and the search target images included in the search target image group;
A feature amount correcting step of weighting the feature amount based on a difference between feature amounts extracted from the search target image and the reference image for each of the search target images;
A similar image extraction step of extracting an image similar to the reference image from the search target image group based on the weighted feature amount;
An image search method comprising: The image search method according to claim 7,
The similar image extraction step, together with the weighted feature amount, further,
An image search method, wherein an image similar to the reference image is extracted from the search target image group based on a difference between the feature amounts before and after the feature amount is weighted. An image search program for finding an image similar to a reference image serving as a search key from a group of search target images including a plurality of search target images,
On the computer,
From each of the reference image and the search target image included in the search target image group, a feature amount extraction step of extracting a plurality of visual feature amounts of the image,
A feature amount selecting step of selecting, for each of the search target images, a feature amount of the search target image that is similar to the feature amount of the reference image by only a predetermined condition;
A similar image extracting step of extracting an image similar to the reference image from the search target image group based on only the selected feature amount;
An image search program characterized by executing the following. The image search program according to claim 9,
The similar image extraction step, together with the selected feature amount, further,
An image similar to the reference image is extracted from the search target image group based on a difference between the number of feature amounts selected for each of the search target images and the total number of feature amounts extracted from the search target image. An image search program characterized by the following. An image search program for finding an image similar to a reference image serving as a search key from a group of search target images including a plurality of search target images,
On the computer,
From each of the reference image and the search target image included in the search target image group, a feature amount extraction step of extracting a plurality of visual feature amounts of the image,
A feature amount correcting step of weighting the feature amount based on a difference between feature amounts extracted from the search target image and the reference image for each of the search target images;
A similar image extraction step of extracting an image similar to the reference image from the search target image group based on the weighted feature amount;
An image search program characterized by executing the following. The image search program according to claim 11,
The similar image extraction step, together with the weighted feature amount, further,
An image search program for extracting an image similar to the reference image from the search target image group based on a difference between the feature amounts before and after the feature amount is weighted.

Images