JP3720573B2 - Image search apparatus and method - Google Patents

Description

【００３９】
以上のように、ラベル間のパターンマッチングの際に隣接するセル同士ではペナルティ（距離）を小さくし、遠いものには大きなペナルティを与えるために図１１に示すようなラベル間でのペナルティマトリックスを導入する。ステップＳ２４ではこのペナルティマトリックスを考慮し、ラベル列同士を比較するが、その際に、オートマトン等のラベルシーケンスを曖昧に比較できるマッチングを行うようにしてもよい。このような曖昧化の手法を用いることにより、余分なラベルの付加、ラベルの欠落や同じラベルの繰り返しに対しては低いペナルティが与えられとともに、ラベル間のペナルティには図１２のカラーラベル間のペナルティマトリックスを用いてラベル列同士の距離計算を行うことで、曖昧なパターンマッチングが行えるようになる。なお、オートマトンとしては、「特開平８−２４１３３５のファジー非決定性有限オートマトンを使用した曖昧な文字列検索方法およびシステム」に記載されている「ファジー非決定性有限オートマトン」を適用することができる。このオートマトンでは、各シンボル間の距離（ペナルティー）が多値でして出来、なお、比較するラベル位置を前後曖昧に移動することが出来、トータルの距離が最小（類似度が最大）となるようなラベル列の比較を実現するための手法として、上述のオートマトンの他に、音声認識等において用いられているＤＰマッチングがあり、この手法も本実施形態に好適に適用できるものである。
【００４０】
更に、上述の曖昧なパターンマッチングに加えて、図７の（ａ）〜（ｃ）のブロック順序の規則を併用することにより、比較する画像のアングルの微妙な違いやずれの影響を少なく、上下左右のどちらのずれに対しても影響の少ないカラーラベル列同士の比較を行うことが可能となる。すなわち、ＤＰマッチングやファジー非決定性オートマトンは、ラベル列の前後の曖昧さを許容するマッチングであり、画像の位置ずれの影響を吸収する性質を有する。また、アングルの違い等により物体の位置が変わり、ブロックによって切りとられる物体の位置が変わることにより、ブロックの色合いも微妙に異なることが予想されるが、この違いは上述のペナルティーマトリクスにより吸収されることになる。このように、ＤＰマッチング或いはファジーオートマトンによる曖昧さを許容するマッチングと、ペナルティーマトリクスによる特徴量の曖昧さの許容との相乗効果によって、上下左右のずれに対して影響の少ないマッチングを可能としている。更に、図７（ａ）〜（ｃ）のような斜めスキャンにより、物体の位置の変化によるラベル位置の変化が低減されるので、より効果的にマッチング時の物体のずれの影響を低減できる。
【００４１】
次に、ステップＳ２５において、画像管理ＤＢ１８を参照して、画像ＩＤ群の各画像ＩＤについてフルパスのファイル名を取得し、これをユーザに提示する。
【００４２】
以上のような処理により、画像のアングルが変ったり、物体の位置が変ったり、あるいは撮影条件が変わったりすることによって生じる、色のある程度の違い等を吸収するなど、ロバストな類似画像検索を高速に行うことが可能となる。
【００４３】
なお、ブロック化できない１つの画像に対して１つのパラメータを加味した類似検索の場合には、本発明で得られる類似度（ペナルティの総和を用いて作る）を１つの新たなる特徴量として、統計的な距離尺度に基づく検索を行うことも可能である。また、上記実施形態では、類似度が所定値を越える類似画像を検索結果として得るが、類似度の高い画像から順に前もって指定された個数の画像を検索結果として出力するようにしてもよいことはいうまでもない。
【００４４】
以上説明したように、第１の実施形態によれば、特徴量群（特徴量空間を分割して得られる特徴量のグループ）を１つのシンボルで表現し（すなわちラベル化し）、ラベル同士の類似度に基づく距離をペナルティーマトリクスによって与える。このため、２つの画像のブロック間の距離の計算量を大幅に減少させることができるとともに、類似した特徴量が同じラベルで表されることになるので、類似画像の検索を良好に行うことができる。
【００４５】
また、（１）ペナルティマトリクスによるラベル同士の距離概念を導入し、（２）ＤＰマッチングやファジー非決定性オートマトン等の、比較するラベル位置を前後曖昧に移動させることが出来、トータルの距離が最小（類似度が最大）となるようなラベル列の比較を実現する手法を導入する、ことにより、画像のアングルが多少変わっても検索することが可能となり、雰囲気の似ている画像を検索できるようになる。
【００４６】
更に上記実施形態によれば、インデックスデータベース（ラベル成分インデックス（図１０））を用いたことにより、画像検索が更に高速化する。
【００４７】
［第２の実施形態］
上記第１の実施形態では、特徴量を表すラベルを１次元に並べて、すなわちラベル列として類似画像検索を行った。第２の実施形態では、特徴量を表すラベルの２次元配置を考慮して、すなわちラベル行列を用いて類似画像検索を行う。なお、第２の実施形態による画像検索装置の制御構成は、第１の実施形態（図１）と同様であるのでここでは説明を省略する。
【００４８】
図１５は本実施形態の画像検索装置の機能構成を示すブロック図である。同図において、ユーザインターフェース部１１、画像入力部１２、画像メモリ１３、画像特徴量抽出部１４、画像蓄積部１７は第１の実施形態（図２）と同様の構成である。
【００４９】
１５ａは特徴量ラベル行列化部であり、画像特徴量抽出部１４によって得られた特徴量に基づいてラベル行列を生成する。１６ａはパターンマッチング部であり、指定された画像のラベル行列と、画像蓄積部１７に蓄積されている画像のラベル行列に対し、後述の２次元ＤＰマッチングを用いて類似度を算出する。１８ａは画像管理データベース（以下、画像管理ＤＢ）であり、図１６で示すデータ形態で画像蓄積部１７に格納された画像データを管理する。また、１９ａはラベル行列インデックスであり、図１０で示したラベル成分インデックスファイルを格納する。
【００５０】
以上のような構成を備えた第２の実施形態の画像検索装置の動作例を以下に説明する。なお、以下の例では色に着目した画像特徴量として、赤（Ｒ）、緑（Ｇ）、青（Ｂ）の三色を採用し、３次元の色空間での処理を用いて説明する。
【００５１】
［画像の登録処理］
図１７は第２の実施形態による画像登録処理の手順を示すフローチャートである。図１７において、ステップＳ１１〜Ｓ１３は第１の実施形態（図４）で説明した処理と同様である。
【００５２】
ステップＳ１１〜Ｓ１３の処理により、各ブロックに対してパラメータラベルが付与されると、ステップＳ１４ａにおいて、各ブロックに付与されたパラメータラベルを所定のブロック順序で並べることにより、パラメータラベル行列（以下、ラベル行列とする）が生成される。図１８はラベル行列を生成する際のブロック順序例を説明する図である。同図の分割画像ブロックの升にある数字に従って上記のパラメータラベルを並べ、ラベル行列を作る。なお、画像管理データベース１８ａやラベル行列インデックス１９ａにラベル行列を格納するに際しては、上述のように２次元的なラベル行列を所定の順序で１次元に並べたものを格納するが、本実施形態ではこのような１次元の形態のものもラベル行列と称することにする。
【００５３】
ここで、図１８の（ａ）では、分割ブロックを左から右へ水平方向へスキャンし、この水平方向のスキャンを上から下へ行う順序となっている。なお、本実施形態に適用可能なスキャンの方法としては、
・水平方向（図１８の（ａ）に示したように、左から右へのスキャンを上から下へ行うという順序の他に、図１８の（ｂ）〜（ｄ）に示すように、左から右へのスキャンを下から上へ行う等、４通りのスキャン方法がある）、
・垂直方向（上から下へのスキャンを左から右へ行う等、４通りのスキャン方法が考えられる）、
・図１８（ｅ）に示すように、偶数ラインと奇数ラインでスキャンを分ける。
【００５４】
なお、第２の実施形態では、図１８の（ａ）に示すスキャン方法を採用するが、上述した他のスキャン方法も適用可能である。
【００５５】
続いてステップＳ１５ａにおいて、以上のようにして得たラベル行列や画像データを画像蓄積部１７、画像管理ＤＢ１８ａ、ラベル行列インデックス１９ａに格納する。すなわち、ステップＳ１１で読み込んだ画像データに対して画像ＩＤを取得し、これらをペアにして画像蓄積部１７に格納する。そして、当該画像ＩＤに対応付けて図１６に示す画像管理ＤＢレコードを生成し、これを画像管理ＤＢ１８ａに登録する。
【００５６】
以上が画像登録時に行われる処理である。
【００５７】
［類似画像検索処理］
次に図１９のフローチャートに従って類似画像検索の処理を説明する。図１９は第２の実施形態による類似画像検索の処理手順を説明するフローチャートである。なお、本実施形態では、予め初期化時において、画像管理ＤＢのラベル系列から、既に登録されている画像のラベル行列群を得て、各ラベル成分をキーとするラベル成分インデックスファイルを生成し、ラベル行列インデックス１９ａに格納しておく。なお、ここでいう初期化時とは、システムの立ち上げ時或いはアプリケーションの起動時のいずれでも良い。また、新規の画像登録があり、これを画像ＤＢに登録した場合にも、このラベル成分インデックスの生成を行う。
【００５８】
本実施形態においても、第１の実施形態で説明したラベル成分インデックス（図１０）を用いる。すなわち、ラベル成分インデックスには、各ラベル成分毎に、そのラベルを内部に持つラベル行列へのアドレス群（列ＩＤ群）が登録される。なお、このラベル成分インデックスファイルは画像の登録及び削除、変更を反映する必要が生じるまで、作成し直す必要はない。もちろん、第１の実施形態で述べたように、毎回登録時にラベル成分インデックスを追記してゆく方法も存在する。
【００５９】
まず、ステップＳ２１において、ユーザーインターフェース部１１から類似検索元画像が指定されると、ステップＳ２２ａにおいて、指定された類似検索元画像の画像ＩＤが取得され、更に画像管理ＤＢ１８から当該元画像のラベル行列（本例ではカラーラベル行列）が取得される。
【００６０】
次にステップＳ２３ａにおいて、ラベル成分インデックスファイルを参照し、類似検索元画像のラベル行列とある程度以上同一のラベルを含むラベル行列群（ラベル系列インデックス中のラベル行列）を取得する。これは登録した画像の全てのラベル列との比較を行うと処理が遅くなるので、予め似ているものに絞った後に、類似検索元画像のラベル行列と一対一で比較するようにし、処理速度を改善するためである。なお、類似したラベル行列とは、類似検索元画像のラベル行列と所定数以上の同一もしくは類似のラベルを含むラベル行列群をいう。ここで、類似のラベルとは図１１で説明したラベル間ペナルティの小さいラベル同士をいう。もちろん、処理が遅くなっても良ければ、登録した画像の全てのラベル行列との比較を行い、精度の高い検索を行ってもよい（この場合、ステップＳ２３ａは省略される）。
【００６１】
次に、ステップＳ２４ａにおいて、ステップＳ２３ａで取得した各ラベル行列と類似検索元画像のラベル行列とを比較し、その類似度を算出する。そして、類似検索元画像のラベル行列に最も近いラベル行列から順にその類似度とともに検索結果として出力する。
【００６２】
ここで、ラベル行列同士の類似比較（類似度の算出）を行う方法について述べる。なお、以下では、ステップＳ２３ａで取得したラベル行列を類似比較先画像と称する。
【００６３】
また、第２の実施形態においてもラベル行列を比較して類似度を求める際には、第１の実施形態と同様にラベル間のペナルティマトリックス（図１１）を用いる。
【００６４】
ラベル間のパターンマッチングの際に隣接するセル同士ではペナルティ（距離）を小さくし、遠いものには大きなペナルティを与えるために図１２に示すようなラベル間でのペナルティマトリックスを導入する。ステップＳ２４ａではこのペナルティマトリックスを考慮し、ラベル行列同士を比較するが、第２の実施形態では、その際に以下に説明する２次元的なＤＰマッチング（以下、２次元ＤＰマッチングという）を使用する。
【００６５】
図２０は本実施形態による類似度算出処理を説明する図である。上述のステップＳ２２ａにおいて取得された類似検索元画像のラベル行列は、そのスキャン方法に従って図２０の（ｂ）のように並べることができる。また、ステップＳ２３ａにおいて抽出されたラベル行列群のうちの一つを類似検索元画像とすると、そのラベル行列は図２０（ａ）のように並べることができる。
【００６６】
まず、類似比較先画像の第１行目のラベル列「ａｂｃ」と、類似検索元画像の第１〜第３行目のラベル列（「１２３」、「４５６」、「７８９」）のそれぞれとの距離をＤＰマッチングによって求め、その距離が最少となるラベル列の類似検索元画像における行番号を類似ライン行列（図２０の（ｃ））の該当する位置に記憶する。また、得られた最小距離が所定の閾値よりも大きい場合には、どの行とも類似していないと判断し、類似ライン行列の該当する位置に「！」を記憶する。ＤＰマッチングの性質により、たとえば画像のアングルが水平方向へ多少変わっていたとしても、上記処理により類似する行（ライン）を検出可能である。以上の処理を、類似比較先画像の全ての行（「ｄｅｆ」「ｇｈｉ」）について行うことで、図２０の（ｃ）のような列方向の類似ライン行列が得られる。
【００６７】
図２０の（ｃ）では、「ａｂｃ」に類似した行が類似検索元画像に存在せず、「ｄｅｆ」に類似した行が類似検索元画像の第１行目、「ｇｈｉ」に類似した行が類似検索元画像の第２行目であったことを示している。以上のようにして得られた類似ライン行列と標準ライン行列（類似検索元画像の行の並びであり、本例では「１２３」となる）との類似度をＤＰマッチングを用いて算出し、これを当該類似検索元画像と当該類似比較先画像との類似度として出力する。なお、ＤＰマッチングでは、周知のように、比較するラベルシーケンスが最も類似距離が小さくなるように、比較するラベルシーケンスを伸縮（比較する相手を次に進めないで我慢する）させて比較するという処理を行う。また、何処まで伸縮（我慢）できるかを制約条件（整合窓の幅）で与えることも可能である。
【００６８】
図２１は第２の実施形態による、２次元ＤＰマッチングを採用した類似度算出の手順を説明するフローチャートである。上記図２０を参照して説明した処理を、図２１のフローチャートを参照して更に説明する。
【００６９】
まず、ステップＳ１０１において、類似比較先画像の行番号を表す変数ｉと、類似検索元画像の行番号を表す変数ｊを１に初期化し、ともに第１行目を示すように設定する。次に、ステップＳ１０２において、類似比較先画像の第ｉ行目のラベル列を取得する。例えば図２０の場合、ｉ＝１であれば「ａｂｃ」が取得される。そして、ステップＳ１０３において、類似検索元画像の第ｊ行目のラベル列を取得する。例えば、図２０において、ｊ＝１であれば、「１２３」が取得される。
【００７０】
次にステップＳ１０４では、上記ステップＳ１０２、Ｓ１０３で得られた２つのラベル列間の距離を、図１２で説明した色セルペナルティーマトリクスを用いて、ＤＰマッチングによって求める。そして、ステップＳ１０５において、ステップＳ１０４で得た距離が、第ｉ行目に関してそれまでに得られた距離の最小値であれば、当該行番号（ｊ）をライン行列要素ＬＩＮＥ［ｉ］に記憶する。
【００７１】
以上のステップＳ１０３からステップＳ１０５の処理を、類似検索元画像の全ての行について行う（ステップＳ１０６、Ｓ１０７）。以上のようにして、類似比較先画像の第ｉ行目のラベル列に対して、類似検索元画像に含まれる行のうち最も距離の近い行の番号がＬＩＮＥ［ｉ］に格納されることになる。
【００７２】
そして、ステップＳ１０８において、上記処理でられたＬＩＮＥ［ｉ］と所定の閾値（Ｔｈｒｅｓｈ）とを比較する。そして、ＬＩＮＥ［ｉ］がＴｈｒｅｓｈ以上であればステップＳ１０９へ進み、いずれの行とも類似していないことを表す「！」をＬＩＮＥ［ｉ］に格納する。
【００７３】
以上説明したステップＳ１０２からＳ１０８の処理を類似比較先画像の全ての行について実行する（ステップＳ１１０、Ｓ１１１）ことにより、ＬＩＮＥ［１］〜ＬＩＮＥ［ｉｍａｘ］が得られるので、これを類似ライン行列ＬＩＮＥ［］として出力する。
【００７４】
次に、ステップＳ１１３において、標準ライン行列「１２…ｉｍａｘ」と類似ライン行列ＬＩＮＥ［］とのＤＰマッチングを行い、両者の距離を算出する。なお、ここで、標準ライン行列とは、１から始まり、列方向に１ずつ増加する行列である。
【００７５】
ここで、標準ライン行列と類似ライン行列間のＤＰマッチングにおいて用いられるペナルティーについて説明する。列方向の類似ライン行列と標準ライン行列とのＤＰマッチングのペナルティーの設定としては２つの方法が考えられる。すなわち、動的なペナルティーと固定的なペナルティーの２つである。
【００７６】
動的なペナルティーとは、動的にライン番号間のペナルティーを設定するものであり、画像によってライン番号間のペナルティーは変化する。本実施形態では、類似検索元画像の自分自身の横（行）方向のラベル行列の距離を求め、これに基づいて各行間のペナルティーを求める。
【００７７】
図２２は第２の実施形態による動的なペナルティー値の設定手順を示すフローチャートである。ステップＳ１２１において、変数ｉを１に、変数ｊを２にそれぞれセットする。次に、ステップＳ１２２において、類似検索元画像の第ｉ行目のラベル列を取得し、ステップＳ１２３において、類似検索もと画像の第ｊ行目のラベル列を取得する。そして、ステップＳ１２４において、類似検索元画像の第ｉ行目のラベル列と第ｊ行目のラベル列とについて、色ペナルティーマトリクスを用いてＤＰマッチングを行い、距離を獲得する。ステップＳ１２５では、ステップＳ１２４で得たＤＰマッチングの距離を、類似検索元画像のｉ行目のラベル列とｊ行目のラベル列間のペナルティーとしてＬＩＮＥ[i][j]に記憶すると共に、、類似検索元画像のｊ行目のラベル列とｉ行目のラベル列間のペナルティーとしてＬＩＮＥ[j][i]に記憶する。
【００７８】
ステップＳ１２６によって、変数ｊの値がｊｍａｘとなるまで、ステップＳ１２３〜Ｓ１２５の処理が繰返される。この結果、第ｉ行目のラベル列について、ｉ＋１〜ｊｍａｘ行目の各ラベル列との間のペナルティー値が決定される。そして、ステップＳ１２８、Ｓ１２９、Ｓ１３０により、ステップＳ１２３〜Ｓ１２６の処理を変数ｉの値がｉｍａｘ−１となるまで繰返される。この結果、ＬＩＮＥ[i]「ｊ］には、ｉ＝ｊの対角成分を除く全てに、上記処理で決定されたペナルティー値が記憶されることになる。
【００７９】
次にステップＳ１３１では、上記処理で決定されていないＬＩＮＥ[i][j]の対角成分のペナルティーを決定する。この部分はｉ＝ｊであり、同一のラベル列であるから、その距離は０であり、従ってペナルティー０が記憶される。また、ステップＳ１３２では、「！」に関してペナルティーを決定する。すなわち、「！」に対するペナルティーは、ＬＩＮＥ[i][j]の全てのペナルティー値の中で、最大のペナルティー値よりもある程度大きな値を設定する。ただし、このペナルティー値を極端に大きくすると、曖昧にヒットする性質が損なわれてしまう。
【００８０】
以上のようにして類似検索元画像に関して計算されたラベル列間のペナルティーを用いて、上記ステップＳ１１３におけるＤＰマッチングを行い、類似度検索元画像と類似比較先画像の類似度を獲得する。
【００８１】
一方、固定的なペナルティーでは、ＤＰマッチングのペナルティーとして、ラベルが一致すればペナルティー「０」を、一致しない場合、もしくは「！」との比較であった場合にはある程度の大きさのペナルティーを与える。この場合は類似検索元画像によらず、常に同じペナルティーとなる。このような固定的なペナルティーを用いてステップＳ１１３の処理を実行し、類似度検索元画像と類似比較先画像の類似度を決定する。
【００８２】
以上説明したマッチング処理は次のような特徴を有する。もし、図２０の（ａ）と（ｂ）が極めて類似していれば、類似ライン行列は「１２３」となり、その距離は０となる。また、類似ライン行列が「！１２」や「２１２」であれば、類似検索元画像に対して類似比較先画像は下方向へずれたものである可能性があるし、類似ライン行列が「２３！」や「２３３」であれば類似検索元画像に対して類似比較先画像が上方向へずれたものである可能性がある。また、類似ライン行列が「１３！」や「！１３」であれば、類似検索元画像に対して類似比較先画像が縮小したものである可能性がある。同様に、類似比較先画像が類似検索元画像を拡大したようなものである場合も検出可能である。
【００８３】
上述のステップＳ１１３で説明したように、類似ライン行列と標準ライン行列との間でＤＰマッチングを行うことにより、垂直方向へのずれが効果的に吸収される。このため、上述したような、上方向や下方向へのずれ、拡大、縮小等に起因する類似検索元画像と類似比較先画像との相違が効果的に吸収され、良好な類似検索を行うことが可能となる。
【００８４】
すなわち、本実施形態の２次元ＤＰマッチングは、ラベル行列の各ラベル列における前後の曖昧さを許容するマッチングであり、画像の位置ずれの影響を吸収する性質を有する。また、アングルの違い等により物体の位置が変わり、ブロックによって切りとられる物体の位置が変わることにより、ブロックの色合いも微妙に異なることが予想されるが、この違いは上述のペナルティーマトリクスにより吸収されることになる。このように、本実施形態の２次元ＤＰマッチングによる曖昧さを許容するマッチングと、ペナルティーマトリクスによる特徴量の曖昧さの許容との相乗効果によって、上下左右拡大、縮小のずれに対しても影響の少ないマッチングを可能としている。
【００８５】
ただし、動的なペナルティーと固定的なペナルティーとでは、動的なペナルティーを用いる方が好ましい。例えば、一面麦畑の類似元検索画像があったとした場合、どのラインも似たようなラベル列となることが考えられる。一方、類似比較先画像にも一面麦畑の画像があったとした場合に、この画像の類似ライン行列には全て最初のライン番号１が入り、「１１１」となってしまう可能性がある。この場合、類似検索元画像のどのラインも似たような画像となっており、ライン番号間のペナルティーが極めて小さくなければ低い距離でのヒットはしない。しかしながら、動的なペナルティーを用いた場合は、ライン番号間のペナルティーが低くなり、類似度の高い結果を得ることができる。
【００８６】
一方、固定的なペナルティーを用いると、「１２３」と「１１１」ではペナルティー値が大きくなり、類似度が低くなってしまう。
【００８７】
ステップＳ２４ａでは、以上説明した類似度の算出処理をステップＳ２３ａで取得した全ラベル行列について行い、得られた類似度を高い順にソートして、ステップＳ２５ａへ進む。ステップＳ２５ａにおいて、画像管理ＤＢ１８ａを参照して、画像ＩＤ群の各画像ＩＤについてフルパスのファイル名を取得し、これをユーザに提示する。
【００８８】
以上のようにして、ＤＰマッチングを水平・鉛直方向、すなわち２次元に行うことにより、水平や鉛直方向、更に斜め方向に画像アングルが変わったり、物体が移動していても検索を行うことが可能である。また、ＤＰマッチングの時系列伸縮特性により、ズームアップ撮影画像やマクロ撮影画像をも検索することが可能となる。
【００８９】
なお、上記実施形態では、水平方向のブロック並びに対応するラベル列を用いて類似ライン行列を得たが、垂直方向のブロック並びに対応するラベル列を用いて類似ライン行列を得るようにすることも、上記と同様の手法で実現可能である。
【００９０】
以上説明したように、第２の実施形態によれば、特徴量群（特徴量空間を分割して得られる特徴量のグループ）を１つのシンボルで表現し（すなわちラベル化し）、ラベル同士の類似度に基づく距離を上述の２次元ＤＰマッチングとペナルティーマトリクスによって与える。このため、２つの画像のブロック間の距離の計算量を大幅に減少させることができるとともに、類似した特徴量が同じラベルで表されることになるので、類似画像の検索を良好に行うことができる。
【００９１】
また、（１）ペナルティマトリクスによるラベル同士の距離概念を導入し、（２）比較するラベル位置を前後曖昧に移動させることが出来、トータルの距離が最小（類似度が最大）となるようなラベル行列の比較を実現する上記２次元ＤＰマッチングを導入したことにより、画像のアングルが多少変わっても検索することが可能となり、雰囲気の似ている画像を検索できるようになる。
【００９２】
更に上記実施形態によれば、インデックスデータベース（ラベル成分インデックス）を用いたことにより、画像検索が更に高速化する。
【００９３】
すなわち、上記実施形態によれば、画像の特徴量の配置を考慮した類似画像の高速な検索が行われるとともに、撮影条件の変動等による違いを吸収した類似画像の検索が可能となり、従来難しかった画像のアングルが変ったり、物体の位置が変ったり、あるいは他の撮影条件が変動したりすることによる画像の特徴量のある程度の違いを吸収するなど、ロバストな類似画像検索を行うことが可能となる。
【００９４】
なお、上記各実施形態においては、自然画像検索を行う例を説明したが、本発明はＣＧやＣＡＤ等の人工的な画像の検索にも適応可能な技術であることは当業者には明らかである。
【００９５】
また、上記各実施形態では画像特徴量として色情報を選んだが、本発明はこれに限られるものではなく、その他の画像パラメータを画像分割ブロックごとに求めることで実施することも可能である。
【００９６】
また、上記各実施形態では１つの特徴量での認識の例を挙げたが、その他の特徴量での検索結果との論理演算を行うことにより、複数の特徴量からの高速な検索を行うことも可能である。
【００９７】
１つの画像に対して複数の画像特徴量を用いた検索を行う場合には、本発明で得られる類似度を１つの新たなる画像特徴量とみなし、複数のパラメータを用いた多変量解析を行い統計的な距離尺度を用いた検索を行うことも可能である。また、上記実施形態では、類似度が所定値を越える類似画像を検索結果として得るが、類似度の高い画像から順に前もって指定された個数の画像を検索結果として出力するようにしてもよいことはいうまでもない。
【００９８】
なお、曖昧度を指定することにより、ＤＰマッチングにおけるいわゆる整合窓の幅を変更することにより、検索の曖昧度を所望に設定可能とすることもできる。図２３はＤＰマッチングにおける整合窓を説明する図である。図２３において直線ＡはＪ＝Ｉ＋ｒで表され、直線ＢはＪ＝Ｉ−ｒで表される。整合窓の幅はｒの値を変更することで行える。したがって、キーボード１０４から曖昧度を指定することにより、このｒの値が変更されるように構成すれば、ユーザの所望の曖昧度（整合窓の幅）で類似度検索を行えるようになる。
【００９９】
なお、上記実施形態のような２次元ＤＰマッチングにおいては、水平方向のＤＰマッチングにおける整合窓の幅と、垂直方向のＤＰマッチングにおける整合窓の幅とを別々に設定できるようにしてもよい。或いは、両整合窓が異なる変化率で変化するように構成してもよい。このようにすれば、ユーザは、類似画像検索に際してよりきめ細かい曖昧さの設定を行えるようになる。例えば、図１８で示されたようなブロック順序を用いた場合において、検索元の画像中における注目物体の横方向への移動を許容したいような場合や、検索元の画像が横長画像であるような場合には、横方向への曖昧度を大きくするために水平方向のＤＰマッチングにおける整合窓の幅をより大きくすればよい。
【０１００】
なお、本発明は、例えばホストコンピュータ，インタフェイス機器，リーダ，プリンタなどの複数の機器から構成されるシステムに適用しても、一つの機器からなる装置（例えば、複写機，ファクシミリ装置など）に適用してもよい。
【０１０１】
また、本発明の目的は、前述した実施形態の機能を実現するソフトウェアのプログラムコードを記録した記憶媒体を、システムあるいは装置に供給し、そのシステムあるいは装置のコンピュータ（またはＣＰＵやＭＰＵ）が記憶媒体に格納されたプログラムコードを読出し実行することによっても、達成されることは言うまでもない。
【０１０２】
この場合、記憶媒体から読出されたプログラムコード自体が前述した実施形態の機能を実現することになり、そのプログラムコードを記憶した記憶媒体は本発明を構成することになる。
【０１０３】
プログラムコードを供給するための記憶媒体としては、例えば、フロッピディスク，ハードディスク，光ディスク，光磁気ディスク，ＣＤ−ＲＯＭ，ＣＤ−Ｒ，磁気テープ，不揮発性のメモリカード，ＲＯＭなどを用いることができる。
【０１０４】
また、コンピュータが読出したプログラムコードを実行することにより、前述した実施形態の機能が実現されるだけでなく、そのプログラムコードの指示に基づき、コンピュータ上で稼働しているＯＳ（オペレーティングシステム）などが実際の処理の一部または全部を行い、その処理によって前述した実施形態の機能が実現される場合も含まれることは言うまでもない。
【０１０５】
さらに、記憶媒体から読出されたプログラムコードが、コンピュータに挿入された機能拡張ボードやコンピュータに接続された機能拡張ユニットに備わるメモリに書込まれた後、そのプログラムコードの指示に基づき、その機能拡張ボードや機能拡張ユニットに備わるＣＰＵなどが実際の処理の一部または全部を行い、その処理によって前述した実施形態の機能が実現される場合も含まれることは言うまでもない。
【０１０６】
【発明の効果】
以上説明したように本発明によれば、画像の特徴量の配置を考慮した類似画像の検索が可能となる。
【０１０７】
また、本発明によれば、画像の特徴量の配置を考慮した類似画像の検索が行われるとともに、撮影条件の変動等による違いを吸収した類似画像の検索が可能となり、従来難しかった画像のアングルが変ったり、物体の位置が変ったり、あるいは他の撮影条件が変動したりすることによる画像の特徴量のある程度の違いを吸収するなど、ロバストな類似画像検索を行うことが可能となる。
【０１０８】
また、本発明によれば、特徴量ラベルに基づいて画像データを登録するインデックステーブルを導入することにより、類似度計算の対象とする画像データを効率よく絞り込むことが可能となり、類似検索の処理速度が著しく向上する。
【０１０９】
【図面の簡単な説明】
【図１】第１の実施形態による画像検索装置の制御構成を示すブロック図である。
【図２】第１の実施形態の画像検索装置の機能構成を示すブロック図である。
【図３】画像蓄積部１７における画像データの格納状態を説明する図である。
【図４】第１の実施形態による画像登録処理の手順を表すフローチャートである。
【図５】本実施形態による画像のブロック分割例を示す図である。
【図６】本実施形態による多次元特徴量空間を説明する図である。
【図７】ラベル列を生成する際のブロック順序例を説明する図である。
【図８】第１の実施形態における、画像管理データベースによる画像データの格納形態を説明する図である。
【図９】第１の実施形態による類似画像検索の処理手順を説明するフローチャートである。
【図１０】ラベル成分インデックスのデータ構成例を示す図である。
【図１１】ラベル列を比較し類似度を求める際に用いるラベル間のペナルティマトリックスの一例を示す図である。
【図１２】ＤＰマッチングによるラベル列間の距離の算出を説明する図である。
【図１３】ＤＰマッチングによるラベル列間の距離の算出を説明する図である。
【図１４】ＤＰマッチングによるラベル列間の距離の算出を説明する図である。
【図１５】本実施形態の画像検索装置の機能構成を示すブロック図である。
【図１６】第２の実施形態における、画像管理データベースによる画像データの格納形態を説明する図である。
【図１７】第２の実施形態による画像登録処理の手順を示すフローチャートである。
【図１８】ラベル行列を生成する際のブロック順序例を説明する図である。
【図１９】第２の実施形態による類似画像検索の処理手順を説明するフローチャートである。
【図２０】本実施形態による類似度算出処理を説明する図である。
【図２１】第２の実施形態による、２次元ＤＰマッチングを採用した類似度算出の手順を説明するフローチャートである。
【図２２】第２の実施形態による動的なペナルティー値の設定手順を示すフローチャートである。
【図２３】ＤＰマッチングにおける整合窓の調整を説明する図である。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image search apparatus and method for searching for an image.
[0002]
[Prior art]
Conventionally, various techniques for searching for similar images have been proposed. Many techniques that have been put to practical use for performing similar image search on natural images often use color information as image feature amounts. Many of them perform a search using a combination of colors that are present in the image and a ratio of RGB by taking a histogram relating to color information.
[0003]
[Problems to be solved by the invention]
However, in the above method, color position information is lost, so that the search accuracy is not necessarily high. For example, Japanese Patent Laid-Open No. 8-249349 discloses pattern matching using an image divided into a plurality of blocks and using respective feature amounts (representative colors). However, in this method, the distance of the feature amount between each block must be calculated for two images to be matched, which requires a huge amount of calculation. In particular, when a representative color is used as a feature quantity, three RGB data must be handled, and the calculation is further complicated. In addition, since the comparison is performed using the feature amount itself, the accuracy of the comparison is increased, but there is a problem that a similar image cannot be searched just by changing the angle of the image or the position of the object. That is, the image angle is changed, the position of the object is changed, or the image search is appropriately performed while having a certain degree of ambiguity, such as absorbing a certain amount of difference in the image feature amount depending on the shooting conditions. So-called robust similar image search could not be performed.
[0004]
Therefore, when searching for a natural image in the prior art, it is common to assign a keyword to the image and perform an image search using this keyword. However, this keyword attaching operation is a labor-intensive operation. Further, for an image that has not been assigned a keyword, an operation of presenting a reduced image and selecting it manually is required, and the search operation is complicated. It was.
[0005]
The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an image search apparatus and method capable of searching for similar images at high speed in consideration of the arrangement of image feature amounts.
[0006]
Another object of the present invention is to provide an image search apparatus and method that can search for a similar image in consideration of the arrangement of image feature amounts and can search for a similar image that absorbs differences due to changes in shooting conditions. It is to provide.
[0007]
Another object of the present invention is to reduce the amount of calculation of similarity by expressing a feature group with one label, expressing an image with a label string or a matrix, and calculating the similarity between images, It is to enable a quick similar image search.
[0008]
Another object of the present invention is to appropriately manage a label column or a label matrix, and to remarkably improve the processing speed of image search processing using labels.
[0009]
Another object of the present invention is to allow ambiguity before and after the label position such as DP matching and fuzzy nondeterministic automaton when the similarity between the original image and the comparison target image is performed by comparing the label sequence or the label matrix. The method is applied to enable a more effective similar image search.
[0010]
[Means for Solving the Problems]
In order to achieve the above object, an image search apparatus according to the present invention comprises the following arrangement. That is,
An assigning unit that divides the image data into a plurality of blocks and assigns a label according to the feature amount acquired for each block;
Generating means for generating a label string by arranging the labels given by the giving means based on a predetermined block order;
First storage means for storing the label sequence generated by the generation means in association with the image data;
Based on the generated label sequence, the image data includes the label for each label, using the image data as a key. Image ID indicating And a second storage means for storing an index table for registering the number of labels contained in each image data,
Search source Search Given the original image data, The search source A label sequence corresponding to image data is obtained, and the index table Based on the image ID and the number of labels contained in Extraction means for extracting an image data group similar to the acquired label sequence;
For each of the image data groups extracted by the extraction means, based on the label sequence stored in the first storage means Search source Search means for searching for image data corresponding to the image data.
[0011]
According to another aspect of the present invention, an image search method corresponding to the image search device and a storage medium for storing a control program for realizing the image search method are provided.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a preferred embodiment of the present invention will be described with reference to the accompanying drawings.
[0013]
[First Embodiment]
FIG. 1 is a block diagram showing a control configuration of the image search apparatus according to the first embodiment. In the figure, reference numeral 101 denotes a CPU which executes various controls in the image search apparatus of this embodiment. Reference numeral 102 denotes a ROM which stores a boot program executed when the apparatus is started up and various data. Reference numeral 103 denotes a RAM which stores a control program for processing by the CPU 101 and provides a work area when the CPU 101 executes various controls. A keyboard 104 and a mouse 105 provide various input operation environments for the user.
[0014]
An external storage device 106 includes a hard disk, a floppy (registered trademark) disk, a CD-ROM, and the like. 107 is It is a display. 108 It is a network interface and enables communication with each device on the network. Reference numeral 109 denotes an interface, and 110 denotes a scanner for reading an image. Reference numeral 111 denotes a bus connecting the above-described components. It should be noted that a control program that realizes processing shown in each flowchart described below may be stored in the ROM 102 or may be loaded into the RAM 103 from the external storage device 106.
[0015]
In the above configuration, the scanner 110 and the external storage device 106 may be replaced with those arranged on the network.
[0016]
FIG. 2 is a block diagram illustrating a functional configuration of the image search apparatus according to the first embodiment. In the figure, reference numeral 11 denotes a user interface unit, which detects various operation inputs from the user using the display 107, the keyboard 104, and the mouse 105. An image input unit 12 reads an image by the scanner 110. An image memory 13 stores image data obtained by the image input unit 12 in a predetermined area of the RAM 103. Reference numeral 14 denotes an image feature amount extraction unit, which extracts a feature amount of an image stored in the image memory 13 by a procedure described later. Reference numeral 15 denotes a feature amount label array unit that generates a label sequence based on the feature amount obtained by the image feature amount extraction unit 14. Reference numeral 16 denotes a pattern matching unit, which calculates a similarity between the specified image label sequence and the image label sequence stored in the image storage unit 17 and searches for similar images.
[0017]
An image storage unit 17 stores image data obtained by the image input unit 12 or the like. FIG. 3 is a diagram for explaining a storage state of image data in the image storage unit 17. An image ID 111 is assigned to each image data 112, and these are stored in the image storage unit 17 in pairs. Reference numeral 18 denotes an image management database (hereinafter referred to as an image management DB), which manages image data stored in the image storage unit 17 in the data format shown in FIG. Reference numeral 19 denotes a label column index, which stores the label component index file shown in FIG. The use of the label component index will be described later with reference to the flowchart of FIG.
[0018]
An operation example of the image search apparatus of the present embodiment having the above configuration will be described below. In the following example, three colors of red (R), green (G), and blue (B) are adopted as image feature amounts focused on color, and description will be made using processing in a three-dimensional color space.
[0019]
[Image registration process]
First, processing performed at the time of image registration will be described. FIG. 4 is a flowchart showing the procedure of the image registration process according to the first embodiment. First, in step S <b> 11, an image using the image input unit 12 is read in accordance with a user instruction via the user interface unit 11 and held in the image memory 13. Next, in step S12, this image is divided into a plurality of blocks. In this embodiment, an image is divided into a plurality of vertical and horizontal blocks. FIG. 5 is a diagram showing an example of block division of an image according to the present embodiment. As shown in the figure, in this embodiment, it is assumed that an image is divided into a total of 9 blocks of 3 × 3 for the sake of explanation. Next, in step S13, the feature amount of each divided block is calculated, and the obtained feature amount is labeled according to the following procedure.
[0020]
Note that the 3 × 3 division used in the present embodiment is for illustrative purposes only. Actually, it is preferable to set the number of divisions to 10 × 10 or more for natural images. In addition, if the product is shown on a white plain background, the number of divisions is preferably 13 × 13 or more.
[0021]
FIG. 6 is a diagram for explaining a multidimensional feature amount space according to the present embodiment. As shown in FIG. 6, the multi-dimensional feature space (RGB color space) is divided into a plurality of blocks (color blocks), that is, cells (color cells), and each cell (color cell) is uniquely identified by a serial number. Give a label. Here, the reason why the multidimensional feature amount space (RGB color space) is divided into a plurality of blocks is to absorb a subtle difference in feature amount (color).
[0022]
For the multi-dimensional feature space, the image feature is not used as it is, and after obtaining and normalizing (normalizing) the average and variance of each parameter by experiment, for example, orthogonal transformation such as principal component analysis is performed, It is conceivable to use a meaningful dimension. The “significant dimension” is a dimension composed of principal component axes having a large contribution rate in the principal component analysis.
[0023]
In step S13, a predetermined image feature amount calculation process is performed on each divided block obtained in step S12 to determine which cell in the multi-dimensional feature amount space belongs to and a corresponding label. This process is performed for all blocks. That is, for each divided image block, calculation processing is performed to determine which color cell all pixels belong to, and the label of the most frequently used color cell is determined as the parameter label (color label) of the divided image block. Processing is performed for all blocks.
[0024]
When parameter labels are assigned to each block as described above, in step S14, the parameter labels assigned to the blocks are arranged in a predetermined block order, thereby providing a parameter label string (hereinafter referred to as a label string). ) Is generated. FIG. 7 is a diagram illustrating an example of a block order when generating a label string. The above-mentioned parameter labels are arranged according to the numbers at the bottom of the divided image block shown in FIG.
[0025]
Here, in FIG. 7A, the divided blocks are scanned in an oblique direction from the upper right to the lower left. This is because, in order to reduce the influence of subtle differences in the angles of the images to be compared and the shift, as many consecutive label rows as possible along the similar search target object are obtained with a high expected value. As a result, coupled with the operation of the pattern matching unit 16 described later, it is possible to compare label rows that have little influence on both the vertical and horizontal shifts.
[0026]
As a scanning method applicable to this embodiment,
・ Horizontal direction (4 scanning methods are possible, such as scanning from left to right from top to bottom, scanning from left to right from bottom to top),
・ Vertical direction (4 scanning methods are possible, such as scanning from top to bottom from left to right)
-Diagonal direction (there are diagonal scans in two directions for each starting point of the four corners, and there are eight scanning methods including (a) to (c) in FIG. 7),
-Zigzag scan (scanning method adopted in JPEG etc., there are 2 zigzag scans for each start point of the four corners, there are a total of 8 scans),
Etc. In the present embodiment, a scanning method to be adopted is determined from the following viewpoints. That is,
(1) In this embodiment, it is a time-series comparison between label strings, and it is not preferable that the order is reversed. Therefore, it is necessary to scan all images by a predetermined scanning method to form a label array.
(2) It is desirable that blocks with close positions be located close to each other in the label row.
(3) The label of the block caught on the object to be searched appears as early as possible and continues for a long time to facilitate matching.
(4) The label arrangement should not change drastically even if the object moves or the angle changes.
A scanning method that satisfies this condition is adopted. In particular, assuming that most of the target object is in the center of the image, a block including the target object appears as early as possible in the scan, and in this embodiment, the expected value for scanning the target object for a long time is diagonal. Adopt direction scanning. In this embodiment, an oblique scan from the upper right to the lower left as shown in FIG. 7A is adopted. Naturally, an example as shown in FIG. 7B or an example shown in FIG. Any scanning method may be adopted.
[0027]
Subsequently, in step S15, the label sequence and image data obtained as described above are stored in the image storage unit 17, the image management DB 18, and the label sequence index 19. That is, an image ID is acquired for the image data read in step S11, and these are paired and stored in the image storage unit 17. Then, an image management DB record shown in FIG. 8 is generated in association with the image ID, and this is registered in the image management DB 18.
[0028]
The above is the process performed at the time of image registration.
[0029]
[Similar image search processing]
Next, similar image search processing will be described with reference to the flowchart of FIG. FIG. 9 is a flowchart for explaining the processing procedure of the similar image search according to the first embodiment. In the first embodiment, at the time of initialization, a label component index file (see FIG. 5) is obtained by obtaining a label sequence group of already registered images from the label series of the image management DB and using each label component as a key. 10) is generated and stored in the label column index 19. Here, the initialization time may be either when the system is started up or when an application is started. Also, when there is a new image registration and this is registered in the image DB, this label component index is generated. FIG. 10 is a diagram illustrating a data configuration example of a label component index. As shown in FIG. 10, the label component index has, for each label component, an address group (column ID group) to a label column having the label therein. The label component index file does not need to be recreated until it is necessary to reflect the registration, deletion, and change of the image. Of course, there is also a method of adding a label component index every time image data is registered. In this case, the process described above is performed on the newly registered image after step S15 in FIG.
[0030]
The label component index shown in FIG. 10 stores an image ID including the label component and the number of the labels included in the label component as a key. If such an index is used, the image ID of the image data including the label component of the search source image can be directly extracted. If the image IDs stored for each key are sorted and registered in ascending or descending order according to the number of labels, the processing speed can be further improved.
[0031]
First, when a similar search source image is specified from the user interface unit 11 in step S21, the image ID of the specified similar search source image is acquired in step S22, and the label sequence of the original image is further acquired from the image management DB 18. (In this example, a color label row) is acquired.
[0032]
Next, in step S23, the label component index file is referred to, and a label sequence group (label sequence in the label series index) including a label that is at least somewhat identical to the label sequence of the similar search source image is acquired. This is because the processing is slowed down if all the label sequences of the registered images are compared, so after narrowing down to similar ones in advance, the label sequence of the similar search source image is compared on a one-to-one basis. It is for improving. Note that the similar label string refers to a label string group including a predetermined number or more of the same or similar labels as the label string of the similar search source image. Here, similar labels refer to labels having a small penalty between labels in FIG. Of course, if the processing can be delayed, comparison with all the label sequences of the registered image may be performed to perform a highly accurate search (in this case, step S23 is omitted).
[0033]
As a method of narrowing down an image using the above-described label component index file, the following method can be considered. For example, in this example, a label string consisting of nine labels is used, and an image including three labels (fourth to sixth labels) at the center of the label string is extracted. This is because there is generally characteristic information of the image at the center of the image.
[0034]
Next, in step S24, each label sequence acquired in step S23 is compared with the label sequence of the similar search source image, and the similarity is calculated. And it outputs as a search result with the similarity in order from the label sequence nearest to the label sequence of the similar search source image.
[0035]
Here, a method for performing similarity comparison (calculation of similarity) between label sequences will be described.
[0036]
FIG. 11 is a diagram illustrating an example of a penalty matrix between labels used when comparing label strings to obtain similarity. The smaller the value in the matrix, the more similar. For example, the penalty of label 2 and label 6 is “7”. Moreover, the penalty between the same labels is “0” as a matter of course. The purpose of using this matrix is to perform distance determination according to the similarity of labels. That is, in this embodiment, since the RGB color space is used as the feature amount space, the distance determination according to the similarity of colors can be performed.
[0037]
For example, the distance is obtained by referring to the penalty matrix in FIG. 11 from the label values of the corresponding positions in the label sequence of the search source image and the search target image, and the sum of the distances for all the labels in the label sequence is obtained. Thus, the distance between both label rows is obtained. For example, in the example shown in FIG. 12, since the label sequence of the search source image is “112313441” and the label sequence of the search target image is “1132244452”, by performing DP matching using the penalty matrix of FIG. As shown in FIG. 13, the distance (final solution) is obtained. In this example, the following conditions are used as the tilt restriction. That is, in FIG. 14, the costs on the lattice points (i−1, j), (i−1, j−1) and (i, j−1) are respectively expressed as g (i−1, j) and g (i -1, j-1), g (i, j-1), and the penalty on grid point (i, j) is d (i, j), the cost on grid point (i, j) g (i, j) is obtained by the following recurrence formula.
[0038]
[Expression 1]

[0039]
As described above, in order to reduce the penalty (distance) between adjacent cells when pattern matching between labels and to give a large penalty to distant cells, a penalty matrix between labels as shown in FIG. 11 is introduced. To do. In step S24, this penalty matrix is taken into consideration and the label sequences are compared with each other. At this time, matching that allows an ambiguous comparison of label sequences such as automata may be performed. By using such an obfuscation technique, a low penalty is given to the addition of an extra label, the loss of a label, and the repetition of the same label, and the penalty between labels is between the color labels in FIG. By calculating the distance between label sequences using a penalty matrix, ambiguous pattern matching can be performed. As the automaton, the “fuzzy nondeterministic finite automaton” described in “Ambiguous Character String Retrieval Method and System Using Fuzzy Nondeterministic Finite Automaton of JP-A-8-241335” can be applied. In this automaton, the distance between each symbol (penalty) can be multivalued, and the label position to be compared can be moved back and forth vaguely, so that the total distance is minimized (similarity is maximized). In addition to the automaton described above, there is DP matching used in speech recognition or the like as a method for realizing a comparison of simple label sequences, and this method can also be suitably applied to this embodiment.
[0040]
Furthermore, in addition to the ambiguous pattern matching described above, the block order rules shown in FIGS. 7A to 7C are used in combination so that the influence of subtle differences and shifts in the angles of the images to be compared is reduced. It is possible to compare color label rows that have little influence on either the left or right shift. That is, DP matching and fuzzy nondeterministic automata are matching that allows ambiguity before and after the label sequence, and have the property of absorbing the influence of image misalignment. Also, it is expected that the color of the block will be slightly different due to the change in the position of the object due to the difference in angle and the position of the object cut by the block, but this difference is absorbed by the above penalty matrix. Will be. As described above, the synergistic effect of the matching that allows ambiguity by DP matching or fuzzy automaton and the tolerance of ambiguity of the feature value by the penalty matrix enables matching that has little influence on vertical and horizontal shifts. Furthermore, since the change in the label position due to the change in the position of the object is reduced by the oblique scanning as shown in FIGS. 7A to 7C, the influence of the deviation of the object at the time of matching can be reduced more effectively.
[0041]
Next, in step S25, referring to the image management DB 18, a full path file name is acquired for each image ID of the image ID group, and this is presented to the user.
[0042]
Through the above processing, robust similar image search such as absorbing a certain degree of color difference caused by changing the angle of the image, changing the position of the object, or changing the shooting condition is performed at high speed. Can be performed.
[0043]
Note that in the case of a similarity search in which one parameter is added to one image that cannot be blocked, the similarity (created using the total sum of penalties) obtained by the present invention is used as one new feature value, and the statistics It is also possible to perform a search based on a typical distance measure. In the above-described embodiment, similar images having a degree of similarity exceeding a predetermined value are obtained as search results. However, a predetermined number of images may be output as search results in order from images with high similarity. Needless to say.
[0044]
As described above, according to the first embodiment, a feature amount group (a group of feature amounts obtained by dividing a feature amount space) is expressed by one symbol (that is, labeled), and the labels are similar to each other. The distance based on degrees is given by the penalty matrix. For this reason, the amount of calculation of the distance between the blocks of two images can be greatly reduced, and similar feature amounts are represented by the same label, so that similar images can be searched well. it can.
[0045]
In addition, (1) The concept of distance between labels using a penalty matrix is introduced. (2) The label position to be compared can be moved ambiguously back and forth, such as DP matching and fuzzy nondeterministic automata, and the total distance is minimized ( By introducing a method that enables comparison of label sequences with the maximum similarity), it is possible to search even if the angle of the image changes slightly, so that images with similar atmospheres can be searched Become.
[0046]
Furthermore, according to the above embodiment, the use of the index database (label component index (FIG. 10)) further speeds up the image search.
[0047]
[Second Embodiment]
In the first embodiment, the similar image search is performed by arranging the labels representing the feature amounts one-dimensionally, that is, as a label string. In the second embodiment, a similar image search is performed in consideration of a two-dimensional arrangement of labels representing feature quantities, that is, using a label matrix. Note that the control configuration of the image search apparatus according to the second embodiment is the same as that of the first embodiment (FIG. 1), and therefore description thereof is omitted here.
[0048]
FIG. 15 is a block diagram illustrating a functional configuration of the image search apparatus according to the present embodiment. In the figure, a user interface unit 11, an image input unit 12, an image memory 13, an image feature amount extraction unit 14, and an image storage unit 17 have the same configuration as that of the first embodiment (FIG. 2).
[0049]
A feature quantity label matrix forming unit 15 a generates a label matrix based on the feature quantity obtained by the image feature quantity extraction unit 14. Reference numeral 16a denotes a pattern matching unit, which calculates the degree of similarity using the later-described two-dimensional DP matching for the label matrix of the designated image and the label matrix of the image stored in the image storage unit 17. Reference numeral 18a denotes an image management database (hereinafter referred to as image management DB), which manages image data stored in the image storage unit 17 in the data format shown in FIG. Reference numeral 19a denotes a label matrix index, which stores the label component index file shown in FIG.
[0050]
An operation example of the image search apparatus according to the second embodiment having the above configuration will be described below. In the following example, three colors of red (R), green (G), and blue (B) are adopted as image feature amounts focused on color, and description will be made using processing in a three-dimensional color space.
[0051]
[Image registration process]
FIG. 17 is a flowchart showing the procedure of image registration processing according to the second embodiment. In FIG. 17, steps S11 to S13 are the same as the processes described in the first embodiment (FIG. 4).
[0052]
When a parameter label is assigned to each block by the processing of steps S11 to S13, a parameter label matrix (hereinafter referred to as a label) is obtained by arranging the parameter labels assigned to the blocks in a predetermined block order in step S14a. Matrix) is generated. FIG. 18 is a diagram illustrating an example of a block order when generating a label matrix. The above parameter labels are arranged according to the numbers at the bottom of the divided image block shown in FIG. When storing a label matrix in the image management database 18a or the label matrix index 19a, a two-dimensional label matrix arranged in a predetermined order as described above is stored as described above. Such a one-dimensional form is also referred to as a label matrix.
[0053]
Here, in FIG. 18A, the divided blocks are scanned in the horizontal direction from left to right, and this horizontal scanning is performed from top to bottom. As a scanning method applicable to this embodiment,
-Horizontal direction (as shown in FIG. 18A, in addition to the order of scanning from left to right from top to bottom, as shown in FIGS. 18B to 18D, left There are four scanning methods, such as scanning from right to right, from bottom to top)
・ Vertical direction (4 scanning methods are possible, such as scanning from top to bottom from left to right)
As shown in FIG. 18E, the scan is divided into even lines and odd lines.
[0054]
In the second embodiment, the scanning method shown in FIG. 18A is adopted, but the other scanning methods described above are also applicable.
[0055]
In step S15a, the label matrix and image data obtained as described above are stored in the image storage unit 17, the image management DB 18a, and the label matrix index 19a. That is, an image ID is acquired for the image data read in step S11, and these are paired and stored in the image storage unit 17. Then, an image management DB record shown in FIG. 16 is generated in association with the image ID, and this is registered in the image management DB 18a.
[0056]
The above is the process performed at the time of image registration.
[0057]
[Similar image search processing]
Next, similar image search processing will be described with reference to the flowchart of FIG. FIG. 19 is a flowchart for explaining the processing procedure of the similar image search according to the second embodiment. In this embodiment, at the time of initialization in advance, a label matrix group of already registered images is obtained from the label series of the image management DB, and a label component index file with each label component as a key is generated. Stored in the label matrix index 19a. Here, the initialization time may be either when the system is started up or when an application is started. Also, when there is a new image registration and this is registered in the image DB, this label component index is generated.
[0058]
Also in this embodiment, the label component index (FIG. 10) described in the first embodiment is used. That is, for each label component, an address group (column ID group) to a label matrix having the label therein is registered in the label component index. The label component index file does not need to be recreated until it is necessary to reflect the registration, deletion, and change of the image. Of course, as described in the first embodiment, there is a method in which a label component index is additionally written at the time of registration.
[0059]
First, when a similar search source image is specified from the user interface unit 11 in step S21, the image ID of the specified similar search source image is acquired in step S22a, and the label matrix of the original image is further acquired from the image management DB 18. (In this example, a color label matrix) is acquired.
[0060]
Next, in step S23a, the label component index file is referred to, and a label matrix group (label matrix in the label sequence index) including labels that are the same as the label matrix of the similar search source image to some extent is acquired. Since this slows down the process when comparing all the registered image sequences with the label sequence, after narrowing down to similar ones in advance, a one-to-one comparison is made with the label matrix of similar search source images, and the processing speed It is for improving. The similar label matrix means a label matrix group including a predetermined number or more of the same or similar labels as the label matrix of the similar search source image. Here, similar labels refer to labels having a small penalty between labels described with reference to FIG. Of course, if it is acceptable to slow down the processing, it is possible to perform a high-accuracy search by comparing with all the label matrices of the registered image (in this case, step S23a is omitted).
[0061]
Next, in step S24a, each label matrix acquired in step S23a is compared with the label matrix of the similar search source image, and the similarity is calculated. And it outputs as a search result with the similarity in order from the label matrix nearest to the label matrix of the similar search source image.
[0062]
Here, a method for performing similarity comparison (similarity calculation) between label matrices will be described. Hereinafter, the label matrix acquired in step S23a is referred to as a similar comparison destination image.
[0063]
Also in the second embodiment, when the similarity is obtained by comparing the label matrices, a penalty matrix between labels (FIG. 11) is used as in the first embodiment.
[0064]
In order to reduce the penalty (distance) between adjacent cells during pattern matching between labels and to give a large penalty to distant cells, a penalty matrix between labels as shown in FIG. 12 is introduced. In step S24a, the penalty matrices are considered and the label matrices are compared with each other. In the second embodiment, two-dimensional DP matching described below (hereinafter referred to as two-dimensional DP matching) is used at that time. .
[0065]
FIG. 20 is a diagram for explaining similarity calculation processing according to this embodiment. The label matrix of the similar search source image acquired in step S22a described above can be arranged as shown in FIG. 20B according to the scanning method. If one of the label matrix groups extracted in step S23a is a similar search source image, the label matrix can be arranged as shown in FIG.
[0066]
First, the label column “abc” in the first row of the similar comparison destination image and the label columns (“123”, “456”, “789”) in the first to third rows of the similar search source image, respectively Is obtained by DP matching, and the row number in the similar search source image of the label column having the smallest distance is stored in the corresponding position of the similar line matrix ((c) of FIG. 20). If the obtained minimum distance is larger than a predetermined threshold, it is determined that no row is similar, and “!” Is stored at the corresponding position in the similar line matrix. Due to the nature of DP matching, for example, even if the angle of the image is slightly changed in the horizontal direction, a similar line (line) can be detected by the above processing. By performing the above processing for all rows (“def” and “ghi”) of the similar comparison target image, a similar line matrix in the column direction as shown in FIG. 20C is obtained.
[0067]
In FIG. 20C, a row similar to “abc” does not exist in the similar search source image, and a row similar to “def” is the first row of the similar search source image, a row similar to “ghi”. Is the second line of the similar search source image. The similarity between the similar line matrix obtained as described above and the standard line matrix (the row sequence of similar search source images, which is “123” in this example) is calculated using DP matching, Is output as the similarity between the similarity search source image and the similarity comparison target image. In DP matching, as is well known, the comparison is performed by expanding / contracting the label sequence to be compared (withstanding the partner to be compared without proceeding) so that the label sequence to be compared has the smallest similarity distance. I do. It is also possible to give a constraint condition (the width of the matching window) to what extent the expansion / contraction (withstand) can be made.
[0068]
FIG. 21 is a flowchart for explaining the similarity calculation procedure employing the two-dimensional DP matching according to the second embodiment. The processing described with reference to FIG. 20 will be further described with reference to the flowchart of FIG.
[0069]
First, in step S101, a variable i representing the row number of the similar comparison target image and a variable j representing the row number of the similar search source image are initialized to 1, and are set to indicate the first row. In step S102, the i-th label column of the similar comparison target image is acquired. For example, in the case of FIG. 20, if i = 1, “abc” is acquired. In step S103, the label column in the j-th row of the similar search source image is acquired. For example, in FIG. 20, if j = 1, “123” is acquired.
[0070]
In step S104, the distance between the two label columns obtained in steps S102 and S103 is obtained by DP matching using the color cell penalty matrix described in FIG. In step S105, if the distance obtained in step S104 is the minimum distance obtained so far with respect to the i-th row, the row number (j) is stored in the line matrix element LINE [i]. .
[0071]
The processing from step S103 to step S105 is performed for all rows of the similar search source image (steps S106 and S107). As described above, for the i-th label column of the similar comparison target image, the number of the closest row among the rows included in the similar search source image is stored in LINE [i]. Become.
[0072]
In step S108, LINE [i] obtained by the above processing is compared with a predetermined threshold (Thresh). If LINE [i] is greater than or equal to Thresh, step S10. 9 Then, “!” Indicating that no line is similar is stored in LINE [i].
[0073]
LINE [1] to LINE [imax] are obtained by executing the processing of steps S102 to S108 described above for all the rows of the similar comparison target image (steps S110 and S111). Output as [].
[0074]
Next, in step S113, DP matching between the standard line matrix “12... Imax” and the similar line matrix LINE [] is performed, and the distance between the two is calculated. Here, the standard line matrix is a matrix starting from 1 and increasing by 1 in the column direction.
[0075]
Here, the penalty used in DP matching between the standard line matrix and the similar line matrix will be described. Two methods are conceivable for setting the DP matching penalty between the similar line matrix in the column direction and the standard line matrix. That is, there are two dynamic penalties and fixed penalties.
[0076]
The dynamic penalty dynamically sets a penalty between line numbers, and the penalty between line numbers varies depending on the image. In the present embodiment, the distance of the label matrix in the horizontal (row) direction of the similar search source image is obtained, and a penalty between the rows is obtained based on the distance.
[0077]
FIG. 22 is a flowchart showing a procedure for setting a dynamic penalty value according to the second embodiment. In step S121, variable i is set to 1 and variable j is set to 2. Next, in step S122, the label column of the i-th row of the similar search source image is acquired, and in step S123, the label column of the j-th row of the similar search source image is acquired. In step S124, DP matching is performed on the i-th label column and the j-th label column of the similar search source image using a color penalty matrix to obtain a distance. In step S125, the DP matching distance obtained in step S124 is stored in LINE [i] [j] as a penalty between the i-th label column and the j-th label column of the similar search source image, It is stored in LINE [j] [i] as a penalty between the j-th label column and the i-th label column of the similar search source image.
[0078]
Steps S123 to S125 are repeated until the value of variable j reaches jmax in step S126. As a result, for the i-th label column, a penalty value between the i + 1-th and jmax-th label columns is determined. Then, in steps S128, S129, and S130, the processes in steps S123 to S126 are repeated until the value of the variable i becomes imax-1. As a result, the penalty value determined in the above process is stored in LINE [i] “j” except for the diagonal component of i = j.
[0079]
Next, in step S131, the penalty of the diagonal component of LINE [i] [j] that has not been determined in the above process is determined. Since this portion is i = j and is the same label string, the distance is 0, and thus penalty 0 is stored. In step S132, a penalty is determined for “!”. That is, the penalty for “!” Is set to a value that is somewhat larger than the maximum penalty value among all the penalty values of LINE [i] [j]. However, if this penalty value is extremely increased, the ambiguous hit property is lost.
[0080]
DP matching in step S113 is performed using the penalty between the label strings calculated for the similar search source image as described above, and the similarity between the similarity search source image and the similar comparison target image is acquired.
[0081]
On the other hand, with a fixed penalty, a penalty of “0” is given as a penalty for DP matching if the labels match, and a certain amount of penalty is given if they do not match or are compared with “!”. . In this case, the penalty is always the same regardless of the similar search source image. The process of step S113 is executed using such a fixed penalty, and the similarity between the similarity search source image and the similar comparison destination image is determined.
[0082]
The matching process described above has the following characteristics. If (a) and (b) in FIG. 20 are very similar, the similar line matrix is “123” and the distance is zero. If the similar line matrix is “! 12” or “212”, the similar comparison target image may be shifted downward with respect to the similar search source image, and the similar line matrix is “23”. ! "Or" 233 ", there is a possibility that the similar comparison target image is shifted upward with respect to the similar search source image. Further, if the similar line matrix is “13!” Or “! 13”, there is a possibility that the similar comparison destination image is reduced with respect to the similar search source image. Similarly, it is possible to detect a case where the similar comparison target image is an enlargement of the similar search source image.
[0083]
As described in step S113 above, by performing DP matching between the similar line matrix and the standard line matrix, the shift in the vertical direction is effectively absorbed. Therefore, as described above, the difference between the similar search source image and the similar comparison target image due to the upward or downward shift, enlargement, reduction, or the like is effectively absorbed, and a good similarity search is performed. Is possible.
[0084]
That is, the two-dimensional DP matching according to the present embodiment is a matching that allows ambiguity before and after each label column of the label matrix, and has a property of absorbing the influence of image positional deviation. Also, it is expected that the color of the block will be slightly different due to the change in the position of the object due to the difference in angle and the position of the object cut by the block, but this difference is absorbed by the above penalty matrix. Will be. As described above, the synergistic effect of the matching that allows ambiguity by the two-dimensional DP matching of the present embodiment and the allowance of ambiguity of the feature amount by the penalty matrix has no influence on the vertical / horizontal enlargement / reduction shift. Less matching is possible.
[0085]
However, it is preferable to use a dynamic penalty between the dynamic penalty and the fixed penalty. For example, if there is a similar source search image of a whole wheat field, it is conceivable that every line has a similar label sequence. On the other hand, if there is an image of the whole wheat field in the similar comparison destination image, all the similar line matrices of this image may have the first line number 1 and become “111”. In this case, all lines of the similar search source image are similar images, and hits at low distances are not made unless the penalty between line numbers is extremely small. However, when a dynamic penalty is used, the penalty between line numbers is low, and a result with high similarity can be obtained.
[0086]
On the other hand, if a fixed penalty is used, the penalty value increases for “123” and “111”, and the similarity decreases.
[0087]
In step S24a, the similarity calculation process described above is performed for all label matrices acquired in step S23a, and the obtained similarities are sorted in descending order, and the process proceeds to step S25a. In step S25a, with reference to the image management DB 18a, a full path file name is acquired for each image ID of the image ID group, and this is presented to the user.
[0088]
By performing DP matching in the horizontal and vertical directions, that is, two-dimensionally as described above, it is possible to perform a search even if the image angle changes in the horizontal and vertical directions and further in the oblique direction, or the object moves. It is. Also, it is possible to search for a zoomed-up photographed image and a macro photographed image by the DP matching time-series expansion / contraction characteristics.
[0089]
In the above embodiment, the similar line matrix is obtained using the horizontal block and the corresponding label column. However, the similar line matrix may be obtained using the vertical block and the corresponding label column. It can be realized by the same method as described above.
[0090]
As described above, according to the second embodiment, a feature amount group (a group of feature amounts obtained by dividing a feature amount space) is expressed by one symbol (that is, labeled), and the labels are similar to each other. The distance based on the degree is given by the above-described two-dimensional DP matching and penalty matrix. For this reason, the amount of calculation of the distance between the blocks of two images can be greatly reduced, and similar feature amounts are represented by the same label, so that similar images can be searched well. it can.
[0091]
In addition, (1) The concept of distance between labels using a penalty matrix is introduced, and (2) the label position to be compared can be moved ambiguously back and forth so that the total distance is minimum (similarity is maximum). By introducing the above-described two-dimensional DP matching that realizes matrix comparison, it is possible to search even if the angle of the image changes slightly, and it is possible to search for images having similar atmospheres.
[0092]
Furthermore, according to the above embodiment, the use of the index database (label component index) further speeds up the image search.
[0093]
That is, according to the above-described embodiment, it is possible to perform a high-speed search for similar images in consideration of the arrangement of image feature amounts, and to search for similar images that absorb differences due to changes in shooting conditions, which has been difficult in the past. It is possible to perform robust similar image search such as absorbing a certain amount of image feature amount due to image angle change, object position change, or other shooting conditions fluctuation etc. Become.
[0094]
In each of the above embodiments, an example of performing natural image search has been described. However, it is obvious to those skilled in the art that the present invention is a technique that can be applied to search for artificial images such as CG and CAD. is there.
[0095]
In each of the above embodiments, color information is selected as the image feature amount. However, the present invention is not limited to this, and the present invention can be implemented by obtaining other image parameters for each image division block.
[0096]
In each of the above embodiments, an example of recognition with one feature amount has been described. However, a high-speed search from a plurality of feature amounts can be performed by performing a logical operation with a search result with another feature amount. Is also possible.
[0097]
When a search using a plurality of image feature amounts is performed on one image, the similarity obtained in the present invention is regarded as one new image feature amount, and a multivariate analysis using a plurality of parameters is performed. It is also possible to perform a search using a statistical distance measure. In the above-described embodiment, similar images having a degree of similarity exceeding a predetermined value are obtained as search results. However, a predetermined number of images may be output as search results in order from images with high similarity. Needless to say.
[0098]
Note that by specifying the ambiguity, the search ambiguity can be set as desired by changing the width of the so-called matching window in DP matching. FIG. 23 is a diagram for explaining a matching window in DP matching. In FIG. 23, the straight line A is represented by J = I + r, and the straight line B is represented by J = I−r. The width of the alignment window can be changed by changing the value of r. Therefore, if the value of r is changed by designating the degree of ambiguity from the keyboard 104, the similarity search can be performed with the user's desired degree of ambiguity (matching window width).
[0099]
In the two-dimensional DP matching as in the above embodiment, the width of the matching window in the DP matching in the horizontal direction and the width of the matching window in the DP matching in the vertical direction may be set separately. Alternatively, both the matching windows may be configured to change at different rates of change. In this way, the user can set finer ambiguity when searching for similar images. For example, when the block order as shown in FIG. 18 is used, when it is desired to allow the object of interest to move in the horizontal direction in the search source image, or the search source image is a horizontally long image. In this case, the width of the matching window in the horizontal DP matching may be increased in order to increase the ambiguity in the horizontal direction.
[0100]
Note that the present invention can be applied to an apparatus (for example, a copier, a facsimile machine, etc.) composed of a single device even if it is applied to a system composed of a plurality of devices such as a host computer, interface device, reader, and printer. You may apply.
[0101]
Another object of the present invention is to supply a storage medium storing software program codes for implementing the functions of the above-described embodiments to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the storage medium. Needless to say, this can also be achieved by reading and executing the program code stored in the.
[0102]
In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.
[0103]
As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
[0104]
Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0105]
Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.
[0106]
【The invention's effect】
As described above, according to the present invention, it is possible to search for similar images in consideration of the arrangement of image feature amounts.
[0107]
In addition, according to the present invention, it is possible to search for a similar image in consideration of the arrangement of the feature amount of the image, and to search for a similar image that absorbs a difference due to a change in shooting conditions and the like. It is possible to perform a robust similar image search, such as absorbing a certain amount of difference in image feature amounts due to a change in position, an object position, or other imaging conditions.
[0108]
In addition, according to the present invention, by introducing an index table for registering image data based on a feature amount label, it is possible to efficiently narrow down image data to be subjected to similarity calculation, and processing speed of similarity search Is significantly improved.
[0109]
[Brief description of the drawings]
FIG. 1 is a block diagram showing a control configuration of an image search apparatus according to a first embodiment.
FIG. 2 is a block diagram illustrating a functional configuration of the image search apparatus according to the first embodiment.
FIG. 3 is a diagram for explaining a storage state of image data in an image storage unit 17;
FIG. 4 is a flowchart showing a procedure of image registration processing according to the first embodiment.
FIG. 5 is a diagram illustrating an example of block division of an image according to the present embodiment.
FIG. 6 is a diagram illustrating a multidimensional feature amount space according to the present embodiment.
FIG. 7 is a diagram illustrating an example of a block order when generating a label string.
FIG. 8 is a diagram for explaining a storage form of image data by an image management database in the first embodiment.
FIG. 9 is a flowchart illustrating a similar image search processing procedure according to the first embodiment;
FIG. 10 is a diagram illustrating a data configuration example of a label component index.
FIG. 11 is a diagram illustrating an example of a penalty matrix between labels used when label strings are compared to obtain similarity.
FIG. 12 is a diagram for explaining calculation of a distance between label strings by DP matching.
FIG. 13 is a diagram for explaining calculation of a distance between label strings by DP matching.
FIG. 14 is a diagram illustrating calculation of a distance between label strings by DP matching.
FIG. 15 is a block diagram illustrating a functional configuration of the image search apparatus according to the present embodiment.
FIG. 16 is a diagram illustrating a storage form of image data by an image management database in the second embodiment.
FIG. 17 is a flowchart illustrating a procedure of image registration processing according to the second embodiment.
FIG. 18 is a diagram illustrating an example of a block order when generating a label matrix.
FIG. 19 is a flowchart illustrating a similar image search processing procedure according to the second embodiment;
FIG. 20 is a diagram illustrating similarity calculation processing according to the present embodiment.
FIG. 21 is a flowchart illustrating a similarity calculation procedure employing two-dimensional DP matching according to the second embodiment.
FIG. 22 is a flowchart showing a procedure for setting a dynamic penalty value according to the second embodiment;
FIG. 23 is a diagram illustrating adjustment of a matching window in DP matching.

Claims (22)

An assigning unit that divides the image data into a plurality of blocks and assigns a label according to the feature amount acquired for each block;
Generating means for generating a label string by arranging the labels given by the giving means based on a predetermined block order;
First storage means for storing the label sequence generated by the generation means in association with the image data;
An index for registering an image ID indicating image data including the label for each label and registering the number of the label in each image data based on the generated label sequence and using the image data as a label as a key Second storage means for storing a table;
When search source image data serving as a search source is given, a label sequence corresponding to the search source image data is acquired , and the acquisition is performed based on the image ID and the number of labels contained in the index table. Extracting means for extracting an image data group similar to the label sequence;
Search means for searching for image data corresponding to the search source image data based on a label string stored in the first storage means for each of the image data groups extracted by the extraction means. An image search device. The image search apparatus according to claim 1, wherein the image data registered for each label in the index table is sorted based on the number of labels contained therein. The label is a unique label given to each of the cells obtained by dividing the multidimensional feature space into a plurality of cells.
The image search apparatus according to claim 1, wherein the assigning unit calculates a feature amount for each of the blocks, and assigns a label attached to a cell to which the calculated feature amount belongs to the block. .   The plurality of blocks are obtained by dividing an image into a plurality of vertical and horizontal blocks, and the block order used by the generating unit is an order of scanning the plurality of blocks in an oblique direction. Item 2. The image search device according to Item 1.   The plurality of blocks are obtained by dividing an image into a plurality of vertical and horizontal blocks, and the block order used by the generating means is an order of scanning the plurality of blocks in a horizontal or vertical direction. The image search device according to claim 1. The search means includes
A calculation means for calculating a similarity between the label string of the search source image data and the label string of the image data extracted by the extraction means;
The image search apparatus according to claim 1, further comprising: an output unit that outputs, as a search result, image data whose similarity by the calculation unit exceeds a predetermined value. The computing means has a penalty table for holding a penalty value for each pair of label values, and for each pair of label values obtained from the label sequence of the search source image data and the label sequence of the extracted image data The image search apparatus according to claim 6 , wherein a penalty value is acquired with reference to the penalty table, and the similarity is calculated based on the acquired penalty value. The label is a unique label given to each of the cells obtained by dividing the multidimensional feature space into a plurality of cells.
The image search apparatus according to claim 7 , wherein the penalty value is a value set based on a distance between cells represented by two labels. 8. The image according to claim 7 , wherein the calculation unit further gives a penalty value for excess or deficiency of labels in calculating the similarity between the label sequence of the search source image and the label sequence of the comparison destination image. 9. Search device. The image search apparatus according to claim 7 , wherein the calculation unit performs DP matching using the penalty value and calculates a similarity. The extracting means includes a label string including a predetermined number or more of labels that are the same as or similar to the labels included in the label string of the search source image data among the label strings stored in the first storage unit. The image search apparatus according to claim 1, wherein the image search apparatus is extracted by referring to the image search apparatus. The label sequence represents a two-dimensional label matrix;
The search means
The row-by-row label column extracted from the label matrix of the search source image data and the row-by-row label column extracted from the label matrix of the image data extracted by the extracting means are associated by DP matching and extracted. First matching means for obtaining a line sequence of the image data that has been processed;
The apparatus according to claim 1, further comprising: a second matching unit that obtains a similarity between the row sequence of the label matrix of the search source image data and the row sequence obtained by the first matching unit by DP matching. Image search device. 13. The image search apparatus according to claim 12 , wherein the row-by-row label column is an array corresponding to the horizontal direction of the image or an array corresponding to the vertical direction of the image. 13. The image search apparatus according to claim 12 , further comprising an output unit that outputs an image having a similarity degree obtained by the second matching unit exceeding a predetermined value as a search result. The first matching means has a penalty table that holds a penalty value for each pair of labels, and calculates the distance between the label sequence of the search source image and the label sequence of the comparison destination image using a DP matching method. 13. The image search apparatus according to claim 12 , wherein the penalty table is referred to at the time. The second matching means includes an inter-line penalty table that holds a penalty value for each pair of line numbers in a line arrangement, and uses a DP matching method to calculate the similarity between the line arrangement of the search source image and the line arrangement of the comparison destination image. 13. The image search apparatus according to claim 12 , wherein the calculation is performed by referring to the line spacing penalty table. 17. The storage device according to claim 16 , further comprising a holding unit that determines a penalty value for each pair of rows based on a similarity of each label column in the row direction of the original image data, and holds the penalty value as the penalty table between rows. The image search apparatus described. The first matching unit, when calculating the similarity between the label sequence of the original image data and the label sequence stored in the storage unit, further gives a penalty and a constraint associated with the series expansion / contraction of the label comparison. The image search device according to claim 12 . 13. The image search apparatus according to claim 12 , further comprising first setting means for setting a width of a matching window for DP matching used in the first matching means. 13. The image search apparatus according to claim 12 , further comprising second setting means for setting a width of a matching window for DP matching used in the second matching means. An image feature amount calculating unit divides the image into a plurality of blocks for the image data stored in the memory, and calculates a feature amount of each block from the image data of each block;
A labeling unit, a labeling process for labeling each block according to the feature amount calculated in the calculation process ;
A generation step of generating a label sequence by arranging the label string generating means, based on a label that has been applied in the applying step to the predetermined block order,
First storage means, a first storage step of storing the image management storage means a label sequence generated by said generating step in association with the image data,
Based on the generated label sequence , the second storage means registers an image ID indicating image data including the label for each label using the image data as a label, and the label of the label in each image data A second storage step of storing in the label storage means an index table for registering the content number;
The extraction means acquires the label sequence corresponding to the search source image data by the calculation step, the adding step, and the generation step according to the designation of the search source image data as the search source, and is registered in the index table. An extraction step of extracting an image data group similar to the acquired label sequence based on the image ID and the number of labels included;
Search means for each group of image data extracted by the extraction step, the retrieval result data by searching the image data corresponding to the search source image data based on the label string stored in said image storage means And a search step for outputting the image search method. In order to cause a computer to execute a similar image search process,
An assigning unit that divides an image into a plurality of blocks with respect to image data stored in a memory, calculates a feature amount of each block from the image data of each block, and assigns a label to each block according to the calculated feature amount When,
Generating means for generating a label string by arranging the labels given by the giving means based on a predetermined block order;
First storage means for storing the association data of the label sequence generated by the generation means and the image data in a memory;
An index for registering an image ID indicating image data including the label for each label and registering the number of the label in each image data based on the generated label sequence and using the image data as a label as a key Second storage means for storing the table in a memory;
In accordance with the designation of the search source image data as the search source, a label sequence corresponding to the search source image data is acquired, and the acquisition is performed based on the image ID and the number of labels contained in the index table. Extracting means for extracting an image data group similar to the label sequence;
Search for searching for image data corresponding to the search source image data based on a label string registered in the association data for each of the image data groups extracted by the extraction means and outputting the search result data storage medium characterized by storing a control program to function as the image retrieval and means.

Images