JP4790113B2 - Image processing apparatus and method, and recording medium - Google Patents

Description

【００６４】
図７（Ｃ）の例の場合、第１回目では、図７（Ａ）の例で示したように、ラベリングされた領域の外接枠内の画素を使って平均位置Ｃ１を算出する。第２回目では、平均位置Ｃ１より画面上方にある画素を使って平均位置Ｃ２を算出する。同様に、第３回目以降の平均位置を順次繰り返し算出する。そして、第ｎ回目の平均位置Ｃｎを、最終的な重心Ｃとする。
【００６５】
以上のような重心算出方法のいずれかを用いて、ラベリングされた領域（いまの場合、領域８５および８６）の重心が検出されると、ステップＳ４に進み、領域処理部２３の外接枠検出部５２は、ステップＳ２の処理でラベリングされた各領域（いまの場合、領域８５および８６）を外側から囲む外接枠（矩形）を検出する。
【００６６】
ステップＳ５において、面積検出部５３は、ステップＳ４の処理により検出された外接枠内に含まれる動きブロックの画素数をカウントすることにより面積を算出（検出）する。すなわち、図６（Ｆ）の領域８５の場合、動きブロック（すなわち、黒で示されているブロック）は１３個であり、例えば、１ブロックが１６画素で構成されているとすると、動き画素数は２０８個とカウントされ、そのカウントされた画素数から面積が算出される。
【００６７】
ステップＳ６において、面積検出部５３は、ステップＳ４の処理により検出された外接枠のアスペクト比（縦横比）を算出する。すなわち、図６（Ｆ）の領域８５の場合、縦のブロック数が５個であり、横のブロック数が４個であり、アスペクト比は、１．２５（＝５／４）と算出される。
【００６８】
ステップＳ７において、領域評価部５４は、ステップＳ３の処理で検出された、ラベリングされた領域（いまの場合、領域８５および８６）の重心、ステップＳ５の処理で算出された各領域の動きブロックの面積、ステップＳ６の処理で算出された領域の外接枠のアスペクト比に基づいて、次式（４）に従って、評価値を算出する。
Φ（ｉ）＝α_c｜Ｃ_i−Ｃ_p｜＋α_a｜Ａ_i−Ａ_p｜＋α_s｜Ｓ_i−Ｓ_p｜−α_mＮｍ_i
・・・（４）
【００６９】
なお、上記式（４）において、Ｃ_iは領域ｉの重心位置、Ａ_iは領域ｉのアスペクト比、Ｓ_iは領域ｉの面積、Ｎｍ_iは領域ｉの動き画素数を表わし、Ｃ_p，Ａ_p,Ｓ_pは前フレームで選択された領域ｐの重心位置、アスペクト比、面積をそれぞれ表わし、α_c，α_a，α_s，α_mは、重心位置、アスペクト比、面積、動き画素数の重要度を示すそれぞれの重み付け係数である。例えば、重み付け係数には、α_c＝１０．０，α_a＝１０．２，α_s＝０．２などの値が用いられる。
【００７０】
ステップＳ８において、領域評価部５４は、ステップＳ７の処理で算出された評価値のうち、評価値の最も小さい領域（例えば、領域８５）を選択する。
【００７１】
ステップＳ９において、動き検出部５６は、ステップＳ２の処理でラベリングされた領域（いまの場合、領域８５および８６）の動き方向を検出する。
【００７２】
ここで、図８のフローチャートを参照して、動き方向検出処理について説明する。なお、図９に示すように、３×３画素からなる現フレームの領域と１フレーム前の領域を比較し、現フレームの領域の動き方向を検出するものとして説明する。
【００７３】
ステップＳ４１において、動き検出部５６は、動きの方向を示す変数ｈ，ｖに０を代入する。ステップＳ４２において、動き検出部５６は、縦および横の両方向の動きが検出されたのか否かを判定し、未だ、縦および横の両方向の動きが検出されていないと判定した場合、ステップＳ４３に進み、現フレームの中心画素Ｐ（i,j）と１フレーム前の中心画素Ｐ´（i,j）の輝度差が所定の正の閾値より大きいのか否かを判定する。なお、ｉは、ｙ座標を表わし、ｊはｘ座標を表わしている。
【００７４】
ステップＳ４３において、Ｐ（i,j）−Ｐ´（i,j）が所定の正の閾値より大きいと判定された場合、ステップＳ４４に進み、動き検出部５６は、横方向の動きを検出するのか否か、すなわち、縦方向の動きが検出済みであるのか否かを判定し、横方向の動きを検出すると判定した場合（縦方向の動きが検出済みであると判定した場合）、ステップＳ４５に進む。ステップＳ４５において、動き検出部５６は、中心画素Ｐ（i,j）と画素Ｐ（i,j+1）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j）−Ｐ（i,j+1）が所定の閾値より大きいと判定した場合、ステップＳ４６に進み、変数ｈに１を代入し、ステップＳ４２に戻る。
【００７５】
ステップＳ４５において、Ｐ（i,j）−Ｐ（i,j+1）が所定の閾値より大きくはないと判定された場合、ステップＳ４７に進み、動き検出部５６は、さらに、中心画素Ｐ（i,j）と画素Ｐ（i,j-1）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j）−Ｐ（i,j-1）が所定の閾値より大きいと判定した場合、ステップＳ４８に進み、変数ｈに−１を代入し、ステップＳ４２に戻る。ステップＳ４７において、Ｐ（i,j）−Ｐ（i,j-1）が所定の閾値より大きくはないと判定された場合、ステップＳ４２に戻る。
【００７６】
また、ステップＳ４４において、横方向の動きではなく、縦方向の動きを検出すると判定された場合、ステップＳ４９に進み、動き検出部５６は、中心画素Ｐ（i,j）と画素Ｐ（i+1,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j）−Ｐ（i+1,j）が所定の閾値より大きいと判定した場合、ステップＳ５０に進み、変数ｖに１を代入し、ステップＳ４２に戻る。
【００７７】
ステップＳ４９において、Ｐ（i,j）−Ｐ（i+1,j）が所定の閾値より大きくはないと判定された場合、ステップＳ５１に進み、動き検出部５６は、さらに、中心画素Ｐ（i,j）と画素Ｐ（i-1,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j）−Ｐ（i-1,j）が所定の閾値より大きいと判定した場合、ステップＳ５２に進み、変数ｖに−１を代入し、ステップＳ４２に戻る。ステップＳ５１において、Ｐ（i,j）−Ｐ（i-1,j）が所定の閾値より大きくはないと判定された場合、ステップＳ４２に戻る。
【００７８】
また、ステップＳ４３において、Ｐ（i,j）−Ｐ´（i,j）が所定の正の閾値より大きくはないと判定された場合、ステップＳ５３に進み、動き検出部５６は、さらに、現フレームの中心画素Ｐ（i,j）と１フレーム前の中心画素Ｐ´（i,j）の輝度差が所定の負の閾値より小さいのか否かを判定する。
【００７９】
ステップＳ５３において、Ｐ（i,j）−Ｐ´（i,j）が所定の負の閾値より小さいと判定された場合、ステップＳ５４に進み、動き検出部５６は、横方向の動きを検出するのか否か、すなわち、縦方向の動きが検出済みであるのか否かを判定し、横方向の動きを検出すると判定した場合（縦方向の動きが検出済みであると判定した場合）、ステップＳ５５に進む。ステップＳ５５において、動き検出部５６は、画素Ｐ（i,j+1）と中心画素Ｐ（i,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j+1）−Ｐ（i,j）が所定の閾値より大きいと判定した場合、ステップＳ５６に進み、変数ｈに１を代入し、ステップＳ４２に戻る。
【００８０】
ステップＳ５５において、Ｐ（i,j+1）−Ｐ（i,j）が所定の閾値より大きくはないと判定された場合、ステップＳ５７に進み、動き検出部５６は、さらに、中心画素Ｐ（i,j-1）と画素Ｐ（i,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i,j-1）−Ｐ（i,j）が所定の閾値より大きいと判定した場合、ステップＳ５８に進み、変数ｈに−１を代入し、ステップＳ４２に戻る。ステップＳ５７において、Ｐ（i,j-1）−Ｐ（i,j）が所定の閾値より大きくはないと判定された場合、ステップＳ４２に戻る。
【００８１】
また、ステップＳ５４において、横方向の動きではなく、縦方向の動きを検出すると判定された場合、ステップＳ５９に進み、動き検出部５６は、画素Ｐ（i+1,j）と中心画素Ｐ（i,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i+1,j）−Ｐ（i,j）が所定の閾値より大きいと判定した場合、ステップＳ６０に進み、変数ｖに１を代入し、ステップＳ４２に戻る。
【００８２】
ステップＳ５９において、Ｐ（i+1,j）−Ｐ（i,j）が所定の閾値より大きくはないと判定された場合、ステップＳ６１に進み、動き検出部５６は、さらに、画素Ｐ（i-1,j）と中心画素Ｐ（i,j）の輝度差が所定の閾値より大きいのか否かを判定し、Ｐ（i-1,j）−Ｐ（i,j）が所定の閾値より大きいと判定した場合、ステップＳ６２に進み、変数ｖに−１を代入し、ステップＳ４２に戻る。ステップＳ６１において、Ｐ（i-1,j）−Ｐ（i,j）が所定の閾値より大きくはないと判定された場合、ステップＳ４２に戻る。
【００８３】
そして、ステップＳ４２において、縦および横の両方向の動きが検出されたと判定された場合、または、ステップＳ５３において、Ｐ（i,j）−Ｐ´（i,j）が所定の負の閾値より小さくはないと判定された場合、図４のステップＳ９にリターンする。
【００８４】
図４のステップＳ９において、動き検出部５６は、上述した処理により算出された変数ｈ，ｖの値を、図１０に示す対応図に対応付け、動き方向を示す画素値（インデックス値）を設定する。
【００８５】
図１０の例の場合、LU（Left Upper）は、左上に画素が動いたことを表わし、U（Upper）は、上に画素が動いたことを表わし、RU（Right Upper）は、右上に画素が動いたことを表わし、L（Left）は、左に画素が動いたことを表わし、N（No move）は、画素が動いていないことを表わし、R（Right）は、右に画素が動いたことを表わし、LD（Left Down）は、左下に動いたことを表わし、D（Down）は、下に画素が動いたことを表わし、そして、RD（Right Down）は、右下に画素が動いたことを表わす。
【００８６】
図１０において、例えば、N＝０，LU＝１，U＝２，RU＝３，L＝４，R＝５，LD＝６，D＝７，RD＝８であるとする。算出された変数ｈ，ｖがいずれも−１である場合、動き方向を示す画素値は１に設定される。算出された変数ｈが０、変数ｖが−１である場合、動き方向を示す画素値は２に設定される。算出された変数ｈが１、変数ｖが−１である場合、動き方向を示す画素値は３に設定される。算出された変数ｈが−１、変数ｖが０である場合、動き方向を示す画素値は４に設定される。算出された変数ｈ，ｖがいずれも０である場合、動き方向を示す画素値は０に設定される。算出された変数ｈが１、変数ｖが０である場合、動き方向を示す画素値は５に設定される。算出された変数ｈが−１、変数ｖが１である場合、動き方向を示す画素値は６に設定される。算出された変数ｈが０、変数ｖが１である場合、動き方向を示す画素値は７に設定される。そして、算出された変数ｈ，ｖがいずれも１である場合、動き方向を示す画素値は８に設定される。
【００８７】
すなわち、画素値が１である場合、左上に画素が動いたことになり、画素値が２である場合、上に画素が動いたことになり、画素値が３である場合、右上に画素が動いたことになり、画素値が４である場合、左に画素が動いたことになり、画素値が０である場合、画素が動いていないことになり、画素値が５である場合、右に画素が動いたことになり、画素値が６である場合、左下に動いたことになり、画素値が７である場合、下に画素が動いたことになり、そして、画素値が８である場合、右下に画素が動いたことになる。
【００８８】
なお、本願発明では、これに限らず、ブロックマッチングや動きベクトルマッチングなどの手法を用いて、動き方向を検出することも可能である。
【００８９】
例えば、ステップＳ８の処理で選択された領域のU，Dの画素値を持つ画素数Ｎ_U，Ｎ_Dをそれぞれカウントし、Ｎ_Uが所定の閾値を越えていた場合、領域内で「上向きの動きが発生した」と認識され、Ｎ_Dが所定の閾値を越えていた場合、領域内で「下向きの動きが発生した」と認識される。また、Ｎ_U，Ｎ_Dが、ともに所定の閾値を越えていた場合、「動きなし」と認識される。
【００９０】
図４のステップＳ１０に戻って、エッジ検出部５７は、ステップＳ２の処理でラベリングされた領域（図６（Ｆ）の例の場合、領域８５および８６）のうち、ステップＳ８の処理で選択された評価値の最も小さい領域（例えば、領域８５）のエッジ情報を検出する。
【００９１】
ここで、図１１を参照して、エッジ情報を検出する処理についてさらに詳しく説明する。なお、ステップＳ８の処理で、評価値が最も小さい領域として領域８５が選択されたものとして以下に説明する。
【００９２】
図１１（Ａ）は、ステップＳ３の処理により検出された領域８５の重心Ｃを示している。図１１（Ｂ）は、ステップＳ２の処理により２値化されたフレーム間差分画像を示している。
【００９３】
エッジ検出部５７は、ステップＳ２の処理で２値化されたフレーム間差分画像（図１１（Ｂ））の垂直方向および水平方向のエッジ情報をそれぞれ検出する。これにより、図１１（Ｃ）および図１１（Ｄ）に示されるように、垂直方向のエッジ情報と水平方向のエッジ情報が検出される。
【００９４】
次に、エッジ情報の検出処理についてさらに詳しく説明する。なお、エッジ情報の検出には、図１２に示されるようなフィルタが用いられる。
【００９５】
エッジ検出部５７は、エッジ情報を検出したい画素を中心に、図１２に示されるフィルタと画素値の積和（畳み込み）演算を行い、その演算結果の絶対値を２値化することによりエッジ画素を算出する。そして、エッジ検出部５７は、積和演算の絶対値が所定の閾値より大きい場合、それをエッジ画素として検出する。
【００９６】
まず、現フレームにおける中心画素Ｐ（ｉ，ｊ）（図９）が垂直方向のエッジ画素か否かを判定する例について説明する。エッジ検出部３７は、次式（５）に従って、図１２（Ａ）に示されるフィルタと画素値の積和演算Ｌｖを行う。
Ｌｖ＝−１・Ｐ（i-1,j-1）＋０・Ｐ（i-1,j）＋１・Ｐ（i-1,j+1）−２・Ｐ（i,j-1）
＋０・Ｐ（i,j）＋２・Ｐ（i,j+1）−１・Ｐ（i+1,j-1）＋０・Ｐ（i+1,j）
＋１・Ｐ（i+1,j+1） ・・・（５）
【００９７】
エッジ検出部５７は、上記式（５）の算出の結果、積和演算Ｌｖの絶対値が所定の閾値Ｔｖより大きいか否かを判定し、積和演算Ｌｖの絶対値が所定の閾値Ｔｖより大きいと判定した場合、画素Ｐ（ｉ，ｊ）を垂直方向のエッジ画素（エッジ情報）として検出する。
【００９８】
次に、現フレームにおける中心画素Ｐ（ｉ，ｊ）（図９）が水平方向のエッジ画素か否かを判定する例について説明する。エッジ検出部３７は、次式（６）に従って、図１２（Ｂ）に示されるフィルタと画素値の積和演算Ｌｈを行う。
Ｌｈ＝−１・Ｐ（i-1,j-1）−２・Ｐ（i-1,j）−１・Ｐ（i-1,j+1）＋０・Ｐ（i,j-1）
＋０・Ｐ（i,j）＋０・Ｐ（i,j+1）＋１・Ｐ（i+1,j-1）＋２・Ｐ（i+1,j）
＋１・Ｐ（i+1,j+1） ・・・（６）
【００９９】
エッジ検出部５７は、上記式（６）の算出の結果、積和演算Ｌｈの絶対値が所定の閾値Ｔｈより大きいか否かを判定し、積和演算Ｌｈの絶対値が所定の閾値Ｔｈより大きいと判定した場合、画素Ｐ（ｉ，ｊ）を水平方向のエッジ画素（エッジ情報）として検出する。
【０１００】
このような処理を、２値化されたフレーム間差分画像の各画素に対して行うことにより、垂直方向のエッジ情報と水平方向のエッジ情報が検出される。
【０１０１】
次に、Ｆｗ×Ｆｈの矩形領域９１（図１１（Ｅ））を縦方向に３等分したtop，middle，bottomの３領域について考える。
【０１０２】
エッジ検出部５７は、Ｆｗ×Ｆｈの矩形領域９１の中心が、図１１（Ａ）で示された重心Ｃとなるように設定し、それぞれの領域内の垂直方向のエッジピクセルの数をカウントするとともに（図１１（Ｆ））、水平方向のエッジピクセルの数をカウントする（図１１（Ｇ））。
【０１０３】
すなわち、ユーザ１の手７１（図１８（Ａ））が、指先を画面上方に向けて撮像されている場合、撮像された画像を３分割して各領域毎のエッジピクセルの数をカウントすることにより、手の握り動作時に発生する時間的特徴変化に、手の構造を反映させることができる。
【０１０４】
従って、手を握る動作の場合には、動作開始直後は、領域上部（図１１（Ｅ）のtopの領域）にピクセルエッジ分布が集中し、動作終了時には、領域中央部（図１１（Ｅ）のmiddleの領域）にピクセルエッジ分布が集中する。一方、領域下部（図１１（Ｅ）のbottomの領域）には、ピクセルエッジ分布がほとんど生じない。これに対して、手を開く動作の場合には、動作開始直後は、領域中央部にピクセルエッジが集中し、動作終了時には、領域上部にピクセルエッジ分布が集中する。一方、領域下部には、手を握る動作と同様に、ピクセルエッジ分布がほとんど生じない。
【０１０５】
図４に戻って、ステップＳ１１において、エッジ検出部５７は、topの領域に属する垂直エッジピクセル数をｅ_t ^v、middleの領域の属する垂直エッジピクセル数をｅ_m ^v、および、bottomの領域に属する垂直エッジピクセル数をｅ_b ^v、並びに、topの領域に属する水平エッジピクセル数をｅ_t ^h、middleの領域の属する水平エッジピクセル数をｅ_m ^h、および、bottomの領域に属する水平エッジピクセル数をｅ_b ^hとして、次式（７）に従って、特徴ベクトルｆを算出する。ここで、Ｆｓ＝Ｆｗ×Ｆｈである。
ｆ≡（Ｒ_t ^v，Ｒ_t ^h，Ｒ_m ^v，Ｒ_m ^h，Ｒ_b ^v，Ｒ_b ^h）
＝１／Ｆｓ・（ｅ_t ^v，ｅ_t ^h，ｅ_m ^v，ｅ_m ^h，ｅ_b ^v，ｅ_b ^h） ・・・（７）
【０１０６】
ステップＳ１２において、DP値算出部６２は、連続DPの手法を用いて、ステップＳ１１の処理で算出された特徴ベクトルｆに基づいて、DP値を算出する。ここで、認識対照パターンとして、ステップＳ１１の処理で算出された特徴ベクトルｆが用いられ、参照パターンとして、例えば、予め、手を握ったり開いたりしている動作がビデオカメラ２で撮像され、エッジ検出部５７で算出される特徴ベクトルｆの時系列のうち、手の握りあるいは開き動作に相当する部分が切り出されている。
【０１０７】
DP値算出部６２は、時刻ｔにおいて、参照パターンＲ＝｛ｒ（γ）｜１≦γ≦Ｔ｝と認識対象パターンｑ（ｔ）との距離ｄ（ｔ，γ）＝‖ｑ（ｔ）−ｒ（γ）‖を用いて、累積距離Ｓ（ｔ，γ）を、次式（８）および式（９）に従って算出する。ここで、次式（８）は、累積距離Ｓ（ｔ，γ）を算出するための初期条件とされる。
Ｓ（−１，γ）＝Ｓ（０，γ）＝∞（１≦γ≦Ｔ） ・・・（８）
【０１０８】
１≦γにおける累積距離Ｓ（ｔ，γ）は、次式（９）で表わされる。
【数２】

【０１０９】
ここで、γ＝Ｔとすると、Ｓ（ｔ，Ｔ）は、参照パターンＲの区域（１，Ｔ）について入力時系列パターンに最適に適合したときの累積距離を表わすことになる。これを重みの和３・Ｔで正規化すると、次式（１０）で表わされる。
【数３】

これを時刻ｔにおける連続DP値に決定する。
【０１１０】
図１３に示されるように、参照パターンＲに類似したパターンが発生したときに、この連続DP値は最小値minを取る。この最小値minが所定の閾値以下になれば、参照パターンＲとマッチングしたと判定される。従って、最小値minが検出されたとき、握り動作が認識されることになる。
【０１１１】
このように、DP値算出部６２は、認識対象パターンが手の握り参照パターンにマッチした場合、「握り」を握り動作判定部６１へ出力し、認識対象パターンが手の開き参照パターンにマッチした場合、「開き」を握り動作判定部６１へ出力する。
【０１１２】
握り動作判定部６１は、動き検出部５６から「上向きの動き」が入力された後、Ｆmoveフレームの間にDP値算出部６２から「開き」が入力されたとき、手の開き動作を認識することができる。また、握り動作判定部６１は、動き検出部５６から「下向きの動き」が入力された後、Ｆmoveフレームの間にDP算出部６２から「握り」が入力されたとき、手の握り動作を認識することができる。なお、Ｆmoveの値は、任意に設定することが可能であり、例えば、１５フレームなどの値が設定される。
【０１１３】
ステップＳ１３において、重心軌跡認識部３２は、ステップＳ３の処理で検出された重心に基づいて、手の振り動作を認識する。
【０１１４】
ここで、図１４および図１５のフローチャートを参照して、手の振り動作認識処理について説明する。
【０１１５】
ステップＳ１０１において、重心軌跡認識部３２は、図１６に示すような１０個のバッファで構成されるリングバッファに、各フレームで選択された領域の重心位置をそれぞれ代入する。ステップＳ１０２において、重心軌跡認識部３２は、ステップＳ１０１の処理で代入された現フレームの重心位置Ｘ，Ｙと、１フレーム前の重心位置Ｘ´，Ｙ´の更新を行う。いまの場合、現フレームの重心位置Ｘ＝Ｘ₅，Ｙ＝Ｙ₅、１フレーム前の重心位置Ｘ´＝Ｘ₄，Ｙ´＝Ｙ₄に更新される。
【０１１６】
ステップＳ１０３において、重心軌跡認識部３２は、Ｘ−Ｘ´の絶対値が所定の正の閾値より大きいのか否か、すなわち、現フレームのＸ座標とｎフレーム（いまの場合、１フレーム）前のＸ´座標を比較し、変化があったのか否かを判定し、Ｘ−Ｘ´の絶対値が所定の正の閾値より大きくないと判定された場合、すなわち、変化がないと判定された場合、ステップＳ１０４に進む。
【０１１７】
ステップＳ１０４におてい、重心軌跡認識部３２は、リングバッファが１周したのか否かを判定し、リングバッファが未だ１周していないと判定した場合、ステップＳ１０２に戻り、現フレームの重心位置Ｘ，Ｙと、１フレーム前の重心位置Ｘ´，Ｙ´の更新を行い（いまの場合、現フレームの重心位置Ｘ＝Ｘ₄，Ｙ＝Ｙ₄、１フレーム前の重心位置Ｘ´＝Ｘ₃，Ｙ´＝Ｙ₃に更新し）、それ以降の処理を繰り返す。すなわち、Ｘ座標が変化するまで、１フレームずつさかのぼった、現フレームの重心位置Ｘ，Ｙと１フレーム前の重心位置Ｘ´，Ｙ´に更新される。また、ステップＳ１０４において、リングバッファが１周したと判定された場合、ステップＳ１２８に進む。
【０１１８】
また、ステップＳ１０３において、Ｘ−Ｘ´の絶対値が所定の正の閾値より大きいと判定された場合、ステップＳ１０５に進み、重心軌跡認識部３２は、変化したときの座標Ｘ´，Ｙ´をＸｓ，Ｙｓにそれぞれ代入し、変数ｍａｘを０に、変数BigSkipをFALSEに初期設定する。ステップＳ１０６において、重心軌跡認識部３２は、Ｘ−Ｘ´が０より大きいのか否かを判定し、０より大きい場合（Ｘ−Ｘ´が正の値の場合）、ステップＳ１０７に進み、さらに、Ｘ−Ｘ´が所定の正の閾値より大きいのか否かを判定する。
【０１１９】
ステップＳ１０７において、Ｘ−Ｘ´が所定の正の閾値より大きいと判定された場合、ステップＳ１０８に進み、重心軌跡認識部３２は、変数BigSkipにTRUEを代入する。また、ステップＳ１０７において、Ｘ−Ｘ´が所定の正の閾値より大きくはないと判定された場合、ステップＳ１０９に進み、重心軌跡認識部３２は、さらに、Ｘ−Ｘ´が所定の負の閾値より小さいのか否かを判定し、所定の負の閾値より小さいと判定した場合、ステップＳ１２３に進み、所定の負の閾値より小さくはないと判定した場合、ステップＳ１１０に進む。
【０１２０】
ステップＳ１０８またはＳ１０９の処理の後、ステップＳ１１０において、重心軌跡認識部３２は、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値（いまの場合、０）より大きいのか否かを判定し、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値より大きいと判定した場合、ステップＳ１１１に進み、変数ｍａｘにＸ−Ｘｓの値を代入する。また、ステップＳ１１０において、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値より大きくはないと判定した場合、ステップＳ１１１の処理をスキップし、ステップＳ１１２に進む。
【０１２１】
ステップＳ１１０またはステップＳ１１１の処理の後、ステップＳ１１２において、重心軌跡認識部３２は、Ｙ−Ｙｓの絶対値が所定の閾値より小さいのか否かを判定し、所定の閾値より小さいと判定した場合、ステップＳ１１３に進み、Ｙ−Ｙｓの絶対値が所定の閾値より小さくはないと判定した場合、ステップＳ１２８に進む。
【０１２２】
ステップＳ１１３において、重心軌跡認識部３２は、リングバッファが１周したのか否かを判定し、リングバッファが未だ１周していないと判定した場合、ステップＳ１１４に進み、重心軌跡認識部３２は、現フレームの重心位置Ｘ，Ｙと、１フレーム前の重心位置Ｘ´，Ｙ´の更新を行い、ステップＳ１０７に戻る。いまの場合、１フレームずつさかのぼった、現フレームの重心位置Ｘ＝Ｘ₄，Ｙ＝Ｙ₄、１フレーム前の重心位置Ｘ´＝Ｘ₃，Ｙ´＝Ｙ₃に更新される。また、ステップＳ１１３において、リングバッファが１周したと判定された場合、ステップＳ１２３に進む。
【０１２３】
また、ステップＳ１０６において、Ｘ−Ｘ´が０より大きくはないと判定された場合（Ｘ−Ｘ´が負の値であると判定された場合）、ステップＳ１１５に進み、重心軌跡認識部２３は、Ｘ−Ｘ´が所定の負の閾値より小さいのか否かを判定し、所定の負の閾値より小さいと判定した場合、ステップＳ１１６に進む。ステップＳ１１６において、重心軌跡認識部３２は、変数BigSkipにTRUEを代入する。また、ステップＳ１１５において、Ｘ−Ｘ´が所定の負の閾値より小さくはないと判定された場合、ステップＳ１１７に進み、重心軌跡認識部３２は、さらに、Ｘ−Ｘ´が所定の正の閾値より大きいのか否かを判定し、所定の正の閾値より大きくはないと判定した場合、ステップＳ１１８に進み、所定の正の閾値より大きいと判定した場合、ステップＳ１２３に進む。
【０１２４】
ステップＳ１１６またはＳ１１７の処理の後、ステップＳ１１８において、重心軌跡認識部３２は、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値より大きいのか否かを判定し、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値より大きいと判定した場合、ステップＳ１１９に進み、変数ｍａｘにＸ−Ｘｓの値を代入する。また、ステップＳ１１８において、Ｘ−Ｘｓの絶対値が変数ｍａｘの絶対値より大きくはないと判定した場合、ステップＳ１１９の処理をスキップし、ステップＳ１２０に進む。
【０１２５】
ステップＳ１１８またはステップＳ１１９の処理の後、ステップＳ１２０において、重心軌跡認識部３２は、Ｙ−Ｙｓの絶対値が所定の閾値より小さいのか否かを判定し、所定の閾値より小さいと判定した場合、ステップＳ１２１に進み、Ｙ−Ｙｓの絶対値が所定の閾値より小さくはないと判定した場合、ステップＳ１２８に進む。
【０１２６】
ステップＳ１２１において、重心軌跡認識部３２は、リングバッファが１周したのか否かを判定し、リングバッファが未だ１周していないと判定した場合、ステップＳ１２２に進み、重心軌跡認識部３２は、現フレームの重心位置Ｘ，Ｙと、１フレーム前の重心位置Ｘ´，Ｙ´の更新を行い、ステップＳ１１５に戻る。いまの場合、１フレームずつさかのぼった、現フレームの重心位置Ｘ＝Ｘ₄，Ｙ＝Ｙ₄、１フレーム前の重心位置Ｘ´＝Ｘ₃，Ｙ´＝Ｙ₃に更新される。また、ステップＳ１２１において、リングバッファが１周したと判定された場合、ステップＳ１２３に進む。
【０１２７】
ステップＳ１０９，１１３，１１７、またはステップＳ１２１の処理の後、ステップＳ１２３において、重心軌跡認識部３２は、変数BigSkipがFALSEであるのか否かを判定し、変数BigSkipがFALSEであると判定した場合、ステップＳ１２４に進む、ステップＳ１２４において、重心軌跡認識部３２は、変数ｍａｘの値が所定の正の閾値より大きいのか否かを判定し、変数ｍａｘの値が所定の正の閾値より大きいと判定した場合、ステップＳ１２５に進み、右の振りがあったと認識し、図４のステップＳ１４にリターンする。
【０１２８】
また、ステップＳ１２４において、変数ｍａｘが所定の正の閾値より大きくはないと判定された場合、ステップＳ１２６に進み、重心軌跡認識部３２は、さらに、変数ｍａｘが所定の負の閾値より小さいのか否かを判定し、変数ｍａｘの値が所定の負の閾値より小さいと判定した場合、ステップＳ１２７に進み、左の振りがあったと認識し、図４のステップＳ１４にリターンする。
【０１２９】
ステップＳ１２３において、変数BigSkipがFALSEではないと判定された場合、ステップＳ１２６において、変数ｍａｘが所定の負の閾値より小さくはないと判定された場合、また、ステップＳ１０４，１１２，またはステップＳ１２０の処理の後、ステップＳ１２８において、重心軌跡認識部３２は、手の振りはなかったと認識し、図４のステップＳ１４にリターンする。
【０１３０】
また、他の例の振り動作認識処理について、図１７のフローチャートを参照して説明する。ステップＳ１４１において、重心軌跡認識部３２は、curr_gesにRight，Left、またはNoneを代入する。すなわち、図１５のステップＳ１２５，Ｓ１２７、または、Ｓ１２８での処理結果（右振り、左振り、または手振りなし）を代入する。ステップＳ１４２において、重心軌跡認識部３２は、現フレームの時刻と基準時刻とを比較し、現フレームの経過時間を算出する。ステップＳ１４３において、重心軌跡認識部３２は、現フレームが右方向または左方向の動きがあったのか否かを判定し、右方向または左方向の動きがあったと判定した場合、ステップＳ１４４に進む。ステップＳ１４４において、重心軌跡認識部３２は、経過時間を算出するための基準時刻を現フレームの時刻に更新する。
【０１３１】
また、ステップＳ１４３において、右方向または左方向の動きがなかったと判定された場合、ステップＳ１４４の処理をスキップし、ステップＳ１４５に進む。ステップＳ１４５において、重心軌跡認識部３２は、ステップＳ１４２の処理で算出された現フレームの経過時間が所定の閾値より長いのか否かを判定し、現フレームの経過時間が所定の閾値より長いと判定した場合、ステップＳ１４６に進み、変数prev_gesに保存されているジェスチャ（右振りまたは左振り）を出力する。
【０１３２】
ステップＳ１４５において、現フレームの経過時間が所定の閾値より長くはないと判定された場合、ステップＳ１４７に進み、重心軌跡認識部３２は、現フレームのジェスチャ（右振りまたは左振り）であるcurr_gesを出力する。ステップＳ１４８において、重心軌跡認識部３２は、ステップＳ１４７の処理で出力された現フレームのジェスチャであるcurr_gesをprev_gesに代入（保存）する。
【０１３３】
ステップＳ１４９において、重心軌跡認識部３２は、ステップＳ１４６またはＳ１４７の処理で出力されたジェスチャから、振り動作を認識し、図４のステップＳ１４にリターンする。
【０１３４】
上述した処理は、短時間で一方の動き（例えば、左方向の動き）から他方の動き（例えば、右方向の動き）に変化した場合、一方（左方向）の動きを排除して、他方（右方向）の動きを認識するので、処理を高速化することができる。なお、図１４および図１５、または図１７による処理と同様にして上下方向の手振りを認識することもできる。
【０１３５】
図４に戻って、ステップＳ１４において、表示・出力装置４は、画像処理装置３の握り動作判定部６１より供給された認識結果（握りまたは開き動作）、および、重心軌跡認識部３２より供給された認識結果（身振りや手振り）を表示する。ステップＳ１５において、画像処理装置３は、ユーザ１の動作が終了したのか否かを判定し、未だ、動作が終了していないと判定した場合、ステップＳ１に戻り、上述した処理を繰り返す。そして、ステップＳ１５において、ユーザ１の動作が終了したと判定されると、処理は終了される。
【０１３６】
以上の握り動作認識処理の順序が明らかなようにまとめると、図１８に示されるようになる。そこで、次に、図１８を参照して、手の握り動作認識処理について説明する。なお、簡単のため、動きブロックの領域は、領域７２のみとされ、動きブロックの領域が複数存在した場合に行われる評価値の最も小さい領域を選択するための処理はここでは省略する。
【０１３７】
ビデオカメラ２は、ユーザ１の手７１を撮像し（図１８（Ａ））、撮像された画像データを輝度データに変換し、画像記憶部２１に記憶させる。領域分割部２２のフレーム間差分算出部４１は、ビデオカメラ２で変換された現フレームの画像データの輝度データと画像記憶部２１に記憶されている１フレーム前の画像データの輝度データとの間のフレーム間差分値を算出する。閾値処理部４２は、フレーム間差分値のうち、所定の閾値以上のフレーム間差分値がある画素を１とし、所定の閾値に満たないフレーム間差分値の画素を０として２値化する（図１８（Ｂ））。
【０１３８】
ラベリング部４６は、２値化された画像の動きブロックの領域７２をラベリング処理する。重心検出部５１は、ラベリングされた領域７２内の重心を検出する（図１８（Ｃ））。
【０１３９】
エッジ検出部５７は、領域７２の垂直方向および水平方向のエッジ情報をそれぞれ検出する。すなわち、エッジ検出部５７は、領域７２の重心Ｃが矩形領域９１の中心になるように設定して矩形領域９１を３等分したtop，middle，bottomの３領域内の垂直方向のエッジピクセルの数をカウントするとともに（図１８（Ｅ））、水平方向のエッジピクセルの数をカウントする（図１８（Ｆ））。
【０１４０】
エッジ検出部５７は、カウントされたエッジピクセル数に基づいて、上記式（７）に従って、特徴ベクトルｆを算出する。DP値算出部６２は、連続DPの手法を用いて、算出された特徴ベクトルｆから連続DP値を算出し、参照パターンにマッチした動作を握り動作判定部６１に出力する。これにより、握り動作判定部６１は、手の握りまたは開き動作を認識することができる。
【０１４１】
以上のように、ビデオカメラで撮像した画像データから、ユーザの手の握りまたは開き動作を認識することができるので、マウスやキーボードなどの入力装置を用いることなく、離れた場所から容易にコンピュータ等を操作することができる。
【０１４２】
また、特別なデバイスや複数台のビデオカメラを使用していないため、低コストなシステムを実現することができる。
【０１４３】
また、以上においては、手の握り・開き動作を認識することは勿論、手の移動（軌跡）動作および手の振り動作も認識することができるので、例えば、コンピュータに対応付けた場合、手を移動（マウス移動）、手を握る（アイコン選択）、手を握ったまま移動（アイコンを選択したままドラッグ）、手を開く（アイコン選択解除）という操作を行ったり、左方向の振り（マウス左ボタンクリック）、右方向の振り（マウス右ボタンクリック）という操作を行ったりすることができる。従って、ユーザは、マウスを用いなくても、直感的な操作で、画面にアプリケーションを起動させたり、画面に表示されているウィンドウを移動させたりすることができる。
【０１４４】
これにより、例えば、ディスプレイに表示されているファイルを、手を握ることにより選択し、手を移動させることによりドラッグし、所定の場所（例えば、ハードディスクドライブなど）で手を開くことにより、所望のファイルの移動やコピーなどを直感的な動作で行うことができる。
【０１４５】
上述した一連の処理は、ハードウェアにより実行させることもできるが、ソフトウェアにより実行させることもできる。一連の処理をソフトウェアにより実行させる場合には、そのソフトウェアを構成するプログラムが、専用のハードウェアに組み込まれているコンピュータ、または、各種のプログラムをインストールすることで、各種の機能を実行することが可能な、例えば汎用のパーソナルコンピュータなどに、記録媒体からインストールされる。
【０１４６】
この記録媒体は、コンピュータとは別に、ユーザにプログラムを提供するために配布される、プログラムが記録されている磁気ディスク（フロッピディスクを含む）、光ディスク（CD-ROM（Compact Disk-Read Only Memory），DVD（Digital Versatile Disk）を含む、光磁気ディスク（MD（Mini-Disk）を含む）、若しくは半導体メモリなどよりなるパッケージメディアにより構成されるだけでなく、コンピュータに予め組み込まれた状態でユーザに提供される、プログラムが記録されているROMや、ハードディスクなどで構成される。
【０１４７】
なお、本明細書において、記録媒体に記録されるプログラムを記述するステップは、記載された順序に沿って時系列的に行われる処理はもちろん、必ずしも時系列的に処理されなくとも、並列的あるいは個別に実行される処理をも含むものである。
【０１４８】
また、本明細書において、システムとは、複数の装置により構成される装置全体を表すものである。
【０１４９】
【発明の効果】
以上のように、本発明の画像処理装置、画像処理方法、および記録媒体に記録されているプログラムによれば、手を撮像し、撮像された画像を輝度情報に変換し、変換された現フレームの輝度情報と、１フレーム前の輝度情報の差に基づいて、動き領域を検出し、検出された動き領域が手であるかを評価する評価値を算出し、算出された評価値に基づいて、複数の動き領域のうちの所定の領域を手の領域として選択し、選択された手の領域における現フレームの輝度情報と、１フレーム前の輝度情報の差によってできた画像のエッジ情報を抽出し、抽出されたエッジ情報の特徴ベクトルを特徴量として算出し、算出された特徴量に基づいて、ＤＰマッチングの手法を用いてＤＰ値を算出し、算出されたＤＰ値が所定値より小さいか否かを判定し、ＤＰ値が所定値より小さいと判定された場合、手の握り動作を認識するようにしたので、低コストに、かつ、高精度に、ユーザの手の握りおよび開き動作を画像処理により認識することができる。
【図面の簡単な説明】
【図１】本発明を適用した画像処理システムの構成例を示す図である。
【図２】図１の画像処理装置の構成を示すブロック図である。
【図３】図２の画像処理部およびジェスチャ認識部の詳細な構成を示すブロック図である。
【図４】手の握り・振り動作認識処理を説明するフローチャートである。
【図５】図４のステップＳ２の領域分割処理を説明するフローチャートである。
【図６】領域分割処理を説明するための図である。
【図７】重心を算出する動作を説明するための図である。
【図８】図４のステップＳ９の動き方向検出処理を説明するフローチャートである。
【図９】動き方向を検出するフレームを説明するための図である。
【図１０】動き方向を示す画素値を設定するための対応図である。
【図１１】エッジ情報検出処理を説明するための図である。
【図１２】エッジ情報検出処理に利用されるフィルタを示す図である。
【図１３】 DPマッチングを説明するための図である。
【図１４】図４のステップＳ１３の振り動作認識処理を説明するフローチャートである。
【図１５】図１４に続くフローチャートである。
【図１６】リングバッファを説明する図である。
【図１７】図４のステップＳ１３の他の例の振り動作認識処理を説明するフローチャートである。
【図１８】握り動作認識処理を説明するための図である。
【符号の説明】
２ ビデオカメラ， ３ 画像処理装置， ４ 表示・出力装置， １１ 画像処理部， １２ ジェスチャ認識部， ２１ 画像記憶部, ２２ 領域分割部， ２３ 領域処理部， ３１ 握り動作認識部, ３２ 重心軌跡認識部, ４１ フレーム間差分算出部， ４２ 閾値処理部, ４３ ブロック内カウント部, ４４ 閾値処理部, ４５ 周辺探索部, ４６ ラベリング部, ５１ 重心検出部, ５２ 外接枠検出部, ５３ 面積検出部, ５４ 領域評価部, ５５ 領域記憶部, ６１ 握り動作判定部, ６２ DP値算出部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, and a recording medium, and more particularly, to an image processing apparatus and method, and a recording medium that can recognize a user's hand gripping and opening operation by image processing.
[0002]
[Prior art]
Conventionally, several methods for interfacing between a person and a computer by incorporating user gestures and gestures as computer graphics have been proposed.
[0003]
For example, Japanese Patent Laid-Open No. 11-195140 discloses a technique for editing three-dimensional computer graphics data by operating a computer with gestures or hand gestures using a data glove.
[0004]
Further, for example, in Japanese Patent Laid-Open No. 9-311759, a user gestures using a penlight that lights and blinks in a predetermined color, and the penlight captures and recognizes an image with a camera. Techniques for interfacing are disclosed.
[0005]
Furthermore, for example, Japanese Patent Laid-Open No. 9-102046 discloses a technique for performing an interface between a computer and a person by estimating and recognizing a hand shape from a silhouette image captured by a camera.
[0006]
Furthermore, for example, Japanese Patent Laid-Open No. 8-212327 discloses a computer and a person by recognizing gestures using an orientation one-dimensional histogram and a two-dimensional spatio-temporal orientation histogram of a hand image captured by a camera. A technique for performing the interface is disclosed.
[0007]
Furthermore, for example, Japanese Patent Application Laid-Open No. 9-179888 discloses a technique for performing an interface between a computer and a person by recognizing the three-dimensional posture and gesture of a hand using two cameras. Yes.
[0008]
Furthermore, for example, Japanese Patent Laid-Open No. 11-162151 discloses a technique for performing an interface between a computer and a person by recognizing a user's gesture using a continuous DP (Continuous Dynamic Programming) technique. Yes.
[0009]
[Problems to be solved by the invention]
In the case of a method using a data glove (Japanese Patent Laid-Open No. 11-195140), the shape of the hand and the movement of the hand can be recognized, but the data glove is inconvenient to attach and detach, and the data glove and the computer are connected. There is a problem that the cable to be troublesome.
[0010]
Therefore, in consideration of user convenience, a method using a penlight (Japanese Patent Laid-Open No. 9-311759) has been proposed. However, the user must have a penlight, and the trouble for the user still remains. Further, since a special device (penlight) is used, there is a problem that lacks versatility.
[0011]
In order to reduce the troublesomeness for the user, methods for recognizing the shape of the hand imaged by the camera (Japanese Patent Laid-Open Nos. 9-102046 and 8-212327) have been proposed. There is a possibility that the shape of the hand may not be accurately detected due to a change or the like, and there is a problem of erroneous recognition.
[0012]
In order to prevent erroneous recognition, a method for recognizing an image using a plurality of cameras has been proposed (Japanese Patent Laid-Open No. 9-179988), but there is a problem that cannot be realized at low cost.
[0013]
Furthermore, although a gesture recognition method using continuous DP has been proposed (Japanese Patent Laid-Open No. 11-162151), there is a problem that it is impossible to recognize a hand movement such as grasping (gripping).
[0014]
Therefore, Japanese Patent Application Laid-Open No. 11-296673 discloses that the position pointed to by the user is specified. However, it is possible to recognize an operation such as grasping (holding) with a hand and selecting with a simple method. There was a problem that was difficult.
[0015]
The present invention has been made in view of such a situation, and can recognize a user's gesture at a low cost and with high accuracy, and can recognize a grasping operation by a hand by image processing. To do.
[0016]
[Means for Solving the Problems]
  An image processing apparatus according to the present invention includes an imaging unit that captures an image of a hand, a conversion unit that converts an image captured by the imaging unit into luminance information, and a conversion unit that converts the image.Of the current frameLuminance informationAnd the difference in luminance information of the previous frameBased on the movement area detection means for detecting the movement area, the evaluation value calculation means for calculating the evaluation value for evaluating whether the movement area detected by the movement area detection means is a hand, and the evaluation value calculation means Multiple motion regions based on the evaluated valuesA predetermined area ofHand areaAsSelection means to select and the hand area selected by the selection meansAn image formed by the difference between the luminance information of the current frame and the luminance information of the previous frameBased on the feature information calculated by the edge information extraction means for extracting the edge information, the feature quantity calculation means for calculating the feature vector of the edge information extracted by the edge information extraction means as the feature quantity, and the feature quantity calculated by the feature quantity calculation means The DP value calculating means for calculating the DP value using the DP matching method, the determining means for determining whether the DP value calculated by the DP value calculating means is smaller than a predetermined value, and the determining means And a recognition means for recognizing a hand gripping operation when it is determined that is smaller than a predetermined value.
[0017]
  The motion region detection means is configured to detect a difference between luminance information of the current frame and luminance information of the previous frameButIt is possible to detect a motion region by determining whether or not it is equal to or greater than a predetermined threshold.
[0018]
The image processing apparatus according to the present invention may further include a centroid calculating unit that calculates the centroid of the motion region detected by the motion region detecting unit, and the edge information extracting unit includes the motion calculated by the centroid calculating unit. A predetermined rectangular area centered on the center of gravity of the area can be divided into three equal parts in the vertical direction, and edge information in the vertical and horizontal directions can be extracted in each area.
[0019]
  The image processing apparatus of the present inventionMotion direction detection means for detecting the motion direction of the motion region selected by the selection meansCan be further provided,The recognizing means recognizes the movement of the hand detected by the movement direction detecting means as a mouse moving operation, recognizes the movement of holding the hand as an operation of selecting an icon by the mouse, and moves the movement while holding the hand to the icon. Is recognized as a dragging operation with the button selected, and the movement to open the hand is recognized as the operation to deselect the icon, and the movement of shaking the hand to the left is recognized as an operation of clicking the left mouse button, and the hand is moved to the right Recognizing a movement in the direction as a click of the right mouse buttonTo be able to.
[0021]
  The image processing method of the present invention includes an imaging step for imaging a hand, a conversion step for converting an image captured by the processing of the imaging step into luminance information, and conversion by the processing of the conversion step.Of the current frameLuminance informationAnd the difference in luminance information of the previous frameA motion region detection step for detecting a motion region, an evaluation value calculation step for calculating an evaluation value for evaluating whether the motion region detected by the processing of the motion region detection step is a hand, and an evaluation value calculation step Based on the evaluation value calculated by the processing ofA predetermined area ofHand areaAsSelection step to select and the hand region selected by the selection step processAn image formed by the difference between the luminance information of the current frame and the luminance information of the previous frameEdge information extracting step for extracting edge information, a feature amount calculating step for calculating a feature vector of edge information extracted by the processing of the edge information extracting step as a feature amount, and a feature calculated by the processing of the feature amount calculating step A DP value calculating step for calculating a DP value using a DP matching technique based on the amount; a determining step for determining whether the DP value calculated by the processing of the DP value calculating step is smaller than a predetermined value; A recognition step of recognizing a hand gripping operation when the DP value is determined to be smaller than a predetermined value by the processing of the determination step.
[0022]
  The program recorded on the recording medium of the present invention is converted by the imaging step of imaging the hand, the conversion step of converting the image captured by the processing of the imaging step into luminance information, and the processing of the conversion stepOf the current frameLuminance informationAnd the difference in luminance information of the previous frameA motion region detection step for detecting a motion region, an evaluation value calculation step for calculating an evaluation value for evaluating whether the motion region detected by the processing of the motion region detection step is a hand, and an evaluation value calculation step Based on the evaluation value calculated by the processing ofA predetermined area ofHand areaAsSelection step to select and the hand region selected by the selection step processAn image formed by the difference between the luminance information of the current frame and the luminance information of the previous frameEdge information extracting step for extracting edge information, a feature amount calculating step for calculating a feature vector of edge information extracted by the processing of the edge information extracting step as a feature amount, and a feature calculated by the processing of the feature amount calculating step A DP value calculating step for calculating a DP value using a DP matching technique based on the amount; a determining step for determining whether the DP value calculated by the processing of the DP value calculating step is smaller than a predetermined value; A recognition step of recognizing a hand gripping operation when the DP value is determined to be smaller than a predetermined value by the processing of the determination step.
[0023]
  In the image processing apparatus, the image processing method, and the program recorded on the recording medium of the present invention, the hand is imaged, and the captured image is converted into luminance information and converted.Of the current frameLuminance informationAnd the difference in luminance information of the previous frameAnd a motion area is detected, an evaluation value for evaluating whether the detected motion area is a hand is calculated, and a plurality of motion areas are calculated based on the calculated evaluation value.A predetermined area ofHand areaAsSelected and selected hand areaAn image formed by the difference between the luminance information of the current frame and the luminance information of the previous frameEdge information is extracted, a feature vector of the extracted edge information is calculated as a feature amount, a DP value is calculated using a DP matching technique based on the calculated feature amount, and the calculated DP value is When it is determined whether or not the DP value is smaller than the predetermined value, it is recognized that the hand gripping action is recognized.
[0024]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram showing an embodiment of an image processing system according to the present invention.
[0025]
The video camera 2 captures the gestures, hand gestures, and hand gripping actions of the user 1, converts the captured image data into luminance data (Y data) used in processing to be described later, and converts the image data into the image processing device 3 To supply. The image processing device 3 performs predetermined processing such as image processing and gesture recognition processing on the image data supplied from the video camera 2 and outputs the processed data to the display / output device 4. The display / output device 4 starts an application on the screen or is displayed on the screen based on the image data supplied from the image processing device 3 and corresponding to the gesture, hand gesture, or hand gripping motion of the user 1. Move the current window.
[0026]
FIG. 2 is a block diagram illustrating a configuration example of the image processing apparatus 3.
[0027]
The image processing unit 11 performs inter-frame difference processing, centroid detection processing, motion detection processing, and the like on the image data supplied from the video camera 2, and supplies the image processing result to the gesture recognition unit 12. The gesture recognition unit 12 performs a gesture recognition process based on the image processing result supplied from the image processing unit 11 and supplies the recognition result to the display / output device 4.
[0028]
FIG. 3 is a block diagram illustrating a detailed configuration example of the image processing unit 11 and the gesture recognition unit 12.
[0029]
The image processing unit 11 includes an image storage unit 21, a region dividing unit 22, and a region processing unit 23. The gesture recognition unit 12 includes a gripping movement recognition unit 31 and a gravity center locus recognition unit 32. Further, the area dividing unit 22 of the image processing unit 11 includes an inter-frame difference unit 41, a threshold processing unit 42, an intra-block counting unit 43, a threshold processing unit 44, a peripheral search unit 45, and a labeling unit 46, and an image The region processing unit 23 of the processing unit 11 includes a centroid detection unit 51, a circumscribed frame detection unit 52, an area detection unit 53, a region evaluation unit 54, a region storage unit 55, a motion detection unit 56, and an edge detection unit 57. The The gripping motion recognition unit 31 of the gesture recognition unit 12 includes a gripping motion determination unit 61 and a DP value calculation unit 62.
[0030]
The image storage unit 21 stores the luminance data of the image data supplied from the video camera 2 for each frame.
[0031]
The inter-frame difference calculation unit 41 reads the luminance data (Y data) of the image data of the previous frame stored in the image storage unit 21, and the luminance data of the previous frame and the current frame supplied from the video camera 2. The luminance difference (inter-frame difference value) is calculated for each pixel from the luminance data of the image data, and the calculation result is supplied to the threshold processing unit 42.
[0032]
Based on the calculation result (interframe difference value) supplied from the interframe difference calculation unit 41, the threshold processing unit 42 sets a pixel having a difference value equal to or greater than a predetermined threshold to 1, and a difference value that does not satisfy the predetermined threshold These pixels are binarized as 0, supplied to the in-block count unit 43, and supplied to the edge detection unit 57 of the area processing unit 23.
[0033]
The intra-block counting unit 43 divides the binarized image supplied from the threshold processing unit 42 into predetermined blocks, counts the pixels whose pixel value is 1 in each block, and thresholds the count number. Supplied to the unit 44.
[0034]
The threshold processing unit 44 recognizes a block having a count number equal to or greater than a predetermined threshold as a motion block based on the count number for each block supplied from the intra-block counting unit 43, and sends the recognition result to the peripheral search unit 45. Supply.
[0035]
Based on the recognition result supplied from the threshold processing unit 44, the peripheral search unit 45 further counts the number of motion blocks in a predetermined area centered on a block that has not been recognized as a motion block. If the count number is equal to or greater than a predetermined threshold, the peripheral search unit 45 newly recognizes the block as a motion block and supplies the recognition result to the labeling unit 46.
[0036]
The labeling unit 46 performs a labeling process based on the recognition result (block recognized as a motion block) supplied from the peripheral search unit 45 and supplies the processing result to the centroid detection unit 51 of the region processing unit 23.
[0037]
The center-of-gravity detection unit 51 detects the center of gravity in each labeled region based on the labeling processing result supplied from the labeling unit 46, and supplies the detection result to the circumscribed frame detection unit 52.
[0038]
The circumscribed frame detection unit 52 detects a rectangle (circumscribed frame) surrounding each labeled region from the outside based on the labeling processing result supplied from the center of gravity detection unit 51, and supplies the detection result to the area detection unit 53. .
[0039]
The area detection unit 53 calculates the area by counting the number of pixels of the motion block included in the circumscribing frame based on the detection result (the circumscribing frame) supplied from the circumscribing frame detection unit 52, and calculates the calculation result. This is supplied to the area evaluation unit 54. The area detection unit 53 also calculates the aspect ratio (aspect ratio) of the circumscribing frame based on the detection result supplied from the circumscribing frame detection unit 52, and also supplies the calculation result to the region evaluation unit 54.
[0040]
The region evaluation unit 54 calculates an evaluation value based on the center of gravity, the area, and the aspect ratio of the circumscribed frame supplied from the area detection unit 53, and selects the region having the smallest evaluation value. The selection result is supplied to the motion detection unit 56, the edge detection unit 57, and the barycentric locus recognition unit 32, respectively.
[0041]
The area storage unit 55 stores an area having the smallest evaluation value selected by the area evaluation unit 54.
[0042]
The motion detection unit 56 calculates the inter-frame difference value and the spatial direction difference value using the pixel one frame before supplied from the image storage unit 21 and the current pixel supplied from the video camera 2, and the calculation result The movement direction is detected, and the detection result is supplied to the gripping motion determination unit 61 of the gripping motion recognition unit 31.
[0043]
Based on the binarized inter-frame difference value supplied from the threshold processing unit 42 of the region dividing unit 22 and the region selection result supplied from the region evaluating unit 54, the edge detecting unit 57 performs a gesture recognition unit described later. The twelve gripping motion recognition units 31 detect edge information (feature vectors) required when calculating the DP value.
[0044]
The gripping motion determination unit 61 recognizes the hand gripping and opening motion based on the detection result (movement direction) supplied from the motion detection unit 56 and the DP value supplied from the DP value calculation unit 62 described later. The recognition result is supplied to the display / output device 4.
[0045]
The DP value calculation unit 62 calculates the DP value using the continuous DP method based on the edge information (feature vector) supplied from the edge detection unit 57. Note that the continuous DP (Continuous Dynamic Programming) method uses an algorithm that calculates the similarity between a reference pattern prepared in advance and a predetermined pattern. For example, Takahashi et al .: Gesture animation Spotting recognition of images, described in D-II vol. J77-D-II no.8 pp.1552-1561 (1994). This is a technique that allows matching while efficiently matching even if the lengths of the reference pattern R and the pattern Q to be recognized are different. Moreover, since it is not necessary to determine the start point and end point of the recognition target pattern, this is one of the effective methods for recognizing the time series pattern.
[0046]
The center-of-gravity locus recognition unit 32 calculates the distance between the center of gravity of the current frame region having the smallest evaluation value supplied from the region evaluation unit 54 of the region processing unit 23 and the center of gravity of the region selected one frame before, The trajectory (movement) of the center of gravity is recognized from the calculation result (that is, the hand shaking motion is recognized), and the recognition result is supplied to the display / output device 4.
[0047]
Next, with reference to the flowchart of FIG. 4, the hand grip / swing motion recognition process executed by the image processing apparatus 3 will be described.
[0048]
In step S <b> 1, the video camera 2 captures the gesture or hand gesture of the user 1, converts the captured image data into luminance data (Y data), and stores it in the image storage unit 21. For example, when the output of the video camera 2 is RGB color data, the video camera 2 calculates Y data in each pixel from the RGB color data of the captured image data according to the following equation (1).
Y = 0.299R + 0.587G + 0.114B (1)
[0049]
In step S2, the area dividing unit 22 performs area division on the image captured in the process of step S1.
[0050]
Here, the region division processing will be described in detail with reference to the flowchart of FIG.
[0051]
In step S <b> 31, the inter-frame difference calculation unit 41 of the region dividing unit 22 and the luminance data (Y data) of the image data of the current frame supplied from the video camera 2 and the previous frame stored by the image storage unit 21. An inter-frame difference value (luminance difference) between the luminance data of the image data is calculated. The threshold processing unit 42 sets a pixel having an inter-frame difference value equal to or greater than a predetermined threshold among the inter-frame difference values calculated by the inter-frame difference calculation unit 41 to the inter-frame difference value that is less than the predetermined threshold. The pixel is binarized with 0. Thereby, a binary image as shown in FIG. 6A is generated.
[0052]
In the case of the example in FIG. 6A, a pixel having an inter-frame difference value larger than a predetermined threshold (that is, a pixel having a pixel value of 1) is represented in black, and a pixel having an inter-frame difference value smaller than the predetermined threshold (that is, , Pixels with a pixel value of 0) are represented in white.
[0053]
In step S32, the intra-block counting unit 43 divides the binary image generated by the process of step S31 into blocks of Bw × Bh size as shown in FIG. 6B, for example. For each block, a pixel having a pixel value of 1 in the block (that is, a pixel displayed in black) is counted.
[0054]
In step S33, the threshold processing unit 44 recognizes, as a motion block, a block whose count is greater than a predetermined threshold based on the count for each block counted in the process of step S32. Thereby, for example, a motion block as shown in FIG. 6C is detected. In the case of the example in FIG. 6C, the shaded block is a motion block. A block whose count is smaller than a predetermined threshold is recognized as noise.
[0055]
In step S34, the peripheral search unit 45 is in a predetermined area centered on a block that is not recognized as a motion block in the process of step S33 (that is, a white block that is not shaded in FIG. 6C). Count the number of motion blocks. For example, as illustrated in FIG. 6C, the peripheral search unit 45 counts the number of motion blocks in a 3 × 3 region 82 centered on a block 81 that has not been recognized as a motion block.
[0056]
In step S35, the peripheral search unit 45 newly recognizes a block whose count number counted in step S34 is larger than a predetermined threshold value as a motion block. That is, as shown in FIG. 6D, the number of motion blocks (that is, blocks shown in black) in the 3 × 3 region 82 centered on the block 81 that is not recognized as a motion block is For example, if the threshold is set to a majority (ie, 5) of the number of blocks in the region 82 (9 in this case), this block 81 is recognized as a new motion block. The
[0057]
Such a process is performed for all blocks that are not recognized as motion blocks in the process of step S33. As a result, as shown in FIG. 6E, the blocks 81, 83, and 84 that are not recognized as motion blocks in the process of step S33 are recognized as new motion blocks.
[0058]
In step S36, the labeling unit 46 performs a labeling process for each region based on the motion block recognized in the processes in steps S33 and S35. As a result, as shown in FIG. 6F, two areas of an area 85 and an area 86 are detected.
[0059]
Returning to step S3 of FIG. 4, the center-of-gravity detection unit 51 detects (calculates) the center of gravity in each region labeled in the process of step S2.
[0060]
Here, a method of calculating the center of gravity in the labeled region will be described with reference to FIG.
[0061]
In the case of the example in FIG. 7A, the average position is calculated using pixels in the circumscribed frame of the labeled area, and this is used as the center of gravity C.
[0062]
In the case of the example of FIG. 7B, a weighted weighted average is calculated in the upper part of the screen and is set as the center of gravity C. That is, the upper left coordinate of the circumscribed frame of the labeled area (X_ul, Y_ul), Lower right coordinates (X_br, Y_br), The weight corresponding to a certain pixel (x, y) is calculated according to the following equation (2), the calculation result is substituted into the following equation (3), and a weighted weighted average is calculated. In this example, when a hand is put out with respect to the video camera 2, the probability that the fingertip is at the top of the screen is high, and weighting is performed at the top of the screen.
(Y_br-Y) / (Y_br-Y_ul(2)
[0063]
[Expression 1]

[0064]
In the case of the example of FIG. 7C, in the first time, as shown in the example of FIG. 7A, the average position C1 is calculated using the pixels in the circumscribed frame of the labeled area. In the second time, the average position C2 is calculated using pixels located above the screen from the average position C1. Similarly, the average position after the third time is sequentially calculated repeatedly. The n-th average position Cn is defined as the final center of gravity C.
[0065]
When the center of gravity of the labeled region (in this case, the regions 85 and 86) is detected using any of the above-described center-of-gravity calculation methods, the process proceeds to step S4, and the circumscribed frame detection unit of the region processing unit 23 52 detects a circumscribed frame (rectangle) surrounding each region (regions 85 and 86 in this case) labeled in the process of step S2 from the outside.
[0066]
In step S5, the area detection unit 53 calculates (detects) the area by counting the number of pixels of the motion block included in the circumscribed frame detected by the process of step S4. That is, in the area 85 of FIG. 6F, there are 13 motion blocks (that is, blocks shown in black). For example, if one block is composed of 16 pixels, the number of motion pixels Is counted as 208, and the area is calculated from the counted number of pixels.
[0067]
In step S6, the area detection unit 53 calculates the aspect ratio (aspect ratio) of the circumscribed frame detected by the process in step S4. That is, in the case of the area 85 in FIG. 6F, the number of vertical blocks is five, the number of horizontal blocks is four, and the aspect ratio is calculated as 1.25 (= 5/4). .
[0068]
In step S7, the region evaluation unit 54 calculates the center of gravity of the labeled regions (in this case, the regions 85 and 86) detected in the process of step S3, and the motion block of each region calculated in the process of step S5. Based on the area and the aspect ratio of the circumscribed frame of the region calculated in step S6, the evaluation value is calculated according to the following equation (4).
Φ (i) = α_c｜ C_i-C_p| + Α_a｜ A_i-A_p| + Α_s｜ S_i-S_p｜ −α_mNm_i
... (4)
[0069]
In the above formula (4), C_iIs the center of gravity of region i, A_iIs the aspect ratio of region i, S_iIs the area of region i, Nm_iRepresents the number of moving pixels in region i, and C_p, A_p, S_pRepresents the barycentric position, aspect ratio, and area of the region p selected in the previous frame, and α_c, Α_a, Α_s, Α_mAre respective weighting coefficients indicating the importance of the position of the center of gravity, the aspect ratio, the area, and the number of moving pixels. For example, the weighting factor includes α_c= 10.0, α_a= 10.2, α_sA value such as = 0.2 is used.
[0070]
In step S8, the region evaluation unit 54 selects a region (for example, the region 85) having the smallest evaluation value among the evaluation values calculated in the process of step S7.
[0071]
In step S9, the motion detection unit 56 detects the motion direction of the region labeled in the process of step S2 (in this case, the regions 85 and 86).
[0072]
Here, the movement direction detection process will be described with reference to the flowchart of FIG. Note that, as shown in FIG. 9, a description will be made assuming that the current frame region composed of 3 × 3 pixels is compared with the previous frame region to detect the movement direction of the current frame region.
[0073]
In step S41, the motion detector 56 substitutes 0 for variables h and v indicating the direction of motion. In step S42, the motion detection unit 56 determines whether or not motion in both the vertical and horizontal directions has been detected. If it is determined that motion in both the vertical and horizontal directions has not yet been detected, the process proceeds to step S43. Then, it is determined whether or not the luminance difference between the center pixel P (i, j) of the current frame and the center pixel P ′ (i, j) of the previous frame is larger than a predetermined positive threshold value. Note that i represents the y coordinate, and j represents the x coordinate.
[0074]
If it is determined in step S43 that P (i, j) −P ′ (i, j) is greater than a predetermined positive threshold value, the process proceeds to step S44, and the motion detection unit 56 detects lateral motion. In other words, if it is determined whether or not the vertical movement has been detected, and it is determined that the horizontal movement is detected (when it is determined that the vertical movement has been detected), step S45. Proceed to In step S45, the motion detection unit 56 determines whether or not the luminance difference between the center pixel P (i, j) and the pixel P (i, j + 1) is larger than a predetermined threshold, and P (i, j) When it is determined that −P (i, j + 1) is larger than the predetermined threshold value, the process proceeds to step S46, 1 is substituted into the variable h, and the process returns to step S42.
[0075]
If it is determined in step S45 that P (i, j) −P (i, j + 1) is not greater than the predetermined threshold value, the process proceeds to step S47, and the motion detection unit 56 further includes the center pixel P ( i, j) and the pixel P (i, j-1) are determined whether or not the luminance difference is larger than a predetermined threshold, and P (i, j) -P (i, j-1) is larger than the predetermined threshold. When it determines with it being large, it progresses to step S48, -1 is substituted to the variable h, and it returns to step S42. If it is determined in step S47 that P (i, j) -P (i, j-1) is not greater than the predetermined threshold value, the process returns to step S42.
[0076]
On the other hand, if it is determined in step S44 that the vertical movement is detected instead of the horizontal movement, the process proceeds to step S49, and the motion detection unit 56 determines that the center pixel P (i, j) and the pixel P (i + It is determined whether or not the luminance difference of (1, j) is larger than a predetermined threshold value. If it is determined that P (i, j) −P (i + 1, j) is larger than the predetermined threshold value, the process proceeds to step S50. Then, 1 is substituted into the variable v, and the process returns to step S42.
[0077]
If it is determined in step S49 that P (i, j) −P (i + 1, j) is not greater than the predetermined threshold value, the process proceeds to step S51, and the motion detection unit 56 further includes the center pixel P ( i, j) and the pixel P (i−1, j) are determined whether or not the luminance difference is larger than a predetermined threshold value, and P (i, j) −P (i−1, j) is larger than the predetermined threshold value. When it determines with it being large, it progresses to step S52, -1 is substituted to the variable v, and it returns to step S42. If it is determined in step S51 that P (i, j) -P (i-1, j) is not greater than the predetermined threshold value, the process returns to step S42.
[0078]
On the other hand, if it is determined in step S43 that P (i, j) −P ′ (i, j) is not greater than the predetermined positive threshold value, the process proceeds to step S53, and the motion detector 56 further It is determined whether or not the luminance difference between the center pixel P (i, j) of the frame and the center pixel P ′ (i, j) one frame before is smaller than a predetermined negative threshold.
[0079]
If it is determined in step S53 that P (i, j) −P ′ (i, j) is smaller than the predetermined negative threshold value, the process proceeds to step S54, and the motion detection unit 56 detects lateral motion. In other words, when it is determined whether or not the vertical movement has been detected, and it is determined that the horizontal movement is detected (when it is determined that the vertical movement has been detected), step S55. Proceed to In step S55, the motion detection unit 56 determines whether or not the luminance difference between the pixel P (i, j + 1) and the center pixel P (i, j) is larger than a predetermined threshold, and P (i, j + 1) When it is determined that -P (i, j) is larger than the predetermined threshold value, the process proceeds to step S56, 1 is substituted into the variable h, and the process returns to step S42.
[0080]
If it is determined in step S55 that P (i, j + 1) −P (i, j) is not greater than the predetermined threshold value, the process proceeds to step S57, and the motion detection unit 56 further includes the center pixel P ( i, j-1) and the pixel P (i, j) are determined whether or not the luminance difference is larger than a predetermined threshold, and P (i, j-1) -P (i, j) is larger than the predetermined threshold. When it determines with it being large, it progresses to step S58, -1 is substituted to the variable h, and it returns to step S42. If it is determined in step S57 that P (i, j-1) -P (i, j) is not greater than the predetermined threshold value, the process returns to step S42.
[0081]
On the other hand, if it is determined in step S54 that the vertical movement is detected instead of the horizontal movement, the process proceeds to step S59, and the motion detection unit 56 determines that the pixel P (i + 1, j) and the central pixel P ( It is determined whether or not the luminance difference of i, j) is larger than a predetermined threshold value. If it is determined that P (i + 1, j) −P (i, j) is larger than the predetermined threshold value, the process proceeds to step S60. Then, 1 is substituted into the variable v, and the process returns to step S42.
[0082]
If it is determined in step S59 that P (i + 1, j) −P (i, j) is not greater than the predetermined threshold value, the process proceeds to step S61, and the motion detection unit 56 further includes the pixel P (i −1, j) and the central pixel P (i, j) are determined whether or not the luminance difference is greater than a predetermined threshold, and P (i−1, j) −P (i, j) is greater than the predetermined threshold. When it determines with it being large, it progresses to step S62, -1 is substituted to the variable v, and it returns to step S42. If it is determined in step S61 that P (i−1, j) −P (i, j) is not greater than the predetermined threshold value, the process returns to step S42.
[0083]
If it is determined in step S42 that motion in both the vertical and horizontal directions has been detected, or in step S53, P (i, j) −P ′ (i, j) is smaller than a predetermined negative threshold. If it is determined that there is no, the process returns to step S9 in FIG.
[0084]
In step S9 of FIG. 4, the motion detection unit 56 associates the values of the variables h and v calculated by the above-described processing with the correspondence diagram shown in FIG. 10, and sets a pixel value (index value) indicating the motion direction. To do.
[0085]
In the example of FIG. 10, LU (Left Upper) represents that the pixel has moved to the upper left, U (Upper) represents that the pixel has moved upward, and RU (Right Upper) represents the pixel to the upper right. L (Left) indicates that the pixel has moved to the left, N (No move) indicates that the pixel has not moved, and R (Right) indicates that the pixel has moved to the right. LD (Left Down) indicates that the pixel has moved to the lower left, D (Down) indicates that the pixel has moved down, and RD (Right Down) indicates that the pixel has moved to the lower right. Indicates that it has moved.
[0086]
In FIG. 10, for example, N = 0, LU = 1, U = 2, RU = 3, L = 4, R = 5, LD = 6, D = 7, and RD = 8. When the calculated variables h and v are both -1, the pixel value indicating the movement direction is set to 1. When the calculated variable h is 0 and the variable v is −1, the pixel value indicating the movement direction is set to 2. When the calculated variable h is 1 and the variable v is −1, the pixel value indicating the movement direction is set to 3. When the calculated variable h is −1 and the variable v is 0, the pixel value indicating the movement direction is set to 4. When the calculated variables h and v are both 0, the pixel value indicating the movement direction is set to 0. When the calculated variable h is 1 and the variable v is 0, the pixel value indicating the movement direction is set to 5. When the calculated variable h is −1 and the variable v is 1, the pixel value indicating the movement direction is set to 6. When the calculated variable h is 0 and the variable v is 1, the pixel value indicating the movement direction is set to 7. When the calculated variables h and v are both 1, the pixel value indicating the movement direction is set to 8.
[0087]
That is, when the pixel value is 1, the pixel has moved to the upper left, when the pixel value is 2, the pixel has moved up, and when the pixel value is 3, the pixel has moved to the upper right. If the pixel value is 4, the pixel has moved to the left, if the pixel value is 0, the pixel has not moved, and if the pixel value is 5, the right When the pixel value is 6, the pixel has moved to the lower left, when the pixel value is 7, the pixel has moved down, and the pixel value is 8. In some cases, the pixel has moved to the lower right.
[0088]
Note that the present invention is not limited to this, and it is also possible to detect the motion direction using a technique such as block matching or motion vector matching.
[0089]
For example, the number N of pixels having U and D pixel values in the region selected in step S8._U, N_DEach and N_UIs over a predetermined threshold value, it is recognized that “upward movement has occurred” within the region, and N_DIs greater than a predetermined threshold value, it is recognized that “downward movement has occurred” within the region. N_U, N_DHowever, if both exceed a predetermined threshold, it is recognized as “no movement”.
[0090]
Returning to step S10 in FIG. 4, the edge detection unit 57 is selected in the process of step S8 from the areas labeled in the process of step S2 (areas 85 and 86 in the example of FIG. 6F). The edge information of the area having the smallest evaluation value (for example, area 85) is detected.
[0091]
Here, the process of detecting edge information will be described in more detail with reference to FIG. In the following description, it is assumed that the region 85 is selected as the region having the smallest evaluation value in the process of step S8.
[0092]
FIG. 11A shows the center of gravity C of the region 85 detected by the process of step S3. FIG. 11B shows the inter-frame difference image binarized by the process of step S2.
[0093]
The edge detection unit 57 detects the edge information in the vertical direction and the horizontal direction of the inter-frame difference image (FIG. 11B) binarized by the processing in step S2. Thereby, as shown in FIGS. 11C and 11D, the edge information in the vertical direction and the edge information in the horizontal direction are detected.
[0094]
Next, the edge information detection process will be described in more detail. For detection of edge information, a filter as shown in FIG. 12 is used.
[0095]
The edge detection unit 57 performs a product sum (convolution) calculation of the filter and the pixel value shown in FIG. 12 around the pixel for which edge information is to be detected, and binarizes the absolute value of the calculation result to obtain the edge pixel. Is calculated. If the absolute value of the product-sum operation is larger than a predetermined threshold, the edge detection unit 57 detects it as an edge pixel.
[0096]
First, an example of determining whether or not the center pixel P (i, j) (FIG. 9) in the current frame is a vertical edge pixel will be described. The edge detection unit 37 performs a product-sum operation Lv of the filter and the pixel value shown in FIG. 12A according to the following equation (5).
Lv = -1 * P (i-1, j-1) + 0 * P (i-1, j) + 1 * P (i-1, j + 1) -2 * P (i, j-1)
+ 0 · P (i, j) + 2 · P (i, j + 1) −1 · P (i + 1, j−1) + 0 · P (i + 1, j)
+ 1 · P (i + 1, j + 1) (5)
[0097]
The edge detection unit 57 determines whether or not the absolute value of the product-sum operation Lv is greater than a predetermined threshold Tv as a result of the calculation of the formula (5), and the absolute value of the product-sum operation Lv is greater than the predetermined threshold Tv. If it is determined that the pixel size is large, the pixel P (i, j) is detected as an edge pixel (edge information) in the vertical direction.
[0098]
Next, an example of determining whether or not the center pixel P (i, j) (FIG. 9) in the current frame is a horizontal edge pixel will be described. The edge detection unit 37 performs a product-sum operation Lh of the filter and the pixel value shown in FIG. 12B according to the following equation (6).
Lh = -1 * P (i-1, j-1) -2 * P (i-1, j) -1 * P (i-1, j + 1) + 0 * P (i, j-1)
+ 0 · P (i, j) + 0 · P (i, j + 1) + 1 · P (i + 1, j-1) + 2 · P (i + 1, j)
+ 1 · P (i + 1, j + 1) (6)
[0099]
The edge detection unit 57 determines whether or not the absolute value of the product-sum operation Lh is larger than a predetermined threshold Th as a result of the calculation of the formula (6), and the absolute value of the product-sum operation Lh is greater than the predetermined threshold Th. When it is determined that the pixel size is large, the pixel P (i, j) is detected as an edge pixel (edge information) in the horizontal direction.
[0100]
By performing such processing on each pixel of the binarized inter-frame difference image, vertical edge information and horizontal edge information are detected.
[0101]
Next, consider the three areas of top, middle, and bottom obtained by dividing the Fw × Fh rectangular area 91 (FIG. 11E) into three equal parts in the vertical direction.
[0102]
The edge detection unit 57 sets the center of the rectangular area 91 of Fw × Fh to be the center of gravity C shown in FIG. 11A, and counts the number of edge pixels in the vertical direction in each area. At the same time (FIG. 11F), the number of edge pixels in the horizontal direction is counted (FIG. 11G).
[0103]
That is, when the user's 1 hand 71 (FIG. 18 (A)) is imaged with the fingertip facing upward, the imaged image is divided into three and the number of edge pixels for each region is counted. Thus, the structure of the hand can be reflected in the temporal feature change that occurs during the hand gripping operation.
[0104]
Therefore, in the case of a hand gripping operation, the pixel edge distribution is concentrated on the upper part of the region (the top region in FIG. 11E) immediately after the start of the operation, and at the end of the operation, the center of the region (FIG. 11E). The pixel edge distribution is concentrated in the middle area. On the other hand, the pixel edge distribution hardly occurs in the lower part of the region (the bottom region in FIG. 11E). On the other hand, in the operation of opening the hand, the pixel edge is concentrated at the center of the region immediately after the start of the operation, and the pixel edge distribution is concentrated at the upper portion of the region at the end of the operation. On the other hand, the pixel edge distribution hardly occurs in the lower part of the region, similar to the operation of grasping the hand.
[0105]
Returning to FIG. 4, in step S <b> 11, the edge detection unit 57 determines the number of vertical edge pixels belonging to the top area as e._t ^v, E is the number of vertical edge pixels to which the middle area belongs_m ^v, And the number of vertical edge pixels belonging to the bottom area e_b ^v, And the number of horizontal edge pixels belonging to the top area e_t ^h, E is the number of horizontal edge pixels to which the middle area belongs_m ^h, And the number of horizontal edge pixels belonging to the bottom area e_b ^hThen, the feature vector f is calculated according to the following equation (7). Here, Fs = Fw × Fh.
f≡ (R_t ^v, R_t ^h, R_m ^v, R_m ^h, R_b ^v, R_b ^h)
= 1 / Fs · (e_t ^v, E_t ^h, E_m ^v, E_m ^h, E_b ^v, E_b ^h(7)
[0106]
In step S12, the DP value calculation unit 62 calculates a DP value based on the feature vector f calculated in the process of step S11 using a continuous DP method. Here, the feature vector f calculated in the process of step S11 is used as the recognition reference pattern, and, for example, the motion of holding or opening the hand is captured by the video camera 2 in advance as the reference pattern, and the edge Of the time series of the feature vector f calculated by the detection unit 57, a portion corresponding to the hand gripping or opening operation is cut out.
[0107]
At time t, the DP value calculation unit 62 determines the distance d (t, γ) = ‖q (t) between the reference pattern R = {r (γ) | 1 ≦ γ ≦ T} and the recognition target pattern q (t). Using -r (γ) ‖, the cumulative distance S (t, γ) is calculated according to the following equations (8) and (9). Here, the following equation (8) is an initial condition for calculating the cumulative distance S (t, γ).
S (-1, γ) = S (0, γ) = ∞ (1 ≦ γ ≦ T) (8)
[0108]
The cumulative distance S (t, γ) when 1 ≦ γ is expressed by the following equation (9).
[Expression 2]

[0109]
Here, if γ = T, S (t, T) represents the cumulative distance when the area (1, T) of the reference pattern R is optimally adapted to the input time series pattern. When this is normalized by the sum of weights 3 · T, it is expressed by the following equation (10).
[Equation 3]

This is determined as a continuous DP value at time t.
[0110]
As shown in FIG. 13, when a pattern similar to the reference pattern R is generated, the continuous DP value takes the minimum value min. If the minimum value min is less than or equal to a predetermined threshold value, it is determined that the reference pattern R has been matched. Therefore, when the minimum value min is detected, the gripping operation is recognized.
[0111]
Thus, when the recognition target pattern matches the hand grip reference pattern, the DP value calculation unit 62 outputs “grip” to the grip movement determination unit 61, and the recognition target pattern matches the hand opening reference pattern. In this case, “open” is gripped and output to the motion determination unit 61.
[0112]
After the “upward movement” is input from the motion detection unit 56 and the “opening” is input from the DP value calculation unit 62 during the Fmove frame, the gripping motion determination unit 61 recognizes the hand opening operation. be able to. The gripping motion determination unit 61 recognizes the gripping motion of the hand when “grip” is input from the DP calculation unit 62 during the Fmove frame after “downward motion” is input from the motion detection unit 56. can do. Note that the value of Fmove can be arbitrarily set. For example, a value such as 15 frames is set.
[0113]
In step S <b> 13, the center-of-gravity locus recognition unit 32 recognizes a hand shaking motion based on the center of gravity detected in the process of step S <b> 3.
[0114]
Here, the hand movement recognition process will be described with reference to the flowcharts of FIGS. 14 and 15.
[0115]
In step S101, the center-of-gravity locus recognition unit 32 substitutes the center-of-gravity position of the area selected in each frame into a ring buffer including ten buffers as shown in FIG. In step S102, the center-of-gravity locus recognition unit 32 updates the center-of-gravity positions X and Y of the current frame and the center-of-gravity positions X ′ and Y ′ one frame before, which are substituted in the process of step S101. In this case, the center of gravity position X = X of the current frame_Five, Y = Y_FiveCenter of gravity position X ′ = X one frame before_Four, Y ′ = Y_FourUpdated to
[0116]
In step S103, the center-of-gravity locus recognition unit 32 determines whether or not the absolute value of XX ′ is greater than a predetermined positive threshold value, that is, the X coordinate of the current frame and the previous n frames (in this case, one frame). When X ′ coordinates are compared to determine whether or not there is a change, and when it is determined that the absolute value of XX ′ is not greater than a predetermined positive threshold, that is, when it is determined that there is no change The process proceeds to step S104.
[0117]
In step S104, the center-of-gravity locus recognition unit 32 determines whether or not the ring buffer has made one round. If it is determined that the ring buffer has not made one round, the process returns to step S102 and the center-of-gravity position of the current frame. X, Y and the center of gravity position X ′, Y ′ one frame before are updated (in this case, the center of gravity position X = X of the current frame)_Four, Y = Y_FourCenter of gravity position X ′ = X one frame before_Three, Y ′ = Y_ThreeUpdate) and repeat the subsequent processing. That is, until the X coordinate changes, the current center of gravity position X, Y and the center of gravity position X ′, Y ′ one frame before are updated by one frame. If it is determined in step S104 that the ring buffer has made one turn, the process proceeds to step S128.
[0118]
If it is determined in step S103 that the absolute value of XX ′ is greater than a predetermined positive threshold value, the process proceeds to step S105, where the center-of-gravity locus recognition unit 32 determines the changed coordinates X ′ and Y ′. Substituting for Xs and Ys respectively, the variable max is initialized to 0 and the variable BigSkip is initialized to FALSE. In step S106, the center-of-gravity locus recognition unit 32 determines whether or not XX ′ is greater than 0, and if greater than 0 (when XX ′ is a positive value), the process proceeds to step S107. It is determined whether XX ′ is greater than a predetermined positive threshold value.
[0119]
If it is determined in step S107 that XX ′ is larger than the predetermined positive threshold value, the process proceeds to step S108, and the barycenter locus recognition unit 32 substitutes TRUE for the variable BigSkip. If it is determined in step S107 that XX ′ is not larger than the predetermined positive threshold value, the process proceeds to step S109, and the center-of-gravity locus recognition unit 32 further determines that XX ′ is a predetermined negative threshold value. If it is determined whether or not it is smaller than the predetermined negative threshold, the process proceeds to step S123. If it is determined that it is not smaller than the predetermined negative threshold, the process proceeds to step S110.
[0120]
After step S108 or S109, in step S110, the center-of-gravity locus recognition unit 32 determines whether the absolute value of X-Xs is greater than the absolute value of the variable max (in this case, 0), and X- When it is determined that the absolute value of Xs is larger than the absolute value of variable max, the process proceeds to step S111, and the value of X−Xs is substituted for variable max. If it is determined in step S110 that the absolute value of X-Xs is not greater than the absolute value of variable max, the process of step S111 is skipped and the process proceeds to step S112.
[0121]
After step S110 or step S111, in step S112, the center-of-gravity locus recognition unit 32 determines whether the absolute value of Y−Ys is smaller than a predetermined threshold value. Proceeding to step S113, if it is determined that the absolute value of Y−Ys is not smaller than the predetermined threshold value, the process proceeds to step S128.
[0122]
In step S113, the center-of-gravity locus recognition unit 32 determines whether or not the ring buffer has made one round. If it is determined that the ring buffer has not made one round, the process proceeds to step S114, where the center-of-gravity locus recognition unit 32 The center-of-gravity positions X and Y of the current frame and the center-of-gravity positions X ′ and Y ′ of the previous frame are updated, and the process returns to step S107. In the present case, the center of gravity position X = X of the current frame going back one frame at a time_Four, Y = Y_FourCenter of gravity position X ′ = X one frame before_Three, Y ′ = Y_ThreeUpdated to If it is determined in step S113 that the ring buffer has made one turn, the process proceeds to step S123.
[0123]
In Step S106, when it is determined that XX ′ is not greater than 0 (when it is determined that XX ′ is a negative value), the process proceeds to Step S115, where the center-of-gravity locus recognition unit 23 , X−X ′ is determined to be smaller than a predetermined negative threshold, and if it is determined to be smaller than the predetermined negative threshold, the process proceeds to step S116. In step S116, the center-of-gravity locus recognition unit 32 substitutes TRUE for the variable BigSkip. If it is determined in step S115 that XX ′ is not smaller than the predetermined negative threshold value, the process proceeds to step S117, and the center-of-gravity locus recognition unit 32 further determines that XX ′ is a predetermined positive threshold value. If it is determined whether or not it is greater than the predetermined positive threshold, the process proceeds to step S118. If it is determined that it is greater than the predetermined positive threshold, the process proceeds to step S123.
[0124]
After step S116 or S117, in step S118, the center-of-gravity locus recognition unit 32 determines whether or not the absolute value of X-Xs is greater than the absolute value of the variable max, and the absolute value of X-Xs is the variable max. If it is determined that the absolute value is greater than the absolute value, the process proceeds to step S119, and the value of X−Xs is substituted into the variable max. If it is determined in step S118 that the absolute value of X−Xs is not greater than the absolute value of variable max, the process of step S119 is skipped and the process proceeds to step S120.
[0125]
After step S118 or step S119, in step S120, the center-of-gravity locus recognition unit 32 determines whether or not the absolute value of Y−Ys is smaller than a predetermined threshold value. Proceeding to step S121, if it is determined that the absolute value of Y−Ys is not smaller than the predetermined threshold value, the process proceeds to step S128.
[0126]
In step S121, the center-of-gravity locus recognition unit 32 determines whether or not the ring buffer has made one round. If it is determined that the ring buffer has not made one round, the process proceeds to step S122, where the center-of-gravity locus recognition unit 32 The center-of-gravity positions X and Y of the current frame and the center-of-gravity positions X ′ and Y ′ of the previous frame are updated, and the process returns to step S115. In the present case, the center of gravity position X = X of the current frame going back one frame at a time_Four, Y = Y_FourCenter of gravity position X ′ = X one frame before_Three, Y ′ = Y_ThreeUpdated to If it is determined in step S121 that the ring buffer has made one turn, the process proceeds to step S123.
[0127]
After step S109, 113, 117 or step S121, in step S123, the barycenter locus recognition unit 32 determines whether or not the variable BigSkip is FALSE, and if the variable BigSkip is determined to be FALSE, Proceeding to step S124, in step S124, the center-of-gravity locus recognition unit 32 determines whether or not the value of the variable max is greater than a predetermined positive threshold value, and determines that the value of the variable max is greater than a predetermined positive threshold value. In this case, the process proceeds to step S125, where it is recognized that there is a right swing, and the process returns to step S14 in FIG.
[0128]
If it is determined in step S124 that the variable max is not greater than the predetermined positive threshold value, the process proceeds to step S126, and the gravity center locus recognition unit 32 further determines whether the variable max is smaller than the predetermined negative threshold value. If it is determined that the value of the variable max is smaller than the predetermined negative threshold value, the process proceeds to step S127, where it is recognized that there is a left swing, and the process returns to step S14 in FIG.
[0129]
If it is determined in step S123 that the variable BigSkip is not FALSE, if it is determined in step S126 that the variable max is not smaller than the predetermined negative threshold value, or if the process of step S104, 112, or step S120 is performed. After that, in step S128, the barycentric locus recognition unit 32 recognizes that no hand has been shaken, and returns to step S14 in FIG.
[0130]
Another example of the swing motion recognition process will be described with reference to the flowchart of FIG. In step S141, the barycenter locus recognition unit 32 substitutes Right, Left, or None for curr_ges. That is, the processing result in step S125, S127, or S128 in FIG. 15 (right swing, left swing, or no hand shake) is substituted. In step S142, the center-of-gravity locus recognition unit 32 compares the time of the current frame with the reference time, and calculates the elapsed time of the current frame. In step S143, the center-of-gravity locus recognition unit 32 determines whether or not the current frame has moved rightward or leftward. If it is determined that there has been rightward or leftward movement, the process proceeds to step S144. In step S144, the center-of-gravity locus recognition unit 32 updates the reference time for calculating the elapsed time to the time of the current frame.
[0131]
If it is determined in step S143 that there is no rightward or leftward movement, the process of step S144 is skipped and the process proceeds to step S145. In step S145, the barycenter locus recognition unit 32 determines whether the elapsed time of the current frame calculated in the process of step S142 is longer than a predetermined threshold, and determines that the elapsed time of the current frame is longer than the predetermined threshold. If so, the process proceeds to step S146, and the gesture (right swing or left swing) stored in the variable prev_ges is output.
[0132]
If it is determined in step S145 that the elapsed time of the current frame is not longer than the predetermined threshold value, the process proceeds to step S147, and the barycentric locus recognition unit 32 sets curr_ges which is a gesture (right or left) of the current frame. Output. In step S148, the center-of-gravity locus recognition unit 32 substitutes (saves) curr_ges, which is a gesture of the current frame output in the process of step S147, in prev_ges.
[0133]
In step S149, the center-of-gravity locus recognition unit 32 recognizes the swing motion from the gesture output in the process of step S146 or S147, and returns to step S14 of FIG.
[0134]
When the process described above changes from one movement (for example, leftward movement) to the other movement (for example, rightward movement) in a short time, the one (leftward) movement is excluded and the other ( Since the movement in the right direction) is recognized, the processing can be speeded up. Note that it is also possible to recognize the hand gesture in the vertical direction in the same manner as the processing in FIG. 14, FIG. 15, or FIG.
[0135]
Returning to FIG. 4, in step S <b> 14, the display / output device 4 is supplied from the recognition result (gripping or opening motion) supplied from the gripping motion determination unit 61 of the image processing device 3 and the barycentric locus recognition unit 32. Recognition results (gestures and hand gestures) are displayed. In step S15, the image processing apparatus 3 determines whether or not the operation of the user 1 has ended. If it is determined that the operation has not yet ended, the image processing apparatus 3 returns to step S1 and repeats the above-described processing. If it is determined in step S15 that the operation of the user 1 has been completed, the processing is terminated.
[0136]
When the order of the above-described gripping motion recognition processing is summarized, it is as shown in FIG. Then, with reference to FIG. 18, the hand gripping movement recognition process will be described next. For simplification, the motion block region is only the region 72, and the processing for selecting the region having the smallest evaluation value performed when there are a plurality of motion block regions is omitted here.
[0137]
The video camera 2 captures the hand 71 of the user 1 (FIG. 18A), converts the captured image data into luminance data, and stores it in the image storage unit 21. The inter-frame difference calculation unit 41 of the region dividing unit 22 is between the luminance data of the image data of the current frame converted by the video camera 2 and the luminance data of the image data of the previous frame stored in the image storage unit 21. The inter-frame difference value is calculated. The threshold processing unit 42 binarizes a pixel having an inter-frame difference value equal to or greater than a predetermined threshold among the inter-frame difference values as 1 and a pixel having an inter-frame difference value that does not satisfy the predetermined threshold as 0 (FIG. 18 (B)).
[0138]
The labeling unit 46 performs a labeling process on the motion block area 72 of the binarized image. The center-of-gravity detection unit 51 detects the center of gravity in the labeled region 72 (FIG. 18C).
[0139]
The edge detection unit 57 detects edge information in the vertical direction and horizontal direction of the region 72, respectively. That is, the edge detection unit 57 sets the center of gravity C of the region 72 to be the center of the rectangular region 91 and sets the edge pixels in the vertical direction in the top, middle, and bottom three regions obtained by dividing the rectangular region 91 into three equal parts. The number is counted (FIG. 18E), and the number of edge pixels in the horizontal direction is counted (FIG. 18F).
[0140]
The edge detection unit 57 calculates the feature vector f according to the above equation (7) based on the counted number of edge pixels. The DP value calculation unit 62 calculates a continuous DP value from the calculated feature vector f using a continuous DP method, and outputs a motion matching the reference pattern to the gripping motion determination unit 61. As a result, the gripping motion determination unit 61 can recognize a hand gripping or opening motion.
[0141]
As described above, since the user's hand gripping or opening operation can be recognized from the image data captured by the video camera, a computer or the like can be easily obtained from a remote place without using an input device such as a mouse or a keyboard. Can be operated.
[0142]
In addition, since no special device or a plurality of video cameras are used, a low-cost system can be realized.
[0143]
In addition, in the above, it is possible to recognize the hand movement (trajectory) movement and the hand swinging movement as well as the hand gripping / opening movement. Move (mouse move), hold hand (icon selection), move while holding hand (drag with icon selected), open hand (deselect icon), swing left (left mouse) Button click) and swinging right (mouse right button click) can be performed. Therefore, the user can start an application on the screen or move a window displayed on the screen by an intuitive operation without using a mouse.
[0144]
Thus, for example, a file displayed on the display is selected by grasping the hand, dragged by moving the hand, and opened in a predetermined place (for example, a hard disk drive, etc.), and the desired file is displayed. You can move and copy files with intuitive actions.
[0145]
The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.
[0146]
The recording medium is distributed to provide a program to the user separately from the computer, and includes a magnetic disk (including a floppy disk) on which the program is recorded, an optical disk (CD-ROM (Compact Disk-Read Only Memory)) , Not only composed of package media consisting of magneto-optical disk (including MD (Mini-Disk)), including DVD (Digital Versatile Disk), or semiconductor memory, but also pre-installed in the computer Consists of a ROM, hard disk, etc. that provide the program.
[0147]
In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.
[0148]
Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.
[0149]
【The invention's effect】
  As described above, according to the image processing apparatus, the image processing method, and the program recorded in the recording medium of the present invention, the hand is imaged, the captured image is converted into luminance information, and the image is converted.Of the current frameLuminance informationAnd the difference in luminance information of the previous frameBased on the movement area, calculating an evaluation value for evaluating whether the detected movement area is a hand, and calculating a plurality of movement areas based on the calculated evaluation valueA predetermined area ofHand areaAsSelect and select hand areaAn image formed by the difference between the luminance information of the current frame and the luminance information of the previous frameEdge information is extracted, a feature vector of the extracted edge information is calculated as a feature amount, a DP value is calculated using a DP matching technique based on the calculated feature amount, and the calculated DP value is It is determined whether or not the DP value is smaller than the predetermined value, and when the DP value is determined to be smaller than the predetermined value, the hand gripping operation is recognized. The gripping and opening operations can be recognized by image processing.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration example of an image processing system to which the present invention is applied.
2 is a block diagram illustrating a configuration of the image processing apparatus in FIG. 1. FIG.
3 is a block diagram illustrating a detailed configuration of an image processing unit and a gesture recognition unit in FIG. 2;
FIG. 4 is a flowchart illustrating hand gripping / swinging motion recognition processing.
FIG. 5 is a flowchart for explaining area division processing in step S2 of FIG. 4;
FIG. 6 is a diagram for explaining area division processing;
FIG. 7 is a diagram for explaining an operation of calculating a center of gravity.
FIG. 8 is a flowchart for explaining a movement direction detection process in step S9 of FIG.
FIG. 9 is a diagram for explaining a frame for detecting a movement direction;
FIG. 10 is a correspondence diagram for setting a pixel value indicating a movement direction.
FIG. 11 is a diagram for explaining edge information detection processing;
FIG. 12 is a diagram illustrating a filter used for edge information detection processing.
FIG. 13 is a diagram for explaining DP matching.
FIG. 14 is a flowchart for explaining a swing motion recognition process in step S13 of FIG.
FIG. 15 is a flowchart following FIG. 14;
FIG. 16 is a diagram illustrating a ring buffer.
FIG. 17 is a flowchart illustrating another example of the swing motion recognition process in step S13 of FIG.
FIG. 18 is a diagram for explaining a gripping motion recognition process.
[Explanation of symbols]
2 video camera, 3 image processing device, 4 display / output device, 11 image processing unit, 12 gesture recognition unit, 21 image storage unit, 22 region dividing unit, 23 region processing unit, 31 grasping motion recognition unit, 32 gravity center locus recognition Unit, 41 frame difference calculation unit, 42 threshold processing unit, 43 intra-block counting unit, 44 threshold processing unit, 45 peripheral search unit, 46 labeling unit, 51 centroid detection unit, 52 circumscribed frame detection unit, 53 area detection unit, 54 region evaluation unit, 55 region storage unit, 61 gripping motion determination unit, 62 DP value calculation unit

Claims (6)

Imaging means for imaging the hand;
Conversion means for converting an image captured by the imaging means into luminance information;
A motion area detecting means for detecting a motion area based on a difference between the luminance information of the current frame converted by the converting means and the luminance information of one frame before ;
Evaluation value calculation means for calculating an evaluation value for evaluating whether the movement area detected by the movement area detection means is the hand;
Selection means for selecting a predetermined area of the plurality of movement areas as the hand area based on the evaluation value calculated by the evaluation value calculation means;
Edge information extraction means for extracting edge information of an image formed by a difference between the luminance information of the current frame in the hand region selected by the selection means and the luminance information of the previous frame ;
Feature amount calculating means for calculating the feature vector of the edge information extracted by the edge information extracting means as a feature amount;
DP value calculation means for calculating a DP value using a DP matching technique based on the feature quantity calculated by the feature quantity calculation means;
Determining means for determining whether the DP value calculated by the DP value calculating means is smaller than a predetermined value;
An image processing apparatus comprising: a recognizing unit that recognizes the hand gripping operation when the determining unit determines that the DP value is smaller than the predetermined value. The motion area detection unit, the said luminance information of the current frame, by a difference of one frame before the luminance information is equal to or greater than a predetermined threshold value, to detect the motion region The image processing apparatus according to claim 1, wherein: Centroid calculating means for calculating the centroid of the motion area detected by the motion area detecting means,
The edge information extracting means divides a predetermined rectangular area centered on the center of gravity of the motion area calculated by the center of gravity calculating means into three equal parts in the vertical direction, and in each of the areas, vertical and horizontal edges The image processing apparatus according to claim 1, wherein information is extracted. A moving direction detecting unit for detecting a moving direction of the moving region selected by the selecting unit;
The recognition means recognizes the movement of the hand detected by the movement direction detection means as a mouse movement operation, recognizes the movement of the hand as an operation of selecting an icon by the mouse, and holds the hand Recognizing the movement operation while the icon is selected as an operation of dragging, recognizing the operation of opening the hand as an operation of canceling the selection of the icon, and the operation of shaking the hand leftward as the left button of the mouse The image processing apparatus according to claim 1, wherein the image processing apparatus recognizes an operation of clicking and recognizes an operation of shaking the hand in the right direction as an operation of clicking the right button of the mouse. An imaging step for imaging the hand;
A conversion step of converting an image captured by the processing of the imaging step into luminance information;
A motion region detection step of detecting a motion region based on a difference between the luminance information of the current frame converted by the processing of the conversion step and the luminance information of the previous frame ;
An evaluation value calculating step for calculating an evaluation value for evaluating whether the moving region detected by the processing of the moving region detecting step is the hand;
Based on the evaluation value calculated by the processing of the evaluation value calculation step, a selection step of selecting a predetermined area of the plurality of movement areas as the hand area;
An edge information extraction step for extracting edge information of an image formed by a difference between the luminance information of the current frame in the hand region selected by the processing of the selection step and the luminance information of the previous frame ;
A feature amount calculating step of calculating a feature vector of the edge information extracted by the processing of the edge information extracting step as a feature amount;
A DP value calculating step of calculating a DP value using a DP matching technique based on the feature amount calculated by the feature amount calculating step;
A determination step of determining whether the DP value calculated by the processing of the DP value calculation step is smaller than a predetermined value;
And a recognition step of recognizing the gripping action of the hand when the DP value is determined to be smaller than the predetermined value by the processing of the determination step. An imaging step for imaging the hand;
A conversion step of converting an image captured by the processing of the imaging step into luminance information;
A motion region detection step of detecting a motion region based on a difference between the luminance information of the current frame converted by the processing of the conversion step and the luminance information of the previous frame ;
An evaluation value calculating step for calculating an evaluation value for evaluating whether the moving region detected by the processing of the moving region detecting step is the hand;
Based on the evaluation value calculated by the processing of the evaluation value calculation step, a selection step of selecting a predetermined area of the plurality of movement areas as the hand area;
An edge information extraction step for extracting edge information of an image formed by a difference between the luminance information of the current frame in the hand region selected by the processing of the selection step and the luminance information of the previous frame ;
A feature amount calculating step of calculating a feature vector of the edge information extracted by the processing of the edge information extracting step as a feature amount;
A DP value calculating step of calculating a DP value using a DP matching technique based on the feature amount calculated by the feature amount calculating step;
A determination step of determining whether the DP value calculated by the processing of the DP value calculation step is smaller than a predetermined value;
When the DP value is determined to be smaller than the predetermined value by the process of the determination step, a recognition step for recognizing the gripping action of the hand is recorded. Recording medium.

Images