JP2004246814A - Indication movement recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an indication movement recognition device capable of an improvement of decision accuracy of an indication position or an indication direction by a user by natural and easy operation. <P>SOLUTION: Video cameras 20A, 20B pick up images of the user from a plurality of directions different from each other. A control part 16 recognizes three-dimensional coordinates of a characteristic point whose position varies according to a change of the indication position or the indication direction when the user makes a movement for indicating a specific position or direction, on the basis of the picked-up images, and decides the indication position or the indication direction by the user on the basis of the three-dimensional coordinates. The control part 16 decides whether the characteristic point is in a stop state or not, on the basis of two-dimensional coordinates in a two-dimensional plane parallel to a large-screen display 14 among the three-dimensional coordinates of the characteristic point. When a movement amount in the two-dimensional plane is a prescribed threshold value or below, the control part 16 decides that it is in the stop state, and changes over a mode to a click mode. When it is not in the stop state, the control part 16 changes over the mode to a pointing mode. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

【０１０４】
上記の結果より次の（３）式が得られる。
【０１０５】
【数２】

【０１０６】
（３）式を（２）式に代入して整理すると、次の（４）式が得られる。
【０１０７】
【数３】

【０１０８】
上記と同様に座標値Ｙ，Ｚについて解くと、次の（５），（６）式が得られる。
【０１０９】
【数４】

【０１１０】
続いて、（４），（５），（６）式を連立させて座標値Ｘ，Ｙ，Ｚを解く。これらを行列式で表すと次の（７）式のようになる。
【０１１１】
【数５】

【０１１２】
そして、（７）式の解は以下の（８）〜（１０）式のようになる。
【０１１３】
【数６】

【０１１４】
従って、ステップ２６８は、演算対象の特徴点の３次元座標データ（ｘ_ｉ，ｙ_ｉ，ｚ_ｉ）を（８）〜（１０）式に各々代入することで、回転中心に相当する点の３次元座標（Ｘ，Ｙ，Ｚ）を求めることができる。また半径Ｒは、演算対象の特徴点の３次元座標データ、及び（８）〜（１０）式の演算結果を（３）式に各々代入することで求めることができる。なお、ステップ２６８は演算手段に対応している。
【０１１５】
次のステップ２７０では、以前に（現在の利用者が情報入力空間に到来してから現在までの間に）特徴点の位置変化の回転中心に相当する点の３次元座標を演算したか否か判定する。判定が否定された場合には何ら処理を行うことなくステップ２７６へ移行するが、判定が肯定された場合にはステップ２７２へ移行し、以前に演算した回転中心に相当する点の３次元座標も考慮して（例えば加重平均値を演算する等により）、回転中心に相当する点の３次元座標（Ｘ，Ｙ，Ｚ）を最終決定する。
【０１１６】
そして次のステップ２７６では、ディスプレイ１４までの距離及びディスプレイ１４の大きさに基づいて回転中心に相当する点の３次元座標（Ｘ，Ｙ，Ｚ）が修正されることで基準点の３次元座標（Ｘ，Ｙ，Ｚ）が演算され、記憶部１８に記憶される。
【０１１７】
このように、本実施形態では、利用者が情報入力空間に到来して指示動作を開始してから若干の時間が経過した後（現時点より所定時間前から現時点迄の期間内に演算された特徴点の３次元座標データが所定数以上になった後）は、特徴点の移動軌跡に基づき、特徴点の位置変化の回転中心に相当する点の３次元座標を最小二乗法によって演算し、この回転中心に相当する点の３次元座標に基づいて設定した基準点の３次元座標を用いて利用者による指示位置を判断しているので、指示動作として利用者が図８（Ａ）〜（Ｃ）に示す各種動作（姿勢）の何れを行っている場合にも、また図９に示すように、利用者の体格が大きく相違している場合にも、これらの影響を受けることなく基準点を適切な位置に設定することができ、利用者による指示位置を精度良く判断することができる。
【０１１８】
また、基準点座標演算処理（及び特徴点座標演算処理）は繰り返し行われるので、例えば図１０（Ａ）、（Ｂ）に示すように利用者が指示動作に用いている腕が利用途中で変化した場合、或いは指示動作として利用者が行っている動作が利用途中で変化した場合（例えばハイポジションからミドルポジション又はローポジションに変化した等の場合）にも、この変化に追随して基準点を適切な位置に切り替えることができ、利用者による指示位置を常に精度良く判断することができる。
【０１１９】
基準点座標演算処理が終了すると、図４のフローチャートのステップ２０６へ移行し、ディスプレイ１４に表示すべきカーソルの座標を演算する。
【０１２０】
具体的には、先の特徴点座標演算処理で演算された特徴点の３次元座標（ｘ，ｙ，ｚ）と、基準点座標演算処理で演算された基準点の３次元座標（Ｘ，Ｙ，Ｚ）に基づき、利用者の指示動作による指示方向を表す線として、図１１に示すように、基準点Ｐ_０と特徴点Ｐ_Ｘとを結ぶ仮想線（図１１の仮想線５４参照）の延びる方向を求め、利用者が指し示している位置（指示位置）として、ディスプレイ１４の表示面を含む平面と、設定した仮想線の交点（図１１に示す点Ｓ参照）の座標（２次元座標）を演算する。
【０１２１】
そして、演算したカーソル位置の座標をＰＣ１２へ通知する。ＰＣ１２は、ステップ２００で設定されたカーソル情報に基づくカーソルを制御部１６から指定された座標に表示させる。これにより、利用者が指先等をディスプレイ１４に向けて指し示すだけで、その位置にカーソルが表示される。なお、ステップ２０６は、本発明の判断手段に対応している。
【０１２２】
次のステップ２０８では、特徴点座標演算処理で演算された特徴点の３次元座標（ｘ，ｙ，ｚ）に基づいて、特徴点が停止状態か否かを判定する。本実施形態では、特徴点のＸ方向の動きを無視し、２次元平面としてのＹＺ平面の動きのみに着目して特徴点が停止状態か否かを判定する。すなわち、２次元座標（ｙ，ｚ）に基づいて特徴点が停止状態か否かを判定する。なお、ステップ２０８は、本発明の停止状態判定手段及び切替手段に対応している。
【０１２３】
具体的には、現在から過去における所定時間内に演算した特徴点の２次元座標（ｙ，ｚ）に基づいて特徴点のＹＺ平面における所定時間内の移動量を算出し、この移動量に基づいて停止状態か否かを判定する。
【０１２４】
例えば、現在時刻（時刻ｔ）における特徴点のＹＺ成分ベクトルをｒ（ｙ，ｚ，ｔ）＝ｒ（ｔ）で表し、特徴点のサンプリング間隔（演算間隔）をΔｔとした場合、過去ｎ回（ｎは自然数）演算した特徴点のＹＺ成分ベクトルの各々と現在のＹＺ成分ベクトルとの差（距離）である｜ｒ（ｔ）−ｒ（ｔ−Δｔ）｜、｜ｒ（ｔ）−ｒ（ｔ−２Δｔ）｜、｜ｒ（ｔ）−ｒ（ｔ−３Δｔ）｜、…｜ｒ（ｔ）−ｒ（ｔ−ｎΔｔ）｜の各々が全て予め定めた所定閾値以下の場合には特徴点が停止状態であると判定し、これ以外の場合には特徴点が停止状態でないと判定することができる。
【０１２５】
また、現在時刻（時刻ｔ）における特徴点のＹＺ平面における移動速度をｖ（ｔ）＝｛ｒ（ｔ）−ｒ（ｔ−Δｔ）｝／Δｔで表し、過去ｎ回演算した特徴点のＹＺ平面における移動速度である｜ｖ（ｔ）｜、｜ｖ（ｔ−Δｔ）｜、｜ｖ（ｔ−２Δｔ）｜、…｜ｖ（ｔ−ｎΔｔ）｜の各々が全て予め定めた所定閾値以下の場合には特徴点が停止状態であると判定し、これ以外の場合には特徴点が停止状態でないと判定するようにしてもよい。なお、上記の所定閾値は、特徴点がＹＺ平面で多少移動してもほぼ停止状態であるとみなすことができる値に設定される。
【０１２６】
このように、Ｘ方向、すなわちクリック動作に必要となる方向の動作を無視し、ポインティング動作に必要なＹＺ平面における特徴点の移動量のみに基づいて特徴点が停止状態か否かを判定するため、簡単な処理で精度よく停止状態を判定することができる。
【０１２７】
ステップ２０８で停止状態でないと判定された場合には、ステップ２０２へ移行し、特徴点が停止状態であると判定されるまで上記と同様の処理、すなわちステップ２０２〜２０８の処理を繰り返す。この間、逐次カーソル位置の座標が演算されてＰＣ１２に通知され、ＰＣ１２では、設定された移動率でカーソルが移動するようにディスプレイ１４を制御する。
【０１２８】
一方、停止状態であると判定された場合にはステップ２１０へ移行する。ステップ２１０では、クリックモード設定処理を行う。具体的には、ステップ２００と同様に、ディスプレイ１４に表示するカーソルに関する設定処理等を行う。
【０１２９】
カーソルに関する設定処理では、カーソルの移動率をポインティングモードのｐ倍（０≦ｐ≦１）に設定する。ここで、ｐ＝１の場合はポインティングモードの場合と同一の移動率となり、ｐ＝０の場合はカーソルが全く移動しないこととなる。
【０１３０】
なお、クリックモードのときは、カーソルを移動させるのが主目的ではなく、指し示した位置をクリックするのが主目的であるため、正確にその位置をクリックできることが優先される場合が多い。従って、クリックモードのときは、ポインティングモードのときよりも移動率を低下させる、すなわちｐを１未満にすることが好ましい。これにより、利用者の腕がぶれたりすることにより、これに連動してカーソル位置が変動し、クリック位置がずれてしまうのを防ぐことができ、利便性を向上させることができる。
【０１３１】
また、カーソルのデザインは、ポインティングモードの場合と異なるデザインを設定する。例えばポインティングモードのときのカーソルのデザインを矢印又は十字のデザインとした場合、クリックモードでは、指の形状やボタンの形状等に設定する。なお、カーソルの色のみを異ならせるようにしてもよい。
【０１３２】
このようにポインティングモードとクリックモードとでカーソルを異ならせることにより、利用者は現在のモードがポインティングモードなのかクリックモードなのかを容易かつ的確に認識することができる。
【０１３３】
ステップ２１２では、ステップ２０２と同様に特徴点座標演算処理を行い、次のステップ２１４では、ステップ２０４と同様に基準点座標演算処理を行う。
【０１３４】
そして、ステップ２１６では、ステップ２０６と同様に、特徴点座標演算処理で演算された特徴点の３次元座標（ｘ，ｙ，ｚ）と、基準点座標演算処理で演算された基準点の３次元座標（Ｘ，Ｙ，Ｚ）と基づいてディスプレイ１４に表示すべきカーソル位置の座標を演算し、ＰＣ１２へ通知する。これにより、ディスプレイ１４には利用者が指し示した位置にクリックモードに対応した形状のカーソルが表示される。
【０１３５】
ところで、クリック動作として、種々の動作を採用することができるが、本実施形態では、利用者が手を前方に素早く移動させる動作（図１２（Ａ）参照、以下「前進クリック」という）や、利用者が手を後方に素早く移動させる動作（図１２（Ｂ）参照、以下「後進クリック」という）の他、利用者が手を後方に素早く移動させる動作を２回繰り返す動作（図１２（Ｃ）参照、以下「後進ダブルクリック」という）、利用者が手を後方に移動させた後、前方に手を移動させ、元の位置に戻す動作（図１２（Ｄ）参照、以下「後ストローク」という）をクリック動作としている。
【０１３６】
このクリック動作は、ディスプレイ１４の表示面上の特定の箇所を指し示して選択する動作等として極めて自然な動作であり、利用者は違和感を感ずることなくクリック動作を行うことができる。
【０１３７】
また、このようなクリック動作の初期動作では、利用者の手が前方向、すなわちディスプレイ１４に近づく方向に移動する動作又は利用者の手が後方向、すなわちディスプレイ１４から離間する方向に移動する動作が必ず含まれることとなる。
【０１３８】
そこで、次のステップ２１８では、特徴点のＸ座標に基づいて、特徴点のＸ方向における移動速度が所定閾値以上か否かを判定することにより、クリック動作における初期動作がなされたか否かを判定する。
【０１３９】
具体的には、特徴点のｔ時刻におけるＸ方向のＸ成分ベクトルをｘ（ｔ）で表すと共に、特徴点のＸ方向のｔ時点における移動速度をＶｘ（ｔ）＝｛ｘ（ｔ）−ｘ（ｔ−Δｔ）｝／Δｔで表した場合、移動速度Ｖｘ（ｔ）が予め定めた前クリック閾値以上又は後クリック閾値以上になったか否かを判定する。
【０１４０】
ここで、前クリック閾値は、特徴点が前方向に素早く移動した場合に前クリック閾値以上となるような値に設定される。同様に、後クリック閾値は、特徴点が後方向に素早く移動した場合に前クリック閾値以上となるような値に設定される。なお、前クリック閾値及び後クリック閾値の絶対値が同一の値となるように各々の閾値を設定してもよいし、異なる値となるようにしてもよい。
【０１４１】
移動速度Ｖｘ（ｔ）が前クリック閾値以上でもなく、後クリック閾値以上でもない場合、すなわちクリック動作の初期動作が行われていない場合には、ステップ２２０へ移行し、移動速度Ｖｘ（ｔ）が前クリック閾値以上又は後クリック閾値以上になった場合、すなわちクリック動作の初期動作が行われた場合には、ステップ２２２へ移行する。
【０１４２】
ステップ２２０では、ステップ２０８と同様に、特徴点座標演算処理で演算された特徴点の３次元座標（ｘ，ｙ，ｚ）、詳しくは２次元座標（ｙ，ｚ）に基づいて、特徴点が停止状態か否かを判定する。なお、ステップ２２０の処理は本発明の停止状態判定手段及び切替手段に対応している。
【０１４３】
このとき、閾値をステップ２０８で用いる所定閾値と異なる値に設定してもよく、ステップ２２０で用いる閾値をステップ２０８で用いる閾値よりも大きい値に設定することが好ましい。これにより、利用者の手がクリック動作をする直前に変動することによって特徴点がＹＺ平面で変動し、誤ってポインティングモードに移行してしまうのを防ぐことができる。
【０１４４】
このように、特徴点のＹＺ平面における移動量がポインティングモードのときよりも大きくならないとポインティングモードへ移行することができないため、利用者の使い勝手を向上させることができる。
【０１４５】
また、利用者が一旦手の動きを止めてクリック動作に入るべくクリックモードへ移行したものの、再度ポインティングモードに移行したい場合には、手をある程度大きくＹＺ平面で移動させるだけでよいので、ポインティングモードに移行するために手の形状を変化させたりするなどの特別な動作をする必要がなく、簡単かつ自然な動作でポインティングモードに切り替えることができる。
【０１４６】
一方、クリック動作の初期動作が行われたと判定された場合には、ステップ２２２において、特徴点の移動方向が前方か否かが判断される。これは、移動速度Ｖｘ（ｔ）が正か負かによって判断することができる。そして、特徴点の移動方向が前方の場合、すなわち移動速度Ｖｘ（ｔ）が正の値の場合にはステップ２２４へ移行し、移動方向が後方の場合、すなわち移動速度Ｖｘ（ｔ）が負の値である場合には、ステップ２２６へ移行する。
【０１４７】
ステップ２２４では、クリック種別を表すクリック種別変数に前進クリックを表す値（例えば‘１’）を代入し、記憶部１８へ記憶する。
【０１４８】
前進クリック動作は、図１２（Ａ）に示したように、特徴点としての利用者の指先が静止している状態を初期位置として、前方に指先を移動させたあと元の場所に戻る動作、すなわち一旦前方に指先を移動させた後、後方向に指先を移動させる動作が基本であるが、本実施形態では、特徴点が所定閾値以上の速度で前方に移動したことのみをもって前進クリック動作がなされたものとして扱う。
【０１４９】
これは、指先が後方向に移動したことを検出してから前進クリックであることを認識し、前進クリックに対応する処理を行ったのでは利用者にとって反応が鈍いと感じてしまう場合があるためである。このように、特徴点が所定閾値以上の速度で前方に移動したことのみをもって前進クリック動作がなされたものとすることにより、利用者は違和感を感じることなく操作することが可能となる。
【０１５０】
なお、反応性に問題がなければ、前方に指先を移動させたあと後方に指先を移動させたことを検出することにより前進クリックと認識するようにしてもよい。
【０１５１】
一方、移動方向が後方の場合には、ステップ２２６において、後進側クリック判定処理を行う。以下、この処理について図１３に示すフローチャートを参照して説明する。
【０１５２】
初期動作が後進側のクリック動作としては、図１２（Ｂ）〜（Ｄ）に示すように、後進クリック、後進ダブルクリック、後進ストローク等があるが、このようなクリック動作では、前進クリックに比べて処理の反応性よりも処理の確実性が要求されるため、初期動作後所定時間の間の特徴点のＸ方向における軌跡、すなわちＸ座標を求め、これに基づいて後進側のクリック動作を判別する。
【０１５３】
このため、図１３に示すように、ステップ２８０では、まずステップ２０８と同様の特徴点座標演算処理を行う。
【０１５４】
次のステップ２８２では、予め定めた所定時間経過したか否かを判定し、所定時間経過していなければステップ２８０へ戻って再び特徴点座標演算処理を行う。この処理はΔｔ時刻毎に行われる。なお、所定時間は、例えば後ストロークや後進ダブルクリック等のクリック動作に比較的時間を要する一連のクリック動作が終了するのに十分な時間に設定される。
【０１５５】
所定時間経過すると、ステップ２８４へ移行し、ステップ２８０で演算された特徴点の３次元座標（ｘ，ｙ，ｚ）のうちＸ座標のみを用いて、ステップ２１８と同様に各時刻の移動速度Ｖｘ（ｔ）を求める。
【０１５６】
次のステップ２８６では、所定時間内に前クリック閾値以上となった移動速度Ｖｘ（ｔ）が存在するか否かを判断し、所定時間内に前クリック閾値以上となった移動速度Ｖｘ（ｔ）が存在しない場合には、ステップ２８８へ移行する。
【０１５７】
ステップ２８８では、クリック種別を表すクリック種別変数に判別不能を表す値（例えば‘０’）を代入して記憶部１８へ記憶し、本ルーチンを終了する。すなわち、初期動作として後方に特徴点が移動し、その後特徴点が所定時間内に前方へ所定速度以上で移動していない場合には、図１２（Ｂ）〜（Ｄ）の何れの動作にも当てはまらないため、判別不能とする。
【０１５８】
一方、所定時間内に前クリック閾値以上となった移動速度Ｖｘ（ｔ）が存在する場合には、ステップ２９０へ移行し、所定時間内において前クリック閾値以上となった後に後クリック閾値以上となった移動速度Ｖｘ（ｔ）が存在するか否かを判断する。
【０１５９】
そして、所定時間内に後クリック閾値以上となった移動速度Ｖｘ（ｔ）が存在しない場合には、ステップ２９２へ移行する。
【０１６０】
ステップ２９２では、クリック種別を表すクリック種別変数に後進クリックを表す値（例えば‘２’）を代入して記憶部１８へ記憶し、本ルーチンを終了する。すなわち、図１４に示すように、初期動作として後方に特徴点が移動し、その後特徴点が前方へ前クリック閾値以上の速度で移動し、その後特徴点が後方へ後クリック閾値以上の速度で移動していない場合には、後進クリック動作がなされたと判断する（図１２（Ｂ）も参照）。
【０１６１】
また、前クリック閾値以上となった後に所定時間内において後クリック閾値以上になった移動速度Ｖｘ（ｔ）が存在する場合には、ステップ２９４へ移行し、前クリック閾値以上となった後に後クリック閾値以上となり、さらに前クリック閾値以上になった移動速度Ｖｘ（ｔ）が存在するか否かを判断する。
【０１６２】
そして、所定時間内において前クリック閾値以上となった後に後クリック閾値以上となり、さらに前クリック閾値以上になった移動速度Ｖｘ（ｔ）が存在しない場合には、ステップ２９６へ移行する。
【０１６３】
ステップ２９６では、クリック種別を表すクリック種別変数に後ストロークを表す値（例えば‘３’）を代入して記憶部１８へ記憶し、本ルーチンを終了する。すなわち、図１４に示すように、初期動作として後方に特徴点が移動し、その後特徴点が前方へ前クリック閾値以上の速度で移動し、その後特徴点が後方へ後クリック閾値以上の速度で移動し、その後特徴点が前方へ前クリック閾値以上の速度で移動していない場合には、後ストローク動作がなされたと判断する（図１２（Ｃ）も参照）。
【０１６４】
前クリック閾値以上となった後に後クリック閾値以上となり、さらに前クリック閾値以上になった移動速度Ｖｘ（ｔ）が存在する場合には、ステップ２９８へ移行し、クリック種別を表すクリック種別変数に後進ダブルクリックを表す値（例えば‘４’）を代入して記憶部１８へ記憶し、本ルーチンを終了する。すなわち、図１４に示すように、初期動作として後方に特徴点が移動し、その後特徴点が前方へ前クリック閾値以上の速度で移動し、その後特徴点が後方へ後クリック閾値以上の速度で移動し、その後特徴点が前方へ前クリック閾値以上の速度で移動した場合には、後進ダブルクリック動作がなされたと判断する（図１２（Ｄ）も参照）。なお、ステップ２１８、２２２、２２４、２２６の処理は本発明のクリック判定手段に対応している。
【０１６５】
このように、特徴点のＸ方向の移動のみを考慮することで後進クリック等の各種クリック動作の判別処理を簡便に行うことができる。
【０１６６】
なお、図１３では最初にステップ２８０〜２８４において一括して所定時間の特徴点の座標を演算すると共に移動速度を演算してからクリック動作の判別を行っているが、図１５に示すように制御してもよい。
【０１６７】
すなわち、ステップ３００において、図１３に示すステップ２８０と同様に特徴点座標演算処理を行い、次のステップ３０２において、その時刻における移動速度Ｖｘ（ｔ）を図１３に示すステップ２８４と同様に行う。そして、ステップ３０４で移動速度Ｖｘ（ｔ）が前クリック閾値以上か否かを判断し、前クリック閾値以上であればステップ３１０へ移行し、前クリック閾値以上でなければステップ３０６へ移行する。
【０１６８】
ステップ３０６では、予め定めた所定時間経過したか否かを判断し、所定時間経過した場合には、ステップ３０８で図１３に示すステップ２８８と同様の処理を行い、所定時間経過していない場合には、ステップ３００へ戻って上記と同様の処理を繰り返す。
【０１６９】
ステップ３１０では、ステップ３００と同様に特徴点座標演算処理を行い、次のステップ３１２において、その時刻における移動速度Ｖｘ（ｔ）をステップ３０２と同様に行う。そして、ステップ３１４で移動速度Ｖｘ（ｔ）が後クリック閾値以上か否かを判断し、後クリック閾値以上であればステップ３２０へ移行し、後クリック閾値以上でなければステップ３１６へ移行する。
【０１７０】
ステップ３１６では、ステップ３０６と同様に予め定めた所定時間経過したか否かを判断し、所定時間経過した場合には、ステップ３１８で図１３に示すステップ２９２と同様の処理を行い、所定時間経過していない場合には、ステップ３１０へ戻って上記と同様の処理を繰り返す。
【０１７１】
ステップ３２０では、ステップ３００と同様に特徴点座標演算処理を行い、次のステップ３２２において、その時刻における移動速度Ｖｘ（ｔ）をステップ３０２と同様に行う。そして、ステップ３２４で移動速度Ｖｘ（ｔ）が前クリック閾値以上か否かを判断し、前クリック閾値以上であればステップ３２６へ移行し、前クリック閾値以上でなければステップ３２６へ移行する。
【０１７２】
ステップ３２６では、ステップ３０６と同様に予め定めた所定時間経過したか否かを判断し、所定時間経過した場合には、ステップ３２８で図１３に示すステップ２９６と同様の処理を行い、所定時間経過していない場合には、ステップ３２０へ戻って上記と同様の処理を繰り返す。
【０１７３】
また、ステップ３３０では、図１３に示すステップ２９８と同様の処理を行う。
【０１７４】
なお、図１３の場合には、所定時間は、後ストローク等の比較的時間を要する一連のクリック動作が終了するのに十分な時間に設定する必要があるが、図１５の場合においては、一連のクリック動作が終了するのに十分な時間である必要はなく、クリック動作を構成する動作（例えば前進クリックや後進クリック）を行うのに十分な時間であればよい。このため、図１５における所定時間は図１３における所定時間よりも短い時間に設定することができる。
【０１７５】
後進側クリック判定処理が終了すると、図４のステップ２２７へ移行し、クリック種別が判別不能であるか否かを判断する。これは、記憶部１８に記憶されたクリック種別変数を参照することにより判断することができる。
【０１７６】
そして、クリック種別が判別不能だった場合には、ステップ２１２へ戻り、上記と同様の処理を繰り返し、クリック種別が判別不能でなかった場合、すなわち、クリック種別が前進クリック、後進クリック、後ストローク、後進クリックの何れかであった場合には、ステップ２２８へ移行する。
【０１７７】
ステップ２２８では、クリック種別をＰＣ１２に通知する。これにより、ＰＣ１２は、通知されたクリック種別に対応する処理を行い、処理結果などをディスプレイ１４に表示させる。
【０１７８】
なお、各クリック動作の機能としては、次に示すようなものが考えられる。例えば、前進クリックは入出力デバイスとしてのマウスの左ボタンのシングルクリックに相当する機能、後進クリックはマウスの右ボタンのシングルクリックに相当する機能、後進ダブルクリックはマウスの左ボタンのダブルクリックに相当する機能、後ストロークはマウスの左ボタンによるドラッグ開始又はドラッグ終了に相当する機能とすることができる。
【０１７９】
このように、Ｘ方向の移動のみによって各種クリック動作を認識するため、自然かつ簡単な動作で様々なクリック動作を行うことができる。
【０１８０】
クリック種別をＰＣ１２に通知した後は、ステップ２２９において、所定時間待機する待機処理を行い、ステップ２００へ戻り、ポインティングモードに移行する。
【０１８１】
このように、クリック動作を行った後に一定時間待機してカーソルの移動やクリック種別の判定を行わないようにすることにより、利用者の腕がぶれたりすることによって利用者の意図に反して前進クリックが連続して入力されてしまったりするのを防ぐことができる。
【０１８２】
また、クリック動作がなされた場合には、自動的にポインティングモードに移行するため、利用者はポインティングモードへ移行するための特別な動作をする必要がなく、利便性を向上させることができる。
【０１８３】
【発明の効果】
以上説明したように、本発明によれば、自然かつ簡単な操作によって利用者による指示位置又は指示方向の判断精度を向上させることができる、という効果を有する。
【図面の簡単な説明】
【図１】本実施形態に係る情報表示装置の概略構成を示すブロック図である。
【図２】カメラと凹面鏡の位置関係、及び各カメラの撮像範囲を示す情報入力空間の側面図である。
【図３】（Ａ）は情報入力空間の正面図、（Ｂ）は情報入力空間の側面図である。
【図４】指示判断処理の内容を示すフローチャートである。
【図５】特徴点座標演算処理の内容を示すフローチャートである。
【図６】（Ａ）は各カメラによる撮像範囲、（Ｂ）は各カメラによって得られた各画像の一例を各々示すイメージ図、（Ｃ）は各画像上での指先位置の差から手の高さを求めるためのマップの内容の一例を示す線図である。
【図７】第１実施形態に係る基準点座標演算処理の内容を示すフローチャートである。
【図８】利用者による各種の指示動作における特徴点の回転中心の一例を各々示すイメージ図である。
【図９】利用者の体格の相違が反映された基準点の位置を示すイメージ図である。
【図１０】利用者が指示動作に用いる腕を変えた場合の基準点の移動を示すイメージ図である。
【図１１】特徴点及び基準点から指示位置の座標を演算する処理を説明するための、（Ａ）は側面図、（Ｂ）及び（Ｃ）は平面図である。
【図１２】（Ａ）は前進クリック動作、（Ｂ）は後進クリック動作、（Ｃ）は後進ダブルクリック動作、（Ｄ）は後ストローク動作を説明するためのイメージ図である。
【図１３】後進側クリック処理の内容を示すフローチャートである。
【図１４】各種クリック動作における指先位置や移動速度について説明するための線図である。
【図１５】後進側クリック処理の内容の他の例を示すフローチャートである。
【符号の説明】
１０ 情報表示装置
１２ ＰＣ
１４ 大画面ディスプレイ
１６ 制御部
１８ 記憶部
２０Ａ、２０Ｂ ビデオカメラ
２２ 撮像制御装置
２４ 利用者検出センサ
２８ 凹面鏡
２６Ａ，２６Ｂ 照明装置[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a pointing motion recognition device, and more particularly to a pointing motion recognition device for determining a position or a pointing direction specified by a user.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a user who has arrived in the vicinity of a display displaying predetermined information has taken an image of a situation in which an arbitrary position on the display has been pointed with a finger or the like by an imaging unit, and based on the image obtained by the imaging, the user has been captured. A hand pointing device that recognizes and determines a position on a display pointed by a user and displays a cursor or the like at a position indicated on the display is known.
[0003]
According to the hand pointing device described above, the user can give various instructions to the information processing device or input various information without touching input devices such as a keyboard and a mouse. The operation for using the information processing device can be simplified.
[0004]
Some of such hand pointing devices determine that the position indicated by the finger has been selected (clicked) when detecting that the finger has stopped for a predetermined time.
[0005]
However, in this case, even if the finger is stopped due to the determination, the position indicated by the finger is recognized as being selected, so that there is a problem that the operation result may be different from the user's judgment. Was.
[0006]
Further, based on the image processing result of the input image of the user's hand, a mode for performing pointing, that is, a mode for instructing movement of a cursor displayed on the display, and a mode other than that, for example, displayed at a cursor position There is also a proposal for switching between a mode for instructing icon selection and a mode for instructing icon selection (for example, see Patent Document 1).
[0007]
[Patent Document 1]
JP-A-10-207618
[0008]
[Problems to be solved by the invention]
However, in the technique described in Patent Document 1, for example, when the user recognizes the shape of the hand with the thumb and index finger extended, the mode is switched to the pointing mode, and the shape of the hand with the thumb closed and the index finger extended is recognized. For example, when the character is recognized, the mode is switched by recognizing a change in the shape of the hand, such as switching to a mode for instructing icon selection.
[0009]
In addition, since each mode is switched by changing the shape of the hand to various shapes, it is difficult to perform a natural operation according to human sensibility, and the user can use various hands corresponding to each mode. There is also a problem that it is difficult to switch the mode smoothly because the shape must be remembered in advance.
[0010]
SUMMARY An advantage of some aspects of the invention is to provide a pointing motion recognition device that can improve the accuracy of a user's pointing position or pointing direction determination by a natural and simple operation. And
[0011]
[Means for Solving the Problems]
In order to achieve the above object, the invention according to claim 1 provides an image pickup unit that images a user from a plurality of different directions, and a user based on an image obtained by the image pickup unit. Recognizing means for recognizing three-dimensional coordinates of a feature point whose position changes in accordance with a change in the designated position or the designated direction when performing an operation indicating the position or direction; and three of the feature points recognized by the recognizing means. Determining means for determining the position or direction indicated by the user based on the dimensional coordinates; and determining whether the feature point is in a stopped state based on the three-dimensional coordinates of the feature point recognized by the recognition means. A pointing mode for performing at least an operation of pointing to a specific position or direction based on the determination result by the stop state determining means. Switching means for switching between a click mode for performing a click operation including an operation of selecting an object existing at the position or direction of the object and a click determination for determining whether or not the click operation has been performed in the click mode Means.
[0012]
According to the present invention, the user is imaged from a plurality of different directions by the imaging unit, and the recognition unit performs an operation in which the user points to a specific position or direction based on the image obtained by the imaging of the imaging unit. The three-dimensional coordinates of a feature point whose position changes in accordance with a change in the designated position or the designated direction during the operation are recognized.
[0013]
As the characteristic point, for example, a point corresponding to the tip of a user's hand or finger or the tip of an indicator held by the user can be used. More specifically, the three-dimensional coordinates of the feature points are obtained, for example, by calculating the positions of the feature points on a plurality of images obtained by imaging from a plurality of directions, and recognizing the positions of the feature points on the plurality of images. can do.
[0014]
The judging means judges the position or the direction indicated by the user based on the three-dimensional coordinates of the feature point recognized by the recognizing means. The determination of the designated position and the designated direction can be performed based on, for example, the three-dimensional coordinates of the position of the user and the position of the feature point.
[0015]
The stopped state determining means determines whether or not the feature point is in the stopped state based on the three-dimensional coordinates of the feature point recognized by the recognition means. For example, stop when the three-dimensional coordinates of the feature points recognized within a predetermined time are all within a predetermined range, or when the moving amount or moving speed of the feature points is less than a predetermined threshold. The state can be determined. In other words, the stopped state refers to a case where the feature point is not completely stopped and other than when the feature point is completely stopped and can be regarded as almost stopped although the feature point is slightly moved. Including.
[0016]
The switching unit includes a pointing mode for performing at least an operation indicating a specific position or direction based on a determination result by the stop state determining unit, and an operation of selecting an object existing at the specified specific position or direction. Switches between click mode and click mode for performing click operation. The target object includes an area in which at least a part of the arbitrary information displayed on the display means is displayed, an operation button provided on a device that the user wants to operate, and the like.
[0017]
The switching unit switches to the click mode when the feature state is determined to be in the stopped state by the stopped state determination unit in the pointing mode, for example.
[0018]
The click determination means determines whether or not the click operation has been performed in the click mode. For example, when the feature point moves in the direction indicated by the user at a speed equal to or higher than a predetermined threshold value, it can be determined that the position indicated by the point has been selected, that is, clicked. Alternatively, it may be determined that a click has occurred when the feature point has moved along a predetermined locus or when the user's hand has changed to a shape corresponding to the click operation.
[0019]
In this way, when the feature point is stopped, the user does not recognize immediately that the click operation has been performed, but determines whether the click operation has been performed only in the click mode state. If a finger or the like is in a stopped state due to hesitating to make a determination, it is possible to prevent an erroneous recognition that an object existing at a specific position or direction pointed to has been selected. For this reason, it is possible to improve the determination accuracy of the click operation, and to prevent an operation result different from the user's intention.
[0020]
In addition, the mode is not switched according to the recognition result of the shape of the human hand, but is switched according to the stop state of the feature point, so that the processing can be prevented from becoming complicated, and the user can perform various operations. Since it is not necessary to remember various hand shapes corresponding to the modes in advance, the modes can be switched smoothly with a natural and simple operation.
[0021]
The invention according to claim 2, wherein the stop state determining means moves the characteristic point on a two-dimensional plane intersecting at least a direction designated by the user at a predetermined two-dimensional plane intersecting with the direction designated by the user. It is characterized in that it is determined whether or not the feature point is in a stopped state based on the quantity.
[0022]
According to the present invention, the stop state determination means does not determine whether the feature point is in the stop state based on the movement amount of the feature point in the three-dimensional space, but at least intersects at least with the direction specified by the user. It is determined whether or not the feature point is in a stopped state based on the movement amount of the feature point on the determined two-dimensional plane.
[0023]
For example, in a situation where the user points to a flat display means, the two-dimensional plane is set to a plane substantially parallel to the display means. In such a case, it is considered that the direction indicated by the user and the set two-dimensional plane often have a substantially orthogonal relationship. Also, in this case, the click operation by the user naturally includes an operation of moving the feature point in a direction indicated by the user, that is, a direction substantially orthogonal to the two-dimensional plane. Therefore, the stop state determination means ignores the movement of the feature point in a direction substantially orthogonal to the two-dimensional plane.
[0024]
As described above, the stop state determination means ignores the operation in the direction necessary for the click operation and sets the feature point in the stop state in consideration of only the movement amount in the direction necessary for the pointing operation, that is, the movement amount in the two-dimensional plane. Therefore, it is possible to accurately determine the stop state by a simple process.
[0025]
In the invention according to claim 3, the switching means switches to a click mode when the feature point is determined to be in a stopped state, and switches to the pointing mode when the click operation is performed in the click mode. It is characterized by the following.
[0026]
According to the present invention, the switching unit switches to the click mode when the feature point is determined to be in the stop state, and automatically switches to the pointing mode when a click operation is performed in this click mode, There is no need to perform a special operation for shifting to the pointing mode after the click operation is performed, and each mode can be switched with a simple and natural operation.
[0027]
In the invention according to claim 4, the switching means switches to a click mode when the feature point is determined to be in a stop state in the click mode, and determines that the feature point is not in a stop state in the click mode. Switching to the pointing mode in the event of a failure.
[0028]
According to this invention, the switching means automatically switches to the pointing mode when it is determined that the feature point is not in the stop state in the click mode, for example, when the movement amount of the feature point on the two-dimensional plane is equal to or more than a predetermined value. For this reason, the feature point is stopped due to hesitation, but even if you want to move to pointing operation without clicking, you do not need to perform any special operation to shift to pointing mode, so it is easy In addition, the mode can be switched to the pointing mode by a natural operation.
[0029]
The invention according to claim 5, wherein the click determining means is configured to determine the type of the click operation based on at least a movement amount of the feature point in a direction substantially orthogonal to a predetermined two-dimensional plane that intersects the direction specified by the user. Is determined.
[0030]
According to the present invention, the type of the click operation is determined based on the movement amount of the feature point in the direction substantially orthogonal to the two-dimensional plane, so that the user can enter the direction substantially orthogonal to the two-dimensional plane, that is, the direction indicated by the user. The click operation can be performed by a natural and simple operation of moving the feature point toward, and the convenience can be improved.
[0031]
According to a sixth aspect of the present invention, there is provided a display means for displaying arbitrary information and displaying a pointing position or a pointing direction by the user determined by the determining means with a cursor, and a display mode for the pointing mode and a click mode. And control means for controlling the display means so that the cursor is different in each case.
[0032]
According to the present invention, the display means displays arbitrary information, and displays the pointing position or the pointing direction by the user determined by the determining means with the cursor, and the control means controls the pointing mode and the click mode. The display means is controlled so that the cursor is different from the case of (1).
[0033]
The control means changes at least one of, for example, the shape, size, color, and pattern of the cursor between the pointing mode and the click mode. Thus, the user can easily recognize which mode is the current mode.
[0034]
The invention according to claim 7 is characterized in that the control means controls the display means such that a movement rate of the cursor in the click mode is smaller than that in the pointing mode.
[0035]
According to the present invention, the control unit controls the display unit such that the cursor movement rate in the click mode, that is, the ratio of the cursor movement amount to the feature point movement amount is smaller than in the pointing mode. In addition, it is possible to prevent the cursor position from fluctuating in conjunction with the movement of the user's arm.
[0036]
By the way, the operation of changing the indicated position or the indicated direction is generally an operation of rotating the position of the feature point in an arc shape around an arbitrary joint of the user. Therefore, when the user performs the above operation. Is moved substantially along a virtual spherical surface centered on the arbitrary joint. In the case of an operation of moving each of a plurality of joints, the center of the virtual spherical surface does not always coincide with a specific joint, but in any case, the position of the feature point changes the designated position or designated direction. In accordance with the operation of moving, it moves substantially along a virtual spherical surface centered on a substantially constant position.
[0037]
Therefore, when the user performs an operation of changing the designated position or the designated direction based on the three-dimensional coordinates of the feature points respectively recognized by the recognition means in a situation where the feature points are located in different places from each other. Calculating means for calculating three-dimensional coordinates of a point corresponding to the rotation center of the change in the position of the characteristic point, wherein the determining means determines the three-dimensional coordinates of the characteristic point recognized by the recognition means; The designated position or designated direction by the user may be determined based on the calculated three-dimensional coordinates of the point corresponding to the rotation center.
[0038]
Thereby, the three-dimensional coordinates of the position corresponding to the rotation center in the operation actually performed by the user as the instruction operation are calculated.
[0039]
Note that the calculation of the three-dimensional coordinates of the point corresponding to the rotation center by the calculation means may be performed, for example, by performing recognition by the recognition means a plurality of times while the user is performing an operation of changing the designated position or designated direction. This can be performed based on the three-dimensional coordinates at a plurality of locations on the movement locus of the feature point, which are obtained as a result. In addition, the calculation by the calculation means may use, for example, the least squares method or another known calculation method.
[0040]
Note that the determination of the designated position or the designated direction is specifically, for example, obtaining a direction in which a virtual line connecting a feature point and a virtual line connecting the point corresponding to the rotation center is obtained, or corresponds to the rotation center. After obtaining the three-dimensional coordinates of the reference point based on the three-dimensional coordinates of the point, the direction in which the virtual line connecting the feature point and the reference point extends can be obtained (for example, the direction in which the virtual line extends is determined as the designated direction). Then, the intersection between the object existing on the extension of the virtual line and the virtual line is determined as the designated position).
[0041]
Thus, when the user performs the operation of changing the designated position or the designated direction, the coordinates of the point corresponding to the rotation center of the position change of the characteristic point are determined to determine the designated position or the designated direction by the user. Accordingly, regardless of what kind of operation the user performs as the pointing operation, it is possible to improve the accuracy of the determination of the pointing position or the pointing direction by the user.
[0042]
Further, the three-dimensional coordinates of the point corresponding to the rotation center are obtained by calculation from the three-dimensional coordinates of the feature point, so that even if the point corresponding to the rotation center is not within the imaging range of the imaging means, the rotation The three-dimensional coordinates of the point corresponding to the center can be calculated, and the degree of freedom of setting the imaging range by the imaging means is improved.
[0043]
Considering that there is a possibility that the instruction operation itself by the user changes over time (the position of the point corresponding to the rotation center changes), the recognizing unit and the calculating unit perform It is preferable to repeatedly perform the recognition of the three-dimensional coordinates or the calculation of the three-dimensional coordinates of the point corresponding to the rotation center.
[0044]
Accordingly, even when the user's instruction operation itself changes, three-dimensional coordinates of a point corresponding to the rotation center of the change in the position of the feature point in the operation after the change can be obtained, and the user's instruction operation itself can be obtained. Irrespective of the change, the position or the direction indicated by the user can be accurately determined.
[0045]
Further, the calculation means can be configured to directly calculate the three-dimensional coordinates of a point corresponding to the rotation center of the change in the position of the feature point from the three-dimensional coordinates of the feature point recognized by the recognition means, for example. However, in the present invention, since the three-dimensional coordinates of the point corresponding to the rotation center are directly calculated from the three-dimensional coordinates of the feature point, the three-dimensional coordinates of the feature point used in the calculation are such that the feature point moves three-dimensionally. It must be data corresponding to the period of time. Therefore, there is a disadvantage that the amount of data used for the calculation tends to be large and the calculation is complicated.
[0046]
In consideration of this, the calculating means calculates the rotation in a plurality of planes based on the two-dimensional coordinates of the feature points in a plurality of mutually intersecting planes, which are obtained from, for example, the three-dimensional coordinates of the feature points recognized by the recognition means. The two-dimensional coordinates of the point corresponding to the center may be calculated, and the three-dimensional coordinates of the point corresponding to the rotation center may be calculated from the two-dimensional coordinates of the reference point in a plurality of planes.
[0047]
According to the present invention, two-dimensional coordinates of a point corresponding to the rotation center in a plurality of planes are respectively calculated based on two-dimensional coordinates of feature points in a plurality of (for example, two) planes intersecting each other, and based on these. Therefore, the three-dimensional coordinates of the point corresponding to the rotation center are calculated. Therefore, even during the period in which the position of the feature point changes only in any one of the plurality of planes, the feature point is recognized in this period. The two-dimensional coordinates of a point corresponding to the rotation center in the plane can be calculated based on the two-dimensional coordinates of the feature point in the plane obtained from the three-dimensional coordinates of the feature point. Therefore, the amount of data used for calculating the individual two-dimensional coordinates of the point corresponding to the rotation center can be reduced, and the calculation is simplified.
[0048]
In the present invention, calculating the two-dimensional coordinates of a point corresponding to the rotation center in a plurality of planes, for example, means that the three-dimensional coordinates of the feature point recognized by the recognition unit are at least one of the plurality of planes. It is determined whether or not the three-dimensional coordinates of a feature point are changing in a specific plane, based on the two-dimensional coordinates of the feature points in a specific plane. Thus, it can be realized by calculating two-dimensional coordinates of a point corresponding to the rotation center in a specific plane.
[0049]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings.
[0050]
FIG. 1 shows a schematic configuration of an information display device 10 having a function as a pointing action recognition device according to the present invention.
[0051]
The information display device 10 includes a personal computer (PC) 12. The PC 12 includes a CPU, a ROM, a RAM, an input / output port, and an HDD (all not shown). The PC 12 is connected to a large screen display 14 (hereinafter, referred to as a display 14) and a control unit 16. Note that the PC 12 corresponds to the control means of the present invention, and the display 14 corresponds to the display means of the present invention.
[0052]
The display 14 includes an LCD, a plasma display, or an organic EL display, is mounted on a wall, and various information is displayed by the PC 12 according to information input to the PC 12 from the control unit 16.
[0053]
The control unit 16 is configured such that a CPU, a ROM, a RAM, and an input / output port (all are not shown) are connected to each other via a bus, and a storage unit 18 is connected.
[0054]
The storage unit 18 stores various data, programs, and the like, and includes a rewritable nonvolatile information storage medium, such as a hard disk and a RAM connected to a backup power supply. It is desirable.
[0055]
In the present embodiment, the user arrives near the disposition position of the display 14 (the location shown in FIGS. 2 and 3B in front of the display 14). 80 "), an instruction operation (pointing operation) for pointing to an arbitrary position in a corresponding display area provided on the display screen of the display 14, and a manual operation is performed from the state in which the instruction operation is performed. By performing a click operation or the like including an operation of moving in a forward direction (a direction approaching the display 14) or a backward direction (a direction away from the display 14), that is, an X direction (the forward direction is defined as positive). Desired information is displayed in the display area. The display 14 displays a cursor indicating the position pointed by the user. This cursor moves according to the position pointed to by the user.
[0056]
Further, the control unit 16 is connected to an imaging control device 22 to which two video cameras 20A and 20B are connected and a user detection sensor 24, respectively. Note that the video cameras 20A and 20B correspond to the imaging means according to the present invention.
[0057]
The user detection sensor 24 is composed of, for example, a load cell, and is embedded in the floor at a position where the user stays (a predetermined position in front of the display 14) when performing an instruction operation or a click operation.
[0058]
As shown in FIG. 3A, lighting devices 26A and 26B that illuminate a user arriving at the information input space are provided above a space (information input space) in front of the display 14.
[0059]
The illumination light emitted from the illumination devices 26A and 26B may be visible light or near-infrared light. However, when the illumination light is near-infrared light, the video cameras 20A and 20B may be used. Incorporates an area sensor consisting of a CCD or the like that has sensitivity to near-infrared light, and transmits only light having a wavelength in the near-infrared region to the light incident side of an imaging lens that forms incident light on the light receiving surface of the area sensor. A video camera having a configuration in which a filter is provided may be used.
[0060]
On the other hand, a concave mirror 28 having a concave reflecting surface corresponding to a part of the paraboloid of revolution (specifically, a part that is not rotationally symmetric) is arranged above the information input space. The video cameras 20A and 20B move the concave mirror 28 to a position near the position where the positional relationship with the concave mirror 28 becomes substantially equal to the positional relationship between the reflector and the primary radiator in the offset parabolic antenna and different positions along the XZ plane. Are arranged in such a direction as to capture an image of the information input space from a position directly above the information input space.
[0061]
Since the concave mirror 28 has an optical performance of reflecting a parallel light beam incident from the information input space side to the video cameras 20A and 20B as a convergent light beam, the imaging range of the video cameras 20A and 20B (FIG. 2 and FIG. 3 (B), the width along the horizontal direction of the video camera 20A is a range bounded by a dashed line, and the video camera 20B is a range bounded by a broken line. Direction).
[0062]
Further, in the present embodiment, the arrangement position of the video camera 20A is determined so that the information input space is accurately imaged from directly above (the boundary of the imaging range indicated by a dashed line in the figure is along the vertical direction). On the other hand, since the arrangement positions of the video cameras 20A and 20B are shifted along the XZ plane, the imaging range of the video camera 20B is shifted in the X direction with respect to the imaging range of the video camera 20A (see FIG. )), An image is captured from a slightly obliquely upper part of the information input space (the boundary of the imaging range indicated by a broken line in each figure is slightly inclined with respect to the vertical direction).
[0063]
Therefore, in the information display device 10 according to the present embodiment, the area (recognition area) in which the operation (instruction operation or pointing operation) by the user can be recognized is the imaging range of the video camera 20A and the imaging range of the video camera 20B. The overlapping area, that is, the area surrounded by the two-dot chain line in FIGS. 2 and 3 (the area indicated by hatching in FIG. 2), and the width of the recognition area along the horizontal direction is also approximately along the vertical direction. It will be constant.
[0064]
As is clear from FIG. 3B, in the present embodiment, the user who has arrived at the information input space is out of the recognition area, and when the user performs an instruction operation or a click operation, the user's hand moves. The imaging range of the video cameras 20A and 20B (the shape and size of the concave mirror 28, the positional relationship between the concave mirror 28, and the video cameras 20A and 20B, and the like) is determined so that only the video signal falls within the recognition area.
[0065]
In addition, a far-infrared camera is used as an imaging unit (camera), and far-infrared rays based on the human (user) body temperature and a temperature separated from the human body temperature by a predetermined value or more (the temperature is higher than the body temperature but lower than the predetermined temperature). May be used to discriminate a person from a background by far-infrared light radiated from a floor set or adjusted in temperature.
[0066]
Next, the operation of the present embodiment will be described. In the present embodiment, each time the user detection sensor 24 detects the arrival of a user, the control unit 16 executes an instruction determination process.
[0067]
Hereinafter, this instruction determination processing will be described with reference to the flowchart of FIG. The instruction determination process is terminated when the user detection sensor 24 detects that the user has left.
[0068]
In step 200, a pointing mode setting process is performed. Specifically, a setting process for a cursor displayed on the display 14 is performed.
[0069]
In the setting process related to the cursor, the PC 12 is notified of the cursor movement rate, that is, the ratio of the amount of movement of the cursor to the amount of movement of the user's fingertip and the like, and cursor information including information on the design such as the shape, size, and color of the cursor. I do. Note that the PC 12 may store in advance information relating to the cursor movement rate and the cursor design corresponding to the pointing mode, and notify the controller 16 of only the pointing mode.
[0070]
In the next step 202, feature point coordinate calculation processing is performed.
[0071]
Hereinafter, the feature point coordinate calculation processing will be described with reference to the flowchart of FIG.
[0072]
In step 230, image data representing the image A output from the video camera 20A is fetched, and in the next step 232, the fetched image data is stored in the RAM 22C or the like. In step 234, it is determined whether or not the image data of the image A has been captured and stored in the previous processing cycle. When the user arrives at the information input space and performs the feature point coordinate calculation process for the first time, since the image data of the image A is not stored, the determination in step 234 is denied and the feature point coordinate calculation process ends. However, at the time of the second or subsequent execution of the processing, since the image data of the image A captured in the previous processing cycle is stored in the RAM or the like of the control unit 16, the determination is affirmed and the process proceeds to step 236.
[0073]
In step 236, using the image data of the image A captured in the previous processing cycle and the image data of the image A captured this time, the difference between the images represented by both image data is calculated for each pixel. By this calculation, among the objects existing within the imaging range of the video camera 20A, only the pixels corresponding to the objects that have moved between the previous processing cycle and the present become “there is a difference”. By extracting only pixels having "difference", an image area corresponding to an object having motion is extracted.
[0074]
In addition, when the user's hand is located in the recognition area, not only when the user performs a click operation such as moving the hand back and forth, but also performs an instruction operation in which the user points to a specific location. It is difficult to completely stop the hand even in the air, even when the user is in the air, so in most cases, the area corresponding to the user's hand has moved between the previous processing cycle and the present. Is extracted as an image area corresponding to the object.
[0075]
In the next step 238, it is determined whether or not an image area corresponding to the user's hand (for example, an image area surrounded by a thick line in FIG. 6B) has been extracted by the processing in step 236. Since the objects existing in the recognition area are illuminated by the illumination devices 26A and 26B, when the user's hand exists in the recognition area, the image A and the image B include the user. Appears as a continuous area composed of high-luminance pixels and having an area equal to or larger than a predetermined value. Therefore, if a user's hand is present in the recognition area, an image area corresponding to the hand can be easily extracted.
[0076]
If the image area corresponding to the moving object is not extracted in step 236, and if it is determined that the image area extracted in step 236 is not the image area corresponding to the hand of the user, the recognition area It can be determined that the user's hand does not exist within the above, so the determination in step 238 is denied and the feature point coordinate calculation processing ends.
[0077]
On the other hand, if the determination in step 238 is affirmative, the process proceeds to step 240. In the present embodiment, a fingertip of the user's hand is used as a feature point for recognizing an instruction operation and a click operation by the user. In step 240, the feature point P on the image A represented by the image data captured this time is used. (Point P shown in FIG. 6 (B) _A (See two-dimensional coordinates).
[0078]
In the present embodiment, since the arrangement position of the video camera 20A is determined so that the information input space is accurately imaged by the video camera 20A from directly above, the feature point moves along the vertical direction (X coordinate value and (The Y coordinate value is moved without change), the point P on the image A corresponding to the feature point _A Is almost unchanged.
[0079]
Thus, the point P on the image A _A Corresponds substantially to the position of the feature point in the XY plane (horizontal plane) of the XYZ coordinate system, and the two-dimensional coordinates (x, y) of the feature point in the horizontal plane are obtained by the calculation in step 240. .
[0080]
In step 244, image data representing the image B output from the video camera 20B is fetched, and in the next step 246, the fetched image data is stored in the RAM or the like of the control unit 16.
[0081]
In step 248, using the image data of the image B captured in the previous processing cycle and the image data of the image B captured this time, the difference between the images represented by both the image data is calculated for each pixel, By extracting only pixels, an image area corresponding to a moving object is extracted. Then, in step 250, the characteristic points (point P shown in FIG. _B (See two-dimensional coordinates).
[0082]
As shown in FIG. 6A, the imaging range of the video camera 20B is deviated in the X direction with respect to the imaging range of the video camera 20A. As an example, the position of the feature point in FIG. _A And point P _B As shown in the figure, the position shifts along the X direction (deviation d occurs).
[0083]
Also, the magnitude of the deviation d changes according to the height position (Z direction position) of the feature point. If the feature point is located on the reflection surface of the concave mirror 28, the deviation d becomes 0, and the feature point Is located on the floor of the information input space, the deviation d becomes maximum (the maximum value of the deviation d matches the shift amount of the imaging range of the video cameras 20A and 20B on the floor of the information input space. ). Therefore, there is a relationship as shown in FIG. 6C between the deviation d and the feature point height (hand height) H. In the present embodiment, a map indicating the relationship between the deviation d and the height H of the feature point as shown in FIG. 6C is stored in the storage unit 18 in advance.
[0084]
Based on the above, in step 252, the deviation d along the X direction between the position of the feature point on the image A and the position of the feature point on the image B is calculated. In the next step 254, the deviation d is stored in the storage unit 18. Based on the map representing the relationship between the deviation d and the height H of the characteristic point, ie, the height H of the characteristic point corresponding to the deviation d obtained in step 252, that is, the height (coordinate of the characteristic point in the XYZ coordinate system) Determine the value z).
[0085]
Then, in step 256, the three-dimensional coordinates (x, y, z) of the feature point are stored in the storage unit 18 or the like in association with the current time (operation time). Since the feature point coordinate calculation processing is repeatedly executed, the three-dimensional coordinates of the feature points calculated by the processing are sequentially updated according to changes in the user's posture and movement.
[0086]
In addition, when the height H of the obtained feature point is extremely high or extremely low, there is a very high possibility that erroneous recognition of the feature point or the like has occurred. If the recognition area shown in FIG. 3 is higher than the upper boundary or lower than the lower boundary, exception processing such as determining that the user's hand does not exist in the recognition area may be performed. Good.
[0087]
Further, the feature point coordinate calculation processing may be performed using, for example, a well-known DLT (Direct Line Transformation) method or a stereo method.
[0088]
When the feature point coordinate calculation processing is completed, the process proceeds to step 204 of the instruction determination processing (FIG. 4), and a reference point coordinate calculation processing for calculating three-dimensional coordinates of the reference point is performed.
[0089]
Hereinafter, the reference point coordinate calculation processing will be described with reference to the flowchart of FIG.
[0090]
In step 260, the three-dimensional coordinate data (x, y, z) of the feature point calculated by the above-described feature point coordinate calculation process (FIG. 5) and stored in the storage unit 18 is referred to for a predetermined time. The three-dimensional coordinate data of the feature points calculated in the period from the previous time to the present time is a predetermined number (for example, a number sufficient for executing the process of calculating the coordinates corresponding to the rotation center of the position change of the feature points (details will be described later)) : For example, 16) or more. When the user first arrives in the information input space, the number of executions of the feature point coordinate calculation process is less than a predetermined number, so the number of three-dimensional coordinate data of the feature points satisfying the above condition does not reach the predetermined number, It is very difficult to calculate the coordinates of the point corresponding to the rotation center with high accuracy.
[0091]
For this reason, if the determination at step 260 is negative, the process proceeds to step 274, and based on the three-dimensional coordinates of the feature point stored in the storage unit 18, a point corresponding to the rotation center of the position change of the feature point is determined. Is estimated and calculated. This estimation calculation can be performed, for example, as follows.
[0092]
That is, as shown in FIG. 8A, for example, the user bends the elbow to a right angle or an angle close to a right angle as an instruction operation, and keeps the hand at a relatively low position (low position). When an operation of changing the designated position by moving only the further part is performed, the position of the feature point often changes about the elbow joint as a center of rotation in many cases.
[0093]
As an example, as shown in FIG. 8 (C), the user extends the arm relatively straight forward as an instruction operation, and keeps the hand at a relatively high position (high position), and When the operation of changing the designated position by moving is performed, the position of the feature point often changes with the shoulder joint as the center of rotation.
[0094]
Further, as shown in FIG. 8B, as an example, the user performs an operation of changing the indicated position in a state where the hand is maintained at an intermediate position between the high position and the low position (middle position). When performing, the positions of the feature points often change about the position slightly behind the shoulder joint as the center of rotation as the elbow and shoulder joints are moved.
[0095]
Therefore, the estimation calculation in step 274 determines whether the posture (state) of the user is the low position / middle position / high position, for example, and determines the three-dimensional coordinates corresponding to the rotation center of the change in the position of the feature point. One of the three-dimensional coordinates of the position corresponding to the joint of the elbow, the three-dimensional coordinates of the position corresponding to the joint of the shoulder, and the three-dimensional coordinates of the position slightly behind the joint of the shoulder. ) Can be performed by selectively performing the calculation.
[0096]
Whether the posture (state) of the user is in the low position / middle position / high position can be determined from the ratio between the height of the user and the height (coordinate value z) of the characteristic point, and the user's posture is roughly determined. Calculates the deviation d along the X direction of the position of the user's head on the images A and B, as in steps 252 and 254 of the feature point coordinate calculation processing (FIG. 5) described above. Then, it can be obtained by using a map representing the relationship between the deviation d and the height H.
[0097]
When the three-dimensional coordinates corresponding to the rotation center of the position change of the feature point are estimated and calculated, the process proceeds to step 276, and the three-dimensional coordinates of the point corresponding to the rotation center according to the distance to the display 14 and the size of the display 14. By correcting (X, Y, Z), the three-dimensional coordinates (X, Y, Z) of the reference point are calculated, and the calculated three-dimensional coordinates (X, Y, Z) of the reference point are stored in the storage unit 18. Then, the reference point coordinate calculation processing is completed.
[0098]
On the other hand, when a short period of time elapses after the user arrives at the information input space and starts the instruction operation, in the reference point coordinate calculation processing, the feature points calculated within a period from a predetermined time before the current time to the current time are calculated. When the three-dimensional coordinate data becomes equal to or more than the predetermined number, the determination in step 260 is affirmed. If the determination in step 260 is affirmative, the process moves to step 262, and among the three-dimensional coordinate data group of the feature points that meet the above conditions, the feature points corresponding to the period during which the user has performed a click operation described later. It is determined whether there is three-dimensional coordinate data. In this determination, for example, the respective data of the three-dimensional coordinate data group of the feature point to be determined are arranged in chronological order in the order of the operation time, and data in which the coordinate value x suddenly changes by a predetermined value or more exists in each data. The search can be performed by searching for data as to whether or not there is data corresponding to each click operation.
[0099]
The position of the feature point corresponding to the period in which the user performed the click operation causes a decrease in the calculation accuracy in the calculation of the coordinates of the point corresponding to the rotation center of the change in the position of the feature point, which will be described later. If the determination is affirmative, the process proceeds to step 264. After performing processing for excluding the three-dimensional coordinate data of the corresponding feature point from the calculation target in the calculation of the coordinates of the point corresponding to the rotation center, the process proceeds to step 266. I do. If the determination in step 262 is negative, the process proceeds to step 266 without performing any processing.
[0100]
In step 266, it is determined whether or not the three-dimensional coordinate data group of the feature point to be operated has a change in coordinates in the horizontal plane (in the XY plane) and in the vertical plane (in the YZ plane). If this determination is denied, for example, it is a case where the user continues to indicate a certain position, and the rotation of the position change of the feature point is performed using the data in such a state. Calculating the center leads to a decrease in the accuracy of the determination of the position indicated by the user. Therefore, if the determination in step 266 is negative, the process proceeds to step 274, and the above-described estimation calculation is performed.
[0101]
On the other hand, if the determination in step 266 is affirmative, the process moves to step 268, where the rotation center of the change in the position of the feature point is determined based on the movement locus of the feature point represented by the three-dimensional coordinate data group of the feature point to be calculated. The three-dimensional coordinates (X, Y, Z) of the corresponding point are calculated.
[0102]
Hereinafter, the calculation in step 268 will be described in detail. The position of a feature point can be considered to move along a surface of a virtual sphere centered on a specific position in space (this position differs depending on the user's motion). A spherical surface having a radius R centered on the coordinates (X, Y, Z) can be expressed by the following equation (1).
(Xx) ² + (Y-Y) ² + (Xx) ² = R ² … (1)
The coordinates (x, y, z) in the expression (1) are coordinates of an arbitrary point on the spherical surface. The n three-dimensional coordinate data of the feature points are represented by (x _i , Y _i , Z _i (I = 1, 2,..., N), and these values are substituted into the equation (1) to obtain the extreme value of the least squares method using the coordinate value X and the radius R, as follows.
[0103]
(Equation 1)

[0104]
The following expression (3) is obtained from the above result.
[0105]
(Equation 2)

[0106]
By substituting equation (3) into equation (2) and rearranging, the following equation (4) is obtained.
[0107]
[Equation 3]

[0108]
When the coordinate values Y and Z are solved in the same manner as above, the following equations (5) and (6) are obtained.
[0109]
(Equation 4)

[0110]
Subsequently, equations (4), (5), and (6) are made simultaneous to solve coordinate values X, Y, and Z. When these are expressed by a determinant, the following equation (7) is obtained.
[0111]
(Equation 5)

[0112]
The solution of the equation (7) is as shown in the following equations (8) to (10).
[0113]
(Equation 6)

[0114]
Therefore, step 268 is performed in three-dimensional coordinate data (x _i , Y _i , Z _i ) Into the equations (8) to (10), the three-dimensional coordinates (X, Y, Z) of the point corresponding to the rotation center can be obtained. The radius R can be obtained by substituting the three-dimensional coordinate data of the feature point to be calculated and the calculation results of equations (8) to (10) into equation (3). Step 268 corresponds to the calculating means.
[0115]
In the next step 270, it is determined whether or not the three-dimensional coordinates of the point corresponding to the rotation center of the change in the position of the feature point have been calculated before (between the present user's arrival in the information input space and the present). judge. When the determination is denied, the process proceeds to step 276 without performing any processing. When the determination is affirmed, the process proceeds to step 272, and the three-dimensional coordinates of the point corresponding to the previously calculated rotation center are also determined. The three-dimensional coordinates (X, Y, Z) of the point corresponding to the rotation center are finally determined by taking into account (for example, by calculating a weighted average value).
[0116]
Then, in the next step 276, the three-dimensional coordinates (X, Y, Z) of the point corresponding to the rotation center are corrected based on the distance to the display 14 and the size of the display 14, thereby obtaining the three-dimensional coordinates of the reference point. (X, Y, Z) is calculated and stored in the storage unit 18.
[0117]
As described above, in the present embodiment, after a certain time has elapsed since the user arrived at the information input space and started the instruction operation (the characteristic calculated within the period from the current time to the current time by a predetermined time). After the three-dimensional coordinate data of the point becomes equal to or more than a predetermined number), the three-dimensional coordinates of the point corresponding to the rotation center of the position change of the feature point are calculated by the least square method based on the movement locus of the feature point. Since the position indicated by the user is determined using the three-dimensional coordinates of the reference point set based on the three-dimensional coordinates of the point corresponding to the center of rotation, the user performs the pointing operation with reference to FIGS. 9), the reference point can be set without being affected by any of the various motions (postures) shown in FIG. It can be set to an appropriate position, depending on the user The shown position may be accurately determined.
[0118]
In addition, since the reference point coordinate calculation process (and the feature point coordinate calculation process) is repeatedly performed, for example, as shown in FIGS. 10A and 10B, the arm used by the user for the instruction operation changes during use. Also, when the operation performed by the user as the instruction operation changes during the use (for example, when the position changes from the high position to the middle position or the low position), the reference point is set following the change. The position can be switched to an appropriate position, and the position designated by the user can always be determined with high accuracy.
[0119]
When the reference point coordinate calculation processing is completed, the process proceeds to step 206 in the flowchart of FIG. 4, and the coordinates of the cursor to be displayed on the display 14 are calculated.
[0120]
Specifically, the three-dimensional coordinates (x, y, z) of the feature point calculated in the preceding feature point coordinate calculation process and the three-dimensional coordinates (X, Y) of the reference point calculated in the reference point coordinate calculation process , Z), the reference point P is used as a line representing the pointing direction by the pointing action of the user, as shown in FIG. ₀ And feature point P _X The direction in which the virtual line (see the virtual line 54 in FIG. 11) extends is determined, and the position (pointed position) pointed by the user is defined as the intersection of the plane including the display surface of the display 14 with the set virtual line ( The coordinates (two-dimensional coordinates) of the point S shown in FIG. 11) are calculated.
[0121]
Then, the PC 12 is notified of the calculated coordinates of the cursor position. The PC 12 displays a cursor based on the cursor information set in step 200 at the coordinates specified by the control unit 16. Thus, the cursor is displayed at the position only by the user pointing his fingertip or the like toward the display 14. Step 206 corresponds to the determining means of the present invention.
[0122]
In the next step 208, based on the three-dimensional coordinates (x, y, z) of the feature point calculated in the feature point coordinate calculation processing, it is determined whether or not the feature point is in a stopped state. In the present embodiment, the movement of the feature point in the X direction is ignored, and it is determined whether or not the feature point is in the stop state by focusing only on the movement of the YZ plane as a two-dimensional plane. That is, it is determined whether or not the feature point is in a stopped state based on the two-dimensional coordinates (y, z). Step 208 corresponds to the stop state determining means and the switching means of the present invention.
[0123]
Specifically, based on the two-dimensional coordinates (y, z) of the feature point calculated within the predetermined time from the present to the past, the movement amount of the feature point within the predetermined time on the YZ plane is calculated, and based on this movement amount. To determine whether or not it is in a stopped state.
[0124]
For example, if the YZ component vector of the feature point at the current time (time t) is represented by r (y, z, t) = r (t), and the sampling interval (calculation interval) of the feature point is Δt, the past n times (N is a natural number) | r (t) −r (t−Δt) |, | r (t) −r, which is the difference (distance) between each of the calculated YZ component vectors of the feature points and the current YZ component vector (T−2Δt) |, | r (t) −r (t−3Δt) |,... | R (t) −r (t−nΔt) | It is determined that the point is in the stopped state, and otherwise, it can be determined that the feature point is not in the stopped state.
[0125]
Also, the moving speed of the feature point on the YZ plane at the current time (time t) is represented by v (t) = {r (t) −r (t−Δt)} / Δt, and the YZ of the feature point calculated n times in the past Each of | v (t) |, | v (t−Δt) |, | v (t−2Δt) |,... | V (t−nΔt) | In the case of, the feature point may be determined to be in the stopped state, and in other cases, it may be determined that the feature point is not in the stopped state. Note that the above-described predetermined threshold is set to a value that can be considered to be substantially in a stopped state even if the feature point slightly moves on the YZ plane.
[0126]
In this way, the operation in the X direction, that is, the direction required for the click operation is ignored, and it is determined whether or not the feature point is in the stop state based on only the movement amount of the feature point on the YZ plane necessary for the pointing operation. The stop state can be accurately determined by simple processing.
[0127]
If it is determined in step 208 that the feature point is not in the stopped state, the process proceeds to step 202, and the same processing as described above, that is, the processing of steps 202 to 208 is repeated until the feature point is determined to be in the stopped state. During this time, the coordinates of the cursor position are sequentially calculated and reported to the PC 12, and the PC 12 controls the display 14 so that the cursor moves at the set movement rate.
[0128]
On the other hand, if it is determined that the vehicle is in the stop state, the process proceeds to step 210. In step 210, a click mode setting process is performed. Specifically, similarly to step 200, a setting process and the like relating to the cursor displayed on the display 14 are performed.
[0129]
In the setting process related to the cursor, the moving rate of the cursor is set to p times (0 ≦ p ≦ 1) of the pointing mode. Here, when p = 1, the movement rate is the same as that in the pointing mode, and when p = 0, the cursor does not move at all.
[0130]
In the click mode, the main purpose is to move the cursor and not to click on the pointed position, so that it is often prioritized to be able to click the position exactly. Therefore, in the click mode, it is preferable that the movement rate is lower than that in the pointing mode, that is, p is less than 1. Thus, it is possible to prevent the user's arm from being shaken, thereby preventing the cursor position from being changed in conjunction with the movement and the click position from being shifted, thereby improving the convenience.
[0131]
Also, the design of the cursor is set to a design different from that in the pointing mode. For example, when the design of the cursor in the pointing mode is an arrow or cross design, in the click mode, the shape of the finger or the shape of the button is set. Note that only the color of the cursor may be different.
[0132]
By making the cursor different between the pointing mode and the click mode in this way, the user can easily and accurately recognize whether the current mode is the pointing mode or the click mode.
[0133]
In step 212, feature point coordinate calculation processing is performed as in step 202, and in next step 214, reference point coordinate calculation processing is performed as in step 204.
[0134]
Then, in step 216, as in step 206, the three-dimensional coordinates (x, y, z) of the feature point calculated in the feature point coordinate calculation process and the three-dimensional coordinates of the reference point calculated in the reference point coordinate calculation process The coordinates of the cursor position to be displayed on the display 14 are calculated based on the coordinates (X, Y, Z), and are notified to the PC 12. As a result, a cursor having a shape corresponding to the click mode is displayed on the display 14 at the position indicated by the user.
[0135]
By the way, various actions can be adopted as the click action. In the present embodiment, an action in which the user moves the hand quickly forward (see FIG. 12A, hereinafter referred to as “forward click”), In addition to an operation in which the user moves the hand quickly backward (see FIG. 12B, hereinafter referred to as “reverse click”), an operation of repeating the operation in which the user quickly moves the hand backward twice is performed (see FIG. 12C ), Hereinafter referred to as “reverse double-click”), an operation in which the user moves his / her hand backward, then moves his / her hand forward and returns to the original position (see FIG. 12 (D), hereinafter referred to as “back stroke”). Click).
[0136]
The click operation is an extremely natural operation such as an operation of pointing to and selecting a specific portion on the display surface of the display 14, and the user can perform the click operation without feeling uncomfortable.
[0137]
Further, in the initial operation of such a click operation, an operation in which the user's hand moves in a forward direction, that is, a direction approaching the display 14, or an operation in which the user's hand moves in a backward direction, that is, a direction in which the user's hand moves away from the display 14 Will always be included.
[0138]
Therefore, in the next step 218, it is determined whether or not the initial operation in the click operation has been performed by determining whether or not the moving speed of the characteristic point in the X direction is equal to or greater than a predetermined threshold based on the X coordinate of the characteristic point. I do.
[0139]
Specifically, the X component vector in the X direction at the time t of the feature point is represented by x (t), and the moving speed of the feature point at the time t in the X direction is Vx (t) =） x (t) −x In the case of (t−Δt)｝ / Δt, it is determined whether or not the moving speed Vx (t) is equal to or greater than a predetermined previous click threshold or equal to or greater than a predetermined later click threshold.
[0140]
Here, the previous click threshold is set to a value that is equal to or higher than the previous click threshold when the feature point moves quickly in the forward direction. Similarly, the post-click threshold is set to a value that is equal to or higher than the pre-click threshold when the feature point moves quickly in the backward direction. In addition, each threshold value may be set so that the absolute value of the previous click threshold value and the absolute value of the subsequent click threshold value are the same value, or may be different values.
[0141]
If the moving speed Vx (t) is neither equal to or higher than the previous click threshold nor equal to or higher than the post-click threshold, that is, if the initial operation of the click operation is not performed, the process proceeds to step 220, and the moving speed Vx (t) is set. If it is equal to or greater than the previous click threshold or equal to or greater than the subsequent click threshold, that is, if the initial operation of the click operation is performed, the process proceeds to step 222.
[0142]
In step 220, as in step 208, the feature points are calculated based on the three-dimensional coordinates (x, y, z) of the feature points calculated by the feature point coordinate calculation process, more specifically, based on the two-dimensional coordinates (y, z). It is determined whether or not it is in the stop state. Note that the processing in step 220 corresponds to the stop state determining means and the switching means of the present invention.
[0143]
At this time, the threshold may be set to a value different from the predetermined threshold used in step 208, and it is preferable to set the threshold used in step 220 to a value larger than the threshold used in step 208. Thus, it is possible to prevent the feature point from fluctuating on the YZ plane due to the fluctuation of the user's hand immediately before performing the click operation, and erroneously shifting to the pointing mode.
[0144]
As described above, it is not possible to shift to the pointing mode unless the movement amount of the feature point on the YZ plane is larger than in the pointing mode, so that the usability of the user can be improved.
[0145]
In addition, when the user temporarily stops the movement of the hand and shifts to the click mode to enter the click operation, but wants to shift to the pointing mode again, it is only necessary to move the hand to a certain extent in the YZ plane. It is not necessary to perform a special operation such as changing the shape of the hand to shift to the mode, and the mode can be switched to the pointing mode with a simple and natural operation.
[0146]
On the other hand, if it is determined that the initial operation of the click operation has been performed, it is determined in step 222 whether or not the moving direction of the feature point is forward. This can be determined based on whether the moving speed Vx (t) is positive or negative. When the moving direction of the feature point is forward, that is, when the moving speed Vx (t) is a positive value, the process proceeds to step 224, and when the moving direction is backward, that is, when the moving speed Vx (t) is negative. If the value is a value, the process proceeds to step 226.
[0147]
In step 224, a value (for example, “1”) representing the forward click is substituted for the click type variable representing the click type, and stored in the storage unit 18.
[0148]
The forward click operation is, as shown in FIG. 12A, an operation in which the state where the user's fingertip as a feature point is stationary is set as an initial position, the fingertip is moved forward, and then returns to the original position. In other words, the basic operation is to move the fingertip forward once and then move the fingertip backward, but in the present embodiment, the forward click operation is performed only when the feature point moves forward at a speed equal to or higher than the predetermined threshold. Treat as done.
[0149]
This is because the user may feel that the reaction is slow if the user detects that the fingertip has moved backward and recognizes that it is a forward click, and performs processing corresponding to the forward click. It is. In this way, by assuming that the forward click operation is performed only when the feature point has moved forward at a speed equal to or higher than the predetermined threshold value, the user can operate without feeling uncomfortable.
[0150]
If there is no problem in the reactivity, a forward click may be recognized by detecting that the fingertip is moved backward after the fingertip is moved forward.
[0151]
On the other hand, if the moving direction is backward, a reverse click determination process is performed in step 226. Hereinafter, this processing will be described with reference to the flowchart shown in FIG.
[0152]
As the click operation in which the initial operation is on the reverse side, as shown in FIGS. 12B to 12D, there are a reverse click, a reverse double click, a reverse stroke, and the like. Therefore, the trajectory of the feature point in the X direction during a predetermined time after the initial operation, that is, the X coordinate, is determined, and the click operation on the reverse side is determined based on the trajectory. I do.
[0153]
For this reason, as shown in FIG. 13, in step 280, the same feature point coordinate calculation processing as in step 208 is performed first.
[0154]
In the next step 282, it is determined whether or not a predetermined time has elapsed. If the predetermined time has not elapsed, the flow returns to step 280 to perform the feature point coordinate calculation process again. This process is performed every Δt time. Note that the predetermined time is set to a time sufficient for ending a series of click operations requiring a relatively long time for a click operation such as a rear stroke or a reverse double click.
[0155]
After a lapse of a predetermined time, the process proceeds to step 284, and only the X coordinate of the three-dimensional coordinates (x, y, z) of the feature point calculated in step 280 is used, and the moving speed Vx at each time is used in the same manner as in step 218. Find (t).
[0156]
In the next step 286, it is determined whether or not there is a moving speed Vx (t) that has exceeded the previous click threshold within a predetermined time, and the moving speed Vx (t) has reached the previous click threshold within the predetermined time. If does not exist, the process proceeds to step 288.
[0157]
In step 288, a value (for example, “0”) indicating indistinguishability is substituted for the click type variable indicating the click type, stored in the storage unit 18, and the routine ends. That is, when the feature point moves backward as an initial operation and thereafter does not move forward at a predetermined speed or more within a predetermined time, any of the operations in FIGS. Since it does not apply, it cannot be determined.
[0158]
On the other hand, if there is a moving speed Vx (t) that is equal to or higher than the previous click threshold within the predetermined time, the process proceeds to step 290, and becomes equal to or higher than the previous click threshold and then equal to or higher than the post-click threshold within the predetermined time. It is determined whether the moving speed Vx (t) exists.
[0159]
Then, if there is no moving speed Vx (t) that has become equal to or higher than the post-click threshold within the predetermined time, the process proceeds to step 292.
[0160]
In step 292, a value (for example, “2”) representing the backward click is substituted for the click type variable representing the click type, stored in the storage unit 18, and the routine ends. That is, as shown in FIG. 14, the feature point moves backward as an initial operation, then the feature point moves forward at a speed equal to or higher than the previous click threshold, and thereafter, the feature point moves backward at a speed equal to or higher than the post-click threshold. If not, it is determined that the reverse click operation has been performed (see also FIG. 12B).
[0161]
If there is a moving speed Vx (t) that is equal to or greater than the post-click threshold within a predetermined time after the value is equal to or greater than the previous click threshold, the process proceeds to step 294, and the post-click is performed after the speed is equal to or greater than the previous click threshold. It is determined whether or not there is a moving speed Vx (t) that is equal to or greater than the threshold value and is equal to or greater than the previous click threshold value.
[0162]
If the moving speed Vx (t) does not exceed the previous click threshold after reaching the previous click threshold within the predetermined time, and then exceeds the previous click threshold, the process proceeds to step 296.
[0163]
In step 296, a value (for example, “3”) representing the subsequent stroke is substituted for the click type variable representing the click type, stored in the storage unit 18, and the routine ends. That is, as shown in FIG. 14, the feature point moves backward as an initial operation, then the feature point moves forward at a speed equal to or higher than the previous click threshold, and thereafter, the feature point moves backward at a speed equal to or higher than the post-click threshold. Then, if the feature point has not moved forward at a speed equal to or higher than the previous click threshold value, it is determined that a post-stroke operation has been performed (see also FIG. 12C).
[0164]
If the moving speed Vx (t) becomes equal to or larger than the previous click threshold after being equal to or larger than the previous click threshold and further equal to or larger than the previous click threshold, the process proceeds to step 298, and the click type variable indicating the click type is shifted backward. A value indicating a double click (for example, “4”) is substituted and stored in the storage unit 18, and the routine ends. That is, as shown in FIG. 14, the feature point moves backward as an initial operation, then the feature point moves forward at a speed equal to or higher than the previous click threshold, and thereafter, the feature point moves backward at a speed equal to or higher than the post-click threshold. Then, when the feature point moves forward at a speed equal to or higher than the previous click threshold value, it is determined that the backward double-click operation has been performed (see also FIG. 12D). Note that the processing of steps 218, 222, 224, and 226 corresponds to the click determination unit of the present invention.
[0165]
As described above, by considering only the movement of the feature point in the X direction, it is possible to easily perform the processing of determining various click operations such as a backward click.
[0166]
In FIG. 13, the coordinates of the feature points for a predetermined time are calculated at the same time in steps 280 to 284 and the moving speed is calculated first, and then the click operation is discriminated. However, as shown in FIG. May be.
[0167]
That is, in step 300, the feature point coordinate calculation process is performed in the same manner as in step 280 shown in FIG. 13, and in the next step 302, the moving speed Vx (t) at that time is performed in the same manner as in step 284 shown in FIG. Then, in step 304, it is determined whether or not the moving speed Vx (t) is equal to or higher than the previous click threshold, and if it is equal to or higher than the previous click threshold, the process proceeds to step 310;
[0168]
In step 306, it is determined whether or not a predetermined time has elapsed. If the predetermined time has elapsed, the same processing as step 288 shown in FIG. 13 is performed in step 308. Returns to step 300 and repeats the same processing as described above.
[0169]
In step 310, the feature point coordinate calculation process is performed in the same manner as in step 300, and in the next step 312, the moving speed Vx (t) at that time is performed in the same manner as in step 302. Then, in step 314, it is determined whether or not the moving speed Vx (t) is equal to or greater than the post-click threshold. If it is equal to or greater than the post-click threshold, the process proceeds to step 320. If not, the process proceeds to step 316.
[0170]
In step 316, it is determined whether or not a predetermined time has elapsed, as in step 306. If the predetermined time has elapsed, the same processing as in step 292 shown in FIG. If not, the process returns to step 310 and the same processing as described above is repeated.
[0171]
In step 320, feature point coordinate calculation processing is performed in the same manner as in step 300, and in the next step 322, the moving speed Vx (t) at that time is performed in the same manner as in step 302. Then, in step 324, it is determined whether or not the moving speed Vx (t) is equal to or higher than the previous click threshold. If the speed is equal to or higher than the previous click threshold, the process proceeds to step 326. If not, the process proceeds to step 326.
[0172]
In step 326, it is determined whether or not a predetermined time has elapsed, as in step 306. If the predetermined time has elapsed, the same processing as in step 296 shown in FIG. If not, the process returns to step 320 and the same processing as described above is repeated.
[0173]
In step 330, the same processing as in step 298 shown in FIG. 13 is performed.
[0174]
Note that in the case of FIG. 13, the predetermined time needs to be set to a time sufficient to end a series of click operations requiring a relatively long time such as a later stroke, but in the case of FIG. It is not necessary that the time be sufficient for the click operation to be completed, but it is sufficient if the time is sufficient to perform an operation (for example, a forward click or a reverse click) constituting the click operation. Therefore, the predetermined time in FIG. 15 can be set shorter than the predetermined time in FIG.
[0175]
When the reverse click determination processing ends, the flow shifts to step 227 in FIG. 4 to determine whether or not the click type cannot be determined. This can be determined by referring to the click type variable stored in the storage unit 18.
[0176]
If the click type cannot be determined, the process returns to step 212, and the same processing as described above is repeated. If the click type is not determined, that is, the click type is a forward click, a reverse click, a rear stroke, If any of the reverse clicks, the process proceeds to step 228.
[0177]
In step 228, the type of the click is notified to the PC 12. Thereby, the PC 12 performs a process corresponding to the notified click type, and causes the display 14 to display a process result and the like.
[0178]
Note that the following functions can be considered as functions of each click operation. For example, a forward click is a function equivalent to a single click of the left mouse button as an input / output device, a backward click is a function equivalent to a single click of the right mouse button, and a backward double click is a function equivalent to a double click of the left mouse button. The function to perform and the subsequent stroke can be a function corresponding to the start or end of dragging by the left mouse button.
[0179]
As described above, since various click operations are recognized only by movement in the X direction, various click operations can be performed with natural and simple operations.
[0180]
After notifying the click type to the PC 12, in step 229, a standby process for waiting for a predetermined time is performed, and the process returns to step 200 to shift to the pointing mode.
[0181]
In this way, by performing a wait for a certain period of time after performing the click operation and not performing the movement of the cursor or the determination of the click type, the user moves forward against the intention of the user due to the blurring of the arm. Clicks can be prevented from being input continuously.
[0182]
In addition, when the click operation is performed, the mode is automatically shifted to the pointing mode, so that the user does not need to perform any special operation for shifting to the pointing mode, and the convenience can be improved.
[0183]
【The invention's effect】
As described above, according to the present invention, there is an effect that it is possible to improve the accuracy of determination of a designated position or a designated direction by a user by a natural and simple operation.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a schematic configuration of an information display device according to an embodiment.
FIG. 2 is a side view of an information input space showing a positional relationship between a camera and a concave mirror, and an imaging range of each camera.
FIG. 3A is a front view of an information input space, and FIG. 3B is a side view of the information input space.
FIG. 4 is a flowchart illustrating the contents of an instruction determination process.
FIG. 5 is a flowchart showing details of a feature point coordinate calculation process.
6A is an image diagram showing an example of an image captured by each camera, FIG. 6B is an image diagram showing an example of each image obtained by each camera, and FIG. 6C is a hand height based on a difference in fingertip position on each image. FIG. 4 is a diagram showing an example of the contents of a map for obtaining the value.
FIG. 7 is a flowchart illustrating the contents of a reference point coordinate calculation process according to the first embodiment.
FIG. 8 is an image diagram illustrating an example of a rotation center of a feature point in various instruction operations by a user.
FIG. 9 is an image diagram showing a position of a reference point on which a difference in physique of a user is reflected.
FIG. 10 is an image diagram showing a movement of a reference point when a user changes an arm used for an instruction operation.
11A is a side view, and FIGS. 11B and 11C are plan views for explaining a process of calculating coordinates of a designated position from a feature point and a reference point.
12A is an image diagram for explaining a forward click operation, FIG. 12B is a reverse click operation, FIG. 12C is a reverse double click operation, and FIG.
FIG. 13 is a flowchart illustrating the content of a reverse click process.
FIG. 14 is a diagram for explaining a fingertip position and a moving speed in various click operations.
FIG. 15 is a flowchart illustrating another example of the content of the reverse click process.
[Explanation of symbols]
10 Information display device
12 PC
14 Large screen display
16 control unit
18 Memory
20A, 20B video camera
22 Imaging control device
24 User detection sensor
28 concave mirror
26A, 26B lighting device

Claims (7)

Imaging means for imaging the user from a plurality of different directions,
Three-dimensional feature point whose position changes according to a change in the indicated position or the indicated direction when the user performs an operation of pointing to a specific position or direction based on an image obtained by imaging by the imaging unit. Recognition means for recognizing coordinates,
Judging means for judging a pointing position or a pointing direction by the user based on the three-dimensional coordinates of the feature points recognized by the recognition means;
Stopping state determining means for determining whether or not the feature point is in a stopped state based on the three-dimensional coordinates of the characteristic point recognized by the recognition means;
Based on the determination result by the stop state determination means, at least a pointing mode for performing an operation indicating a specific position or direction, and a click operation including an operation of selecting an object existing at the specified specific position or direction. Switching means for switching between click mode for performing, and
Click determination means for determining whether the click operation has been performed in the click mode,
An instruction action recognition device including: The stop state determining unit determines whether the feature point is in a stop state based on at least a movement amount of the feature point on a predetermined two-dimensional plane that intersects a direction indicated by the user. The pointing operation recognition device according to claim 1, wherein The switching unit switches to a click mode when it is determined that the feature point is in a stopped state, and switches to the pointing mode when the click operation is performed in the click mode. Alternatively, the pointing motion recognition device according to claim 2. The switching unit switches to the click mode when the feature point is determined to be in the stop state in the click mode, and switches to the pointing mode when it is determined that the feature point is not in the stop state in the click mode. The pointing operation recognition device according to claim 1, wherein The click determination means determines the type of the click operation based on at least a movement amount of the feature point in a direction substantially orthogonal to a predetermined two-dimensional plane that intersects a direction specified by the user. The pointing operation recognition device according to any one of claims 1 to 4. Display means for displaying any information, and displaying, with a cursor, a designated position or designated direction by the user determined by the determining means;
Control means for controlling the display means so that the cursor is different in the case of the pointing mode and the case of the click mode,
The pointing action recognition device according to any one of claims 1 to 5, further comprising: 7. The pointing motion recognition apparatus according to claim 6, wherein the control unit controls the display unit such that a movement rate of the cursor in the click mode is smaller than that in the pointing mode.

Images