JP2005011037A - Pointing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pointing device for easily controlling a pointer by using a biological signal, and for follow up even the quick movement of the pointer. <P>SOLUTION: This pointing device is provided with a biological signal collecting part(EMG signals) for collecting a biological signal, a biological signal processing part(EMG signal processing) for extracting a characteristic pattern from the biological signal, a neural network part(Neural network) for calculating the probability of a pointer moving direction intended by a user from the characteristic pattern, a moving direction deciding part for deciding the moving direction of the pointer from the probability calculated by the neural network part, a pointer control part(Pointer control) having a pointer moving amount calculating part for calculating the moving amount of the pointer from the moving direction of the pointer and the average strength of the biological signal based on a physical model and a parameter storing part for storing a parameter to be used by each part with making it readable. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

ここで、ＥＭＧ_ｌ ^ｓｔｂａｒは安静時におけるＥＭＧ_ｌ（ｔ）の時間平均である。
ＥＭＧ_Ｌ＋１（ｔ）に関しては計測部位をＬチャンネルとは別の部位（例えば首部）とし、この値を最大筋収縮時の値で正規化したものをクリック信号β（ｔ）とする。この値が設定したしきい値を越えた時にクリックイベントが発生したと判定する。
【００１４】
ＥＭＧ信号処理部では、またピーク値の検出が行われ、ピーク値がしきい値が越えたか否かが判定される。しきい値を越えた場合には、ポインタ移動の動作が行われたものと判定され、ニューラルネット部への特徴パターンの出力が行われ、ニューラルネット部での処理が行われる。また、このピーク値がしきい値を越えた場合にポインタ制御部においてポインタの移動が行われるようにしてもよい。ピーク値がしきい値が越えたか否かの判定は、ＥＭＧ_ｌ（ｔ）（ｌ＝１，２，…，Ｌ）の和または二乗和がしきい値が越えたか否かで判定する。あるいは、ＥＭＧ_ｌ（ｔ）（ｌ＝１，２，…，Ｌ）の中の何れかがあるしきい値が越えた場合にピーク値がしきい値が越えたと判定してもよい。
ＥＭＧ信号処理部では、また周波数解析処理を行うようにしてもよい。例えば、ＥＭＧ信号に対してＦＦＴ（Ｆａｓｔ Ｆｏｕｒｉｅｒ Ｔｒａｎｓｆｏｒｍ）処理を加え、周波数軸についての信号変化を解析するようにしてもよい。
【００１５】
〔ニューラルネット部〕
ニューラルネット部では、ＥＭＧ信号処理部で計算した使用者のＥＭＧパターン（ベクトル）ｘ（ｎ）（ｎ＝ｔ−Ｔ＋１，…，ｔ）からポインタの各基準方向への事後確率を推定する。ここでＴは入力パターンの時間方向への長さとする。この計算には本発明者らが提案したＲ−ＬＬＧＭＮを用いる。このネットワークは、統計モデルの一つである混合正規分布と、動的な確率モデルの一つである隠れマルコフモデルを構造として内包しており、使用者のＥＭＧ信号の時系列的な特性を学習により獲得することができる。まずポインタの制御を行う前に、Ｒ−ＬＬＧＭＮを利用してＥＭＧ信号の時系列パターンとポインタの移動方向の関係を学習する。学習後のＲ−ＬＬＧＭＮは各基準方向に対してポインタが移動する事後確率をモデルに基づいて推定することができ、基準方向に対応する出力層の各ユニットからそれぞれの方向に対する事後確率^（５）Ｏ^ｃ（ｔ）（ｃ＝１，…，Ｃ）を出力する。
前向き計算では、まず、入力ベクトルｘ（ｔ）を次の（２）式に従って非線形変換し、新たな入力ベクトルとして用いる。
【００１６】
【数２】

第１層はＨ＝１＋Ｌ（Ｌ＋３）／２個のユニットから構成され、恒等関数を入出力関数として用いている。この新たな入力ベクトルＸ_ｈ（ｔ）（ｈ＝１，２，…，Ｈ）はそのまま^（１）Ｏ_ｈ（ｔ）として出力される。なお、図２においてＸ_ｈ（ｔ）（ｈ＝１，２，…，Ｈ）はＸ_ｈ（ｈ＝１，２，…，Ｈ）と表記されている。
第２層のユニット｛ｃ，ｋ，ｋ’，ｍ｝（ｃ＝１，…，Ｃ；ｋ，ｋ’＝１，…，Ｋ_ｃ；ｍ＝１，…，Ｍ_ｃ，ｋ）は第１層のユニットの出力に重み係数ｗ^ｃ _{ｋ ’ ，ｋ，ｍ，ｈ} が掛け合わされたものを受け取る。第２層の入出力関係は、次の（３）、（４）式により与えられる。
【００１７】
【数３】

ただし、Ｃは基準方向数、Ｋ_ｃは隠れマルコフモデルの状態数に対応するパラメータ、Ｍ_ｃ，ｋは基準方向ｃ、状態ｋに対応する混合ガウス分布モデルのコンポーネント数を表す。
【００１８】
第３層のユニット｛ｃ，ｋ，ｋ’｝への入力は、第２層のユニット｛ｃ，ｋ，ｋ’，ｍ｝（ｍ＝１
，…Ｍ_ｃ，ｋ）の出力を統合したものである。また、その入力に１時刻前の第４層の出力を乗じた値が第３層の出力となる。第３層の入出力関係は、次の（５）、（６）式により与えられる。
【００１９】
【数４】

ただし、初期状態は^（４）Ｏ^ｃ _{ｋ ’}（０）＝１．０とする。
さらに第４層は第３層のユニット｛ｃ，ｋ，ｋ’｝の出力を統合したものを受け取る。
入出力関係は、次の（７）、（８）式により与えられる。
【００２０】
【数５】

【００２１】
最後に、第５層のユニットｃは第４層のＫ_ｃユニット｛ｃ，ｋ｝（ｋ＝１，…，Ｋ_ｃ）の出力を統合したものである。入出力関係は、次の（９）、（１０）式により与えられる。
【００２２】
【数６】

この出力^（５）Ｏ^ｃ（ｔ）が基準方向ｃに対する事後確率となる。
以上のように、このネットワークは、第３層と第４層の間にリカレント結合を有することで、 １時刻前の出力を考慮することができる。また、第１層と第２層の間の重み係数ｗを学習的に調節するだけで、基準方向に対する事後確率を混合正規分布モデルに基づいてモデル化することができる。
【００２３】
学習時には、まずシステムから使用者に対してポインタの目標移動方向φ（０≦φ＜２π）をランダムに提示する。ただし、 φは図１のｘ軸方向を０とし、反時計回りに増加するように定義する。使用者は指定された方向へのポインタ移動を意図して各筋を収縮する。そしてこのときのＥＭＧ信号と目標移動方向φをＮ組採取し、学習用サンプルデータセットとして使用する。
いま、時刻Ｔにおいて、ｎ 番目の入力ベクトルｘ（ｔ）^（ｎ）に対応する教師ベクトルＴ^（ｎ）（φ）＝（Ｔ_１ ^（ｎ）（φ），…，Ｔ_ｃ ^（ｎ）（φ） ，…，Ｔ_ｃ ^（ｎ）（φ））^Ｔ（ｎ＝１，…，Ｎ）を（１１）式により計算する。
【００２４】
【数７】

ただしφ_ｃは、下記の（１２）式のように定義する。
φ_ｃ＝２πｃ／Ｃ （１２）
この教師信号は、目標移動方向φを、その方向を挟む２つの基準方向に分解したときの成分に対応する。もちろん確率であるから、その和は常に１である。例として、基準方向をφ＝０，π／２，π，３／２πｒａｄの４方向とした場合の教師信号を図３に示す。
図から分かるように、目標移動方向が基準方向と一致しない場合にはφに近い二つの基準方向が選択され、その方向に対応する出力ユニットに教師信号が与えられている。この方法により、基準方向以外の方向も学習用データとして用いることが可能となる。
【００２５】
Ｒ−ＬＬＧＭＮは、 Ｃ個の基準方向それぞれに用意されたＬ個の時系列信号（Ｎ＝ＬｘＣ）を学習データとして学習を行う。
学習用データに対するネットワークの評価関数Ｊは、下記の（１３）式の通り定義し、
【００２６】
【数８】

これを最小化、すなわち対数尤度を最大化するように学習を行う。ただし、Ｙ^ｃ（ｔ） は入力ベクトルｘ（ｔ）^（ｎ）に対する時刻Ｔでの出力を意味している。また、Ｒ−ＬＬＧＭＮに含まれるリカレント結合の影響を考慮するため、Ｐ．Ｊ． Ｗｅｒｂｏｓ：「Ｂａｃｋｐｒｏｐａｇａｔｉｏｎ ｔｈｒｏｕｇｈ ｔｉｍｅ：ｗｈａｔ ｉｔ ｄｏｅｓ ａｎｄ ｈｏｗ ｔｏ ｄｏ ｉｔ」， Ｐｒｏｃｅｅｄｉｎｇｓ ｏｆ ｔｈｅ ＩＥＥＥ， Ｖｏｌ．７８，Ｎｏ．１０，１５５０／１５６０，（１９９０）に記載された通時的誤差逆伝播方式（ＢＰＴＴ）を用いる。これは時系列中の誤差勾配を蓄積して、重み修正量を計算するという方式である。
ここで、学習の収束時間を待つことになる操作者の精神的負担を少しでも軽減することを目的として、ＺａｋがＭ． Ｚａｋ： 「Ｔｅｒｍｉｎａｌ Ａｔｔｒａｃｔｏｒｓ ｆｏｒ Ａｄｄｒｅｓｓａｂｌｅ Ｍｅｍｏｒｙ ｉｎ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋｓ」， Ｐｈｙｓｉｃｓ Ｌｅｔｔｅｒｓ Ａ， Ｖｏｌ．１３３，１８／２２（１９８８）において提案したターミナルアトラクタ（Ｔｅｒｍｉｎａｌ Ａｔｔｒａｃｔｏｒ）を学習側に導入し、収束時間の上限値を設定可能とした。
【００２７】
〔ポインタ制御部〕（移動方向決定部）
ここでは、Ｒ−ＬＬＧＭＮからの出力および推定した筋活動レベルα（ｔ）を基にポインタの動きを計算する。
Ｒ−ＬＬＧＭＮからの出力は、図１に示すように各基準方向ｃへポインタが移動する可能性を示している。そこで時刻ｔのＥＭＧパターンによるポインタの移動方向ベクトルｅ（ｔ）＝（ｅ_Ｘ（ｔ），ｅ_Ｙ（ｔ））を、以下の（１４）〜（１７）式のように定義する。
【００２８】
【数９】

【００２９】
【数１０】

ここで、ｖ _Ｘ（ｔ）， ｖ _Ｙ（ｔ）は各基準方向へ対応するベクトル成分のｘ成分、ｙ成分を示す。
【００３０】
〔ポインタ制御部〕（ポインタ移動量算出部）
また筋活動レベルα（ｔ）を以下の（１８）式の通り計算する。
【００３１】
【数１１】

ここでＥＭＧ_ｃ ^ｍａｘｂａｒは各基準方向に対する最大随意筋収縮時のＥＭＧ信号ＥＭＧ_ｃｌ ^ｍａｘ （ｔ）のチャンネル和を時間平均したものである。このような筋活動レベルを利用することで各方向に対する最大随意筋収縮レベルの違いを調節することができ、これをポインタの移動速度の決定に利用する。
【００３２】
使用者の力特性に合ったポインタの操作感覚を実現するためには、我々が日常経験する物理則にしたがったポインタの運動が適すると考えられる。そこで、筋活動レベルα（ｔ）と移動方向ベクトルｅ（ｔ）に基づき、以下の（１９）式のインピーダンスモデルを用いてポインタがどのように移動するかを計算する。
【００３３】
【数１２】

Ｍ_ｅ，Ｂ_ｅはそれぞれ慣性、粘性パラメータでｐ（ｔ）はポインタの位置ベクトルである。インピーダンスモデルに弾性力を加味してもよい。その場合には、（１９）式左辺にＥ_ｅ ｐ（ｔ）を加える。ここでＥ_ｅは弾性パラメータである。またＦ（ｔ）はポインタに加える駆動力で、α（ｔ）、ｅ（ｔ） から０次ホールドを用いて（２０）式の通り計算する。
【００３４】
【数１３】

ただし、 ｇは力ゲインを表す。α_０ はポインタ操作の有無を判定するしきい値である。
（１９）式は粘性摩擦Ｂ_ｅを有する平面に置かれた質量Ｍ_ｅの物体に力Ｆ（ｔ）が加わった際の運動方程式を表している。このような物理モデルに従い、数値積分によってポインタの位置、速度を計算することで、操作者の力感覚に対応した自然な操作感が期待できる。
【００３５】
〔実験〕
構築した実施の形態装置の操作性を調べるための実験を行った。被験者は６名で、被験者（Ｓｕｂｊｅｃｔ）Ａ，Ｂ，Ｃ，Ｄ，Ｅについては、いずれも健常な男子学生（２２才、２３ 才、２２才、２２才、２４ 才）とした。被験者Ｆは男性（ ４５才）で４年前に事故のため右手首から１５ｃｍ程の部位で前腕部を切断している。 どの被験者も事前に十分な練習を行った。
ＥＭＧ信号計測用の電極（ＮＴ−５１１Ｇ、ＮＴ−５１２Ｇ：（株）日本光電製）は材質がＡｇ／ＡｇＣｌ，形状が直径０．０１２ｍの円形、双極差動型のものを用いた。使用数は７ｃｈで右前腕部に６ｃｈ（橈側手根伸筋、橈側手根屈筋、尺側手根伸筋、尺側手根屈筋、 深指屈筋、固有示指伸筋）、左前腕部に１ｃｈ（橈側手根伸筋）を装着した。左前腕の電極は、クリックイベント発生用である。ただし、これらの電極の装着位置に関しては、自由に装着してもシステムに導入したニューラルネットによりある程度、適応が可能である。臨床で使用する際には、ＥＭＧ信号に関して知識不十分 な使用者も想定され、特定の筋の上に電極を装着することが困難と考えられるので、このような適応機能が有効と考えられる。
実験では、ポインタの移動方向を右手首の曲げ方向に対応させてニューラルネットの学習を行った。ただし、切断者の場合は脳内に残された手首の運動イメージによりマウスを操作するように指示した。
データのサンプリング周波数は１００［Ｈｚ］で、Ｒ−ＬＬＧＭＮのパラメータであるコンポーネント数、状態数は、それぞれ、Ｍ＝１，Ｋ＝１，学習に用いた基準方向はφ＝０からπ／２ｒａｄきざみの４方向と設定した。なお本実験では、ディスプレイのサイズは１７インチ、解像度は６４０ｘ４８０とした。
【００３６】
〔操作例〕
図４、図５に操作の一例として被験者Ａが円を描いた際の結果を示す。Ｒ−ＬＬＧＭＮ学習時の時系列長は２０とし、推定時の時系列長は５とした。図４はディスプレイ上のポインタの軌跡を示しており、ポインタの色が濃い区間ではドラッグ操作を行っている。図５は制御中の信号処理の様子を示しており、上から順にＥＭＧ信号、筋活動レベルα（ｔ）、クリック信号β（ｔ）、ポインタの移動速度、移動方向を示している。この図から、ＥＭＧ信号を利用してポインタを随意的に制御可能なことが分かる。
【００３７】
次に、図６、図７に被験者Ｆ（上肢切断者）による操作例を示す。ここでは、ディスプレイ上にランダムに幾つかの円を配置し、その円を順番にクリックするタスクを実行している。使用電極数は８ｃｈで切断された前腕部の残存筋に７ｃｈ、また、左前腕部（橈側手根伸筋）にクリックイベント用の１ｃｈを装着した。Ｒ−ＬＬＧＭＮ学習時の時系列長は２０とし、推定時の時系列長は５とした。図６はディスプレイ上のポインタの軌跡、図７は制御中の信号処理の様子を示しており、上から順にＥＭＧ信号 、筋活動レベルα（ｔ）、クリック信号β（ｔ）、ポインタの移動速度、移動方向を示している。本システムが、上肢切断者に対してもスムーズなポインティングを提供可能なことが分かる。
【００３８】
〔方向精度の検証〕
開発したシステムの全方位への制御特性を調べるために、方向精度の検証を行った。各被験者には、ディスプレイによるフィードバックの無い状態で、手首をπ／８ｒａｄきざみの１６方向に各方向２秒間動作するよう指示した。そして、このとき計測したＥＭＧ信号から推定したポインタ移動方向と操作者への指示方向との誤差を評価指標として計算した。
図８ に実験結果を示す。図８（ａ）は被験者Ａ、図８（ｂ）は被験者Ｂの結果であり、
学習時、推定時に入力する特徴パターンの時系列長を、それぞれ１きざみで１から４０までの計１６００通りに変化させて計算した誤差平均を示している。ただし、誤差平均は全方位に対する誤差の平均値である。図より各被験者とも学習時の時系列長が長く、推定時の時系列長が短くなるに従い誤差が減少していることが分かる。
【００３９】
ここで方向精度には、学習方向、未学習方向で違いが見られたため、図８の実験結果 を学習した４方向と未学習の１２方向に分けて再計算した。図９、図１０はそれぞれ学習方向、未学習方向の誤差平均であり、それぞれ（ａ）は被験者Ａ、 （ｂ）は被験者Ｂ、の結果である。この図より、学習時の時系列長が長く、推定時の時系列長が短くなるにつれて、学習した方向では誤差が増加し、未学習の方向では減少することが分かる。これは、推定時の時系列長が学習時のそれよりも短くなることで、情報量が低下し、Ｒ−ＬＬＧＭＮからの事後確率の出力が曖昧になったためである。この結果より、全方位への精度を平均的に向上させるためには、学習した方向の誤差が増大してしまうものの、推定信号は短いほうが良いことが分かる。以下では、学習時の時系列長を１０、推定時の時系列長を１として操作を行うこととする。
【００４０】
次に、本システムにおけるＲ−ＬＬＧＭＮ の有効性を示すために、ＬＬＧＭＮ、Ｂａｃｋ Ｐｒｏｐａｇａｔｉｏｎ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋ（以下、ＢＰＮＮと記す）を利用した場合のシステムとの比較実験を行った。ＬＬＧＭＮ は、混合正規分布モデルを構造として導入しており、Ｒ−ＬＬＧＭＮからリカレント結合を除いた場合に相当する。被験者にはディスプレイによるフィードバックのある状態で、指定された方向にポインタを移動するように指示し、その際の方向推定精度を求めた。
この実験において、 ＬＬＧＭＮ のコンポーネント数、ターミナルアトラクタのパラメータである学習時間、学習率、サンプリングタイムはＲ−ＬＬＧＭＮのそれらと同数に設定した。ＢＰＮＮのユニット数に関しては、方向推定精度が高くなるように、入力層６、第２層７、第３層６、出力層４と試行錯誤的に決定した。なお、ポインタの移動方向推定処理では、ＢＰＮＮ 出力の総和が１となるように正規化し、 Ｒ−ＬＬＧＭＮの場合と同様のアルゴリズムで方向推定を実施した。学習の評価関数には、教師信号との二乗誤差を利用し、学習率、慣性係数、学習終了誤差しきい値のパラメータをそれぞれ０．０１，０．００１，０．０００３とした。ただし、評価関数の値が５万回の繰り返し計算の後に誤差しきい値以下にならない場合は収束不能とし学習を打ち切った。
図１１に被験者Ｂによる実験結果を示す。各レーダチャートは各指定方向を示してから２秒間における誤差の平均と標準偏差を表しており、
（ａ）はＬＬＧＭＮ、（ｂ）はＢＰＮＮ、（ｃ）は Ｒ−ＬＬＧＭＮの結果である。この図から、Ｒ−ＬＬＧＭＮ は、ＢＰＮＮやＬＬＧＭＮに比べて方向精度にばらつきが無く、指示された方向に正しく推定が行えていることが分かる。
【００４１】
〔操作性の検証〕
最後に本発明デバイスの操作性を評価するための実験を行った。実験時に被験者に提示した画面を、図１２に示す。図におけるポインタの大きさ、ターゲットの大きさ、ポインタとターゲットの距離、それぞれの比率はディスプレイに表示されるものと同等にしてある。この図のように、被験者はディスプレイ上に表示された２点間を往復するタスクを実行した。２ 点間の距離は４００ｐｉｘｅｌｓ、ポインタの移動方向はπ／６ｒａｄきざみの６方向とし、往復操作における操作時間、最短軌道からのずれの積分値を計算する。 ６方向すべてに対する操作を１セッションとし、５セッションのデータの平均値、標準偏差を求めた。Ｒ−ＬＬＧＭＮ， ＬＬＧＭＮ，ＢＰＮＮの設定は〔方向精度の検証〕の場合と同様とした。
に実験結果を示す。
図１３、図１４に実験結果を示す。図１３は往復走査の平均時間、図１４は最短起動からのずれの積分値を示している。また、図１４ のＲ−ＬＬＧＭＮの結果を基準にＢＰＮＮ， ＬＬＧＭＮの結果を検定した（片側検定）。ここで、危険率、５．１ ％で有意差ありと判定された結果にはそれぞれ、 印を表中に示す。各手法を比較すると、Ｒ−ＬＬＧＭＮ が平均時間、軌道誤差ともに小さくなる傾向となっており、操作性が最も高いことが確かめられた。
【００４２】
以上好ましい実施の形態について説明したが、本発明はこれに限定されるものではなく、本発明の要旨を逸脱することのない範囲内において適宜の変更が可能なものである。例えば、実施の形態では、生体信号にはＥＭＧ信号を用いていたが、これに代え若しくはこの信号と併用して脳波、手足の動き（加速度）、接触圧などの信号を用いることができる。
【００４３】
【発明の効果】
以上説明したように、本発明のポインティングデバイスは、Ｒｅｃｕｒｒｅｎｔ構造を有するニューラルネットを用いてポインタの移動方向を推定するものであるので、変化の激しい生体信号に対しても精度よく対応することのできる、また、基準方向以外の学習していない方向に対してもスムースな移動が可能な全方位型ポインティングデバイスを提供することができる。
【図面の簡単な説明】
【図１】本発明のシステムの概略を示すブロック図。
【図２】本発明のポインティングデバイスにおいて用いられるニューラルネット（Ｒ−ＬＬＧＭＮ）の動作説明図。
【図３】本発明のポインティングデバイスにおける学習過程を説明するための、基準方向と教師信号との関係を示す図。
【図４】本発明のポインティングデバイスの使用結果を示す図（その１）。
【図５】図４の結果を得た際の各部の信号波形図。
【図６】本発明のポインティングデバイスの使用結果を示す図（その２）。
【図７】図６の結果を得た際の各部の信号波形図。
【図８】学習時と推定時の時系列長をパラメータとする方向誤差を示すグラフ（その１）。
【図９】学習時と推定時の時系列長をパラメータとする方向誤差を示すグラフ（その２）。
【図１０】学習時と推定時の時系列長をパラメータとする方向誤差を示すグラフ（その３）。
【図１１】ニューラルネット部にＲ−ＬＬＧＭＮを用いた場合と、他の方式を用いた場合との推定方向の精度を示す図。
【図１２】本発明のポインティングデバイスの操作性を検証するために用いたディスプレイ上の目標点を示す図。
【図１３】ニューラルネット部にＲ−ＬＬＧＭＮを用いた場合と、他の方式を用いた場合との往復操作の平均時間を示す図表。
【図１４】ニューラルネット部にＲ−ＬＬＧＭＮを用いた場合と、他の方式を用いた場合との最短軌道からのずれの積分値を示す図表。[0001]
BACKGROUND OF THE INVENTION
The present invention is an all-controllable device that can be controlled using biological signals such as myoelectric potential (EMG), electroencephalogram (EEG), limb movement (acceleration), exhalation, and contact pressure that can be used by severely disabled persons. The present invention relates to an orientation type pointing device.
[0002]
[Prior art]
As for a pointing device using a biological signal, a method of directly designating a cursor coordinate from intensity information of a smooth muscle potential signal has already been proposed (for example, see Patent Document 1). Regarding the identification of biological signals, intent estimation from a myoelectric potential feature pattern using a neural network has been proposed (for example, see Patent Document 2). As for an interface device using a biological signal, there has been proposed a device for a person with severe disability to communicate his / her intention using myoelectric potential, electroencephalogram, ocular potential, and the like (see, for example, Patent Document 3). ).
[0003]
However, the use of the biological signal by the conventional method is either a control depending on the intensity of the signal as described in Patent Document 1, or a finite number of feature patterns as described in Patent Documents 2 to 3. It was limited to the method of identifying the class. For this reason, it has been extremely difficult to realize control in which the cursor is freely moved in all directions in the pointing device. In addition, the conventional system has a problem in operability due to insufficient configuration such as a feedback function to the user and a physical model that determines the movement of the cursor.
[0004]
By the way, the biological signal such as the EMG signal generated according to the human muscle contraction level has useful information such as what operation is being performed with what force and how much the viscoelasticity of the muscle is. If this information is included and can be extracted efficiently, there is a possibility that a new interface device that has never existed can be realized.
Researches for estimating human willingness from EMG signals have been attempted mainly with regard to the control method of an electric prosthetic hand. The present inventors have so far conducted an electric prosthetic hand using an EMG pattern identification by a neural network. The manipulator has been controlled (for example, see Non-Patent Documents 1 to 5). However, in these studies, since the multi-degree-of-freedom control problem of the electric prosthetic hand is considered as a motion identification problem of the EMG pattern, the degrees of freedom of the driving joints are limited to the number of pattern identification classes. When these methods are directly used for pointing device operation, for example, when the direction of movement of the pointer is reduced to a pattern identification problem, the total number of directions in which the pointer can be moved must be set in advance. Learning sample number and learning time are required.
Thus, the present inventors have solved this problem so far by expressing the infinitely movable direction of the pointer by a combination of a finite number of reference directions (see, for example, Non-Patent Document 6). . This method uses the Log-Linearized Gaussian Mixture Network (logarithm linearized mixed normal distribution network: hereinafter referred to as LLGMN) proposed by the present inventors to estimate the moving direction and calculates the posterior probability for each reference direction. The pointers can be moved in all 360 ° directions by combining them as vectors. At this time, the LLGMN only needs to estimate the posterior probability with respect to the set reference direction, and the learning burden can be reduced and the network structure can be simplified. However, since this method can model only the pattern obtained at each time, that is, only the static characteristics of the signal, when the signal variation is severe, the identification accuracy is lowered. In addition, it may be difficult to move the cursor in a direction other than the reference direction that has not been learned.
[0005]
On the other hand, the present inventors proposed a recurrent Log-Linearized Gaussian Mixture Network (recursive LLGMN: hereinafter referred to as R-LLGMN) in which an LLGMN has a recurrent structure (see, for example, Non-Patent Document 7). . This network has a structure that includes a mixed normal distribution model, which is one of statistical models, and a hidden Markov model (HMM), which is one of dynamic probability models. The time-series characteristics of the EMG signal can be acquired by learning.
[0006]
Thus, many studies on human interfaces using EMG signals and EEG signals have been attempted so far. For example, Rosenberg has been researching a two-dimensional pointing device using an EMG signal as an interface of a wearable computer (for example, see Non-Patent Document 8). However, this method uses BPNN (Back Propagation Neural Network), and if the operability is evaluated based on Fitts's law (for example, see Non-Patent Document 9), only 14% of the performance of a normal mouse can be realized. Not. Meanwhile, Barreto et al. Developed a computer interface using EEG signals and EMG signals for supporting the disabled (see Non-Patent Document 10, for example). In this method, pointing and clicking operations in four directions are realized using the frequency information of the EEG signal and the EMG signal measured from the surface stripe. Further, Roberts et al., Mason et al. Reported a pointing study using only an EMG signal (for example, see Non-Patent Documents 11 and 12). Roberts et al. Identified two types of commands from the 1ch EEG signal. On the other hand, Mason et al. Performed a switching operation using the characteristics of the EEG signal. However, these systems have been difficult to apply clinically because the systems could not be adapted to signal changes due to individual differences, fatigue, sweating, and the like.
[0007]
[Patent Document 1]
JP-A-8-36459
[Patent Document 2]
JP 2002-345862 A
[Patent Document 3]
Japanese Patent Laid-Open No. 11-203022
[Non-Patent Document 1]
Toshio Tsuji, Daiichiro Mori, Koji Ito: “EMG motion identification method using neural network incorporating statistical structure”, IEEJ Transactions C, Vol. 112-C, no. 8,465 / 473, (1992)
[Non-Patent Document 2]
Toshio Tsuji, Hiroyuki Ichinobu, Makoto Kaneko: “Feed forward neural network using mixed normal distribution model”, IEICE Transactions, D-II, Vol. 77, no. 10, 2093/2100, (1994)
[Non-Patent Document 3]
Osamu Fukuda, Toshio Tsuji, Makoto Kaneko: "Manually-controlled human support manipulator using EMG signal", Journal of the Robotics Society of Japan, Vol. 18, no. 3,387 / 394, (2000)
[Non-Patent Document 4]
O. Fukuda, T .; Tsuji and M.M. Kaneko: “An EMG Controlled Robotic Manipulator Using Neural Networks”, Proceedings of IEEE International Workshop on Robot and Human Commune, 44/44.
[Non-Patent Document 5]
O. Fukuda, T .; Tsuji, A .; Otsuka and M.M. Kaneko: “An EMG-based Human Robot Interface for Rehabilitation Aid”, Proceedings of IEEE International Conference on Robotics and Automation, 3492/3497 (1998).
[Non-Patent Document 6]
Toshio Tsuji, Osamu Fukuda, Mitsuru Murakami, Makoto Kaneko: “EMG controlled pointing device using neural network”, Transactions of the Society of Instrument and Control Engineers, Vol. 1, No. 1 1, 1/7, (2000)
[Non-Patent Document 7]
T.A. Tsuji, N .; Bu, Fukuda, and M.M. Kaneko: “A Recurrent Log-Linearized Gaussian Mixture Network”, IEEE Trans. On Neural Networks, (in press)
[Non-Patent Document 8]
R. Rosenberg: “The Biofeedback Pointer: EMG Control of a Two Dimensional Pointer”, Proceedings of the 2nd. International Symposium on Wearable Computers, 162/163, (1998)
[Non-patent document 9]
P. M.M. Fitts: "The information capacity of the system in control the amplified of movement", Journal of Experimental Psychol. 47, 381/391, (1954)
[Non-Patent Document 10]
A. B. Barreto, S.H. D. Scalgle and M.M. Adjouradi: “A Practical EMG-based Human-computer Interface for Users with Motor Disabilities”, Journal of Rehabilitation Research and Development (37), 37 (Development).
[Non-Patent Document 11]
S. J. et al. Roberts, W.M. D. Penny: “Real-time Brain-Computer Interfacing: a Preliminary Study Using Bayesian Learning, Medical, Biological Engineering and Computing, Vol. 38, 56/61 (2000).
[Non-Patent Document 12]
S. J. et al. Mason, G. et al. E. Birch: “A Brain-Computer Switch for Asynchronous Control Applications”, IEEE Transactions on Biological Engineering, Vol. 47, no. 10, 1297/1307, (2000)
[0008]
[Problem to be Solved by the Invention]
The present invention has been made in view of the above problems, and its purpose is to use myoelectric potential, electroencephalogram, limb movement (acceleration), exhalation, contact pressure, etc., even if it is a signal that changes drastically, it is accurate. It is intended to provide a pointing device that can be controlled well and can be easily moved even in an unlearned direction.
[0009]
[Means for Solving the Problems]
In order to achieve the above object, according to the present invention, a biological signal collecting unit that collects biological signals, a function of extracting a feature pattern from the biological signals, and a function of calculating an average intensity of the biological signals are provided. A biological signal processing unit, a neural network unit that calculates the probability of the pointer movement direction intended by the user from the feature pattern, and a movement direction determination unit that determines the movement direction of the pointer from the probability calculated by the neural network unit, And a pointer control unit having a pointer movement amount calculation unit for calculating a pointer movement amount based on a physical model from the movement direction of the pointer and the average intensity of the biological signal, and storing / reading parameters used in each unit A pointing device including a possible parameter storage unit, wherein the neural network unit has a recurrent structure. A pointing device, which is characterized in that utilizes a log-linearized Gaussian Mixture Network R-LLGMN with is provided.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an embodiment of the present invention. The pointing device according to the present invention is advantageously used in a physically handicapped person such as an upper limb amputee, but can also be used in a healthy person and is wearable except for the display or including the display. Can be configured.
The pointing device of the present invention includes an EMG signal processing unit that extracts a feature amount of an EMG signal by performing predetermined processing on a plurality of EMG signals emitted by an operator, and calculates a probability of a moving direction from the input feature amount. It comprises three blocks, a neural network unit for calculating and a pointer control unit for calculating the amount of movement of the pointer, and further includes a display that constitutes a feedback unit. The display can graphically present the measured biological signal, the calculated feature pattern, the output of the neural network unit, the movement direction estimation result, the amount of movement of the pointer, and the like to the user. The above three blocks are configured by an information processing device including a central processing unit (CPU) and a memory for storing various data and parameters. This information processing device is used for exchanging signals with the outside. An interface (I / F) is provided. Parameters stored in the memory include the type of signal selected in the input unit, the cutoff frequency of the filter used for feature extraction in the EMG signal processing unit, the threshold value related to the peak value of the signal, and the neural network unit Include parameters of a neural network such as a weighting factor and parameters such as inertia, viscosity, and elasticity in a pointer control unit, and at least parameters related to the neural network unit are stored for each user.
Hereinafter, the operation of each unit will be described in detail.
[0011]
[EMG signal processor]
In the EMG signal processing unit, a cooperative pattern of muscle strength is extracted as a feature amount from an EMG signal obtained from a plurality of muscles. For example, EMG signals for (L + 1) channels measured with a multi-telemeter (Web5000: manufactured by Nihon Kohden Co., Ltd.) are A / D converted and full-wave rectified for each channel, and then a secondary digital Butterworth filter (cut-off) (Frequency: 1 Hz). Furthermore, this signal is resampled at a sampling frequency of 100 Hz, and this time series signal is converted to EMG._l(T) (l = 1, 2,..., L + 1). This EMG_lOf (t), the L channel (l = 1, 2,..., L) is used for the direction and speed, and the 1 channel (l = L + 1) is used for the click event processing.
[0012]
EMG_lFor (t) (l = 1, 2,..., L), a feature pattern x (t) = [x is obtained by normalizing according to the equation (1) so that the sum of the L channels is 1.₁(T), x₂(T), ..., x_L(T)] is input to the neural network unit. Here, x (t) is a vector. In FIG. 2, x₁(T), x₂(T), ..., x_L(T) is x₁, X₂, ..., x_dIt is written.
[0013]
[Expression 1]

Where EMG_l ^stbar is EMG at rest_lIt is a time average of (t).
EMG_{L + 1}With regard to (t), the measurement site is a site (for example, the neck) different from the L channel, and this value is normalized with the value at the time of maximum muscle contraction and used as the click signal β (t). It is determined that a click event has occurred when this value exceeds a set threshold value.
[0014]
The EMG signal processing unit also detects a peak value and determines whether or not the peak value exceeds a threshold value. When the threshold value is exceeded, it is determined that the pointer movement operation has been performed, the feature pattern is output to the neural network unit, and the processing in the neural network unit is performed. Further, when the peak value exceeds the threshold value, the pointer control unit may move the pointer. Whether the peak value exceeds the threshold is determined by EMG_l(T) Judgment is made based on whether the sum or square sum of (l = 1, 2,..., L) exceeds a threshold value. Or EMG_l(T) It may be determined that the peak value has exceeded the threshold when any one of (l = 1, 2,..., L) exceeds a threshold.
The EMG signal processing unit may perform frequency analysis processing again. For example, FFT (Fast Fourier Transform) processing may be applied to the EMG signal to analyze the signal change on the frequency axis.
[0015]
[Neural network part]
The neural network unit estimates the posterior probabilities of the pointers in the reference directions from the user's EMG pattern (vector) x (n) (n = t−T + 1,..., T) calculated by the EMG signal processing unit. Here, T is the length of the input pattern in the time direction. For this calculation, R-LLGMN proposed by the present inventors is used. This network contains a mixed normal distribution, which is one of the statistical models, and a hidden Markov model, which is one of the dynamic probabilistic models, as a structure, and learns the time-series characteristics of the user's EMG signal. It can be acquired by. First, before controlling the pointer, the R-LLGMN is used to learn the relationship between the time series pattern of the EMG signal and the movement direction of the pointer. The R-LLGMN after learning can estimate the posterior probability that the pointer moves with respect to each reference direction based on the model, and the posterior probability for each direction from each unit of the output layer corresponding to the reference direction.⁽⁵⁾O^c(T) (c = 1,..., C) is output.
In the forward calculation, first, the input vector x (t) is nonlinearly transformed according to the following equation (2) and used as a new input vector.
[0016]
[Expression 2]

The first layer is composed of H = 1 + L (L + 3) / 2 units and uses an identity function as an input / output function. This new input vector X_h(T) (h = 1, 2,..., H) remains unchanged⁽¹⁾O_hIt is output as (t). In FIG. 2, X_h(T) (h = 1, 2,..., H) is X_h(H = 1, 2,..., H).
Second layer units {c, k, k ', m} (c = 1,..., C; k, k' = 1,..., K_c; M = 1, ..., M_{c, k}) Is the weighting factor w^c _{k ' , K, m, h}  Receive the product multiplied by. The input / output relationship of the second layer is given by the following equations (3) and (4).
[0017]
[Equation 3]

Where C is the number of reference directions and K_cIs a parameter corresponding to the number of states of the hidden Markov model, M_{c, k}Represents the number of components of the mixed Gaussian distribution model corresponding to the reference direction c and the state k.
[0018]
The input to the unit {c, k, k '} of the third layer is the unit {c, k, k', m} (m = 1) of the second layer.
, ... M_{c, k}) Output. Also, the value obtained by multiplying the input by the output of the fourth layer one time ago becomes the output of the third layer. The input / output relationship of the third layer is given by the following equations (5) and (6).
[0019]
[Expression 4]

However, the initial state is⁽⁴⁾O^c _{k '}(0) = 1.0.
Furthermore, the fourth layer receives the integrated output of the units {c, k, k '} of the third layer.
The input / output relationship is given by the following equations (7) and (8).
[0020]
[Equation 5]

[0021]
Finally, unit c of the fifth layer is K of the fourth layer_cUnit {c, k} (k = 1,..., K_c) Output. The input / output relationship is given by the following equations (9) and (10).
[0022]
[Formula 6]

This output⁽⁵⁾O^c(T) is the posterior probability with respect to the reference direction c.
As described above, this network has a recurrent coupling between the third layer and the fourth layer, so that the output one time before can be taken into consideration. Further, the posterior probability with respect to the reference direction can be modeled based on the mixed normal distribution model only by learningly adjusting the weighting coefficient w between the first layer and the second layer.
[0023]
At the time of learning, first, the target moving direction φ (0 ≦ φ <2π) of the pointer is randomly presented from the system to the user. However, φ is defined so that the x-axis direction in FIG. 1 is 0 and increases counterclockwise. The user contracts each muscle with the intention of moving the pointer in the specified direction. N sets of EMG signals and target moving directions φ at this time are collected and used as a learning sample data set.
Now, at time T, the nth input vector x (t)^(N)Teacher vector T corresponding to^(N)(Φ) = (T₁ ^(N)(Φ), ..., T_c ^(N)(Φ), ..., T_c ^(N)(Φ))^T(N = 1,..., N) is calculated by the equation (11).
[0024]
[Expression 7]

Φ_cIs defined as the following equation (12).
φ_c= 2πc / C (12)
This teacher signal corresponds to a component when the target movement direction φ is decomposed into two reference directions sandwiching the direction. Of course, since it is a probability, the sum is always 1. As an example, FIG. 3 shows a teacher signal when the reference direction is four directions of φ = 0, π / 2, π, and 3 / 2π rad.
As can be seen from the figure, when the target movement direction does not coincide with the reference direction, two reference directions close to φ are selected, and a teacher signal is given to the output unit corresponding to the direction. By this method, directions other than the reference direction can be used as learning data.
[0025]
The R-LLGMN learns using L time series signals (N = LxC) prepared in each of the C reference directions as learning data.
The network evaluation function J for the learning data is defined as the following equation (13):
[0026]
[Equation 8]

Learning is performed to minimize this, that is, to maximize the log likelihood. However, Y^c(T) is the input vector x (t)^(N)Means output at time T. In order to consider the effect of recurrent coupling included in R-LLGMN, P.P. J. et al. Werbos: “Backpropagation through time: what it does and how to doit”, Proceedings of the IEEE, Vol. 78, no. 10, 1550/1560, (1990) is used. This is a method of accumulating error gradients in a time series and calculating a weight correction amount.
Here, for the purpose of reducing the mental burden of the operator who waits for the convergence time of learning as much as possible, Zak: “Terminal Attractors for Addressable Memory in Neural Networks”, Physics Letters A, Vol. A terminal attractor proposed in 133, 18/22 (1988) was introduced on the learning side, and an upper limit value of the convergence time could be set.
[0027]
[Pointer control unit] (Moving direction determination unit)
Here, the movement of the pointer is calculated based on the output from the R-LLGMN and the estimated muscle activity level α (t).
The output from the R-LLGMN indicates the possibility that the pointer moves in each reference direction c as shown in FIG. Therefore, the pointer moving direction vector e (t) = (e_X(T), e_Y(T)) is defined as the following equations (14) to (17).
[0028]
[Equation 9]

[0029]
[Expression 10]

Where v_X(T), v_Y(T) shows the x component and y component of the vector component corresponding to each reference direction.
[0030]
[Pointer control unit] (Pointer movement amount calculation unit)
Further, the muscle activity level α (t) is calculated according to the following equation (18).
[0031]
## EQU11 ##

Where EMG_c ^maxbar is the EMG signal EMG when the maximum voluntary muscle contraction for each reference direction_cl ^max  This is a time average of the channel sum of (t). By using such a muscle activity level, the difference in the maximum voluntary muscle contraction level in each direction can be adjusted, and this is used to determine the moving speed of the pointer.
[0032]
In order to realize the operation feeling of the pointer that matches the user's power characteristics, it is considered that the movement of the pointer according to the physical rules we experience everyday is suitable. Therefore, based on the muscle activity level α (t) and the moving direction vector e (t), how the pointer moves is calculated using an impedance model of the following equation (19).
[0033]
[Expression 12]

M_e, B_eAre inertia and viscosity parameters, respectively, and p (t) is a pointer position vector. An elastic force may be added to the impedance model. In that case, E on the left side of equation (19)_e  Add p (t). Where E_eIs an elastic parameter. F (t) is a driving force applied to the pointer, and is calculated from α (t) and e (t) using the 0th-order hold as shown in equation (20).
[0034]
[Formula 13]

However, g represents a force gain. α₀Is a threshold value for determining the presence or absence of a pointer operation.
Equation (19) is viscous friction B_eMass M placed in a plane with_eRepresents an equation of motion when a force F (t) is applied to the object. By calculating the position and speed of the pointer by numerical integration according to such a physical model, a natural feeling of operation corresponding to the operator's sense of power can be expected.
[0035]
[Experiment]
An experiment was conducted to examine the operability of the constructed embodiment apparatus. There were 6 subjects, and subjects A, B, C, D, and E were all healthy male students (22, 23, 22, 22, 24). Subject F is a male (45 years old) who has cut the forearm at a site about 15 cm from the right wrist due to an accident four years ago. All subjects practiced in advance.
The electrodes for measuring EMG signals (NT-511G, NT-512G: manufactured by Nihon Kohden Co., Ltd.) were Ag / AgCl, circular, bipolar differential type with a diameter of 0.012 m. The number of use is 7ch, 6ch in the right forearm (gap side carpal extensor, heel side carpal flexor, ulnar carpal extensor, ulnar carpal flexor, deep finger flexor, eigen finger extensor), 1ch in the left forearm A heel side carpal extensor was attached. The left forearm electrode is for generating a click event. However, these electrode mounting positions can be adapted to some extent by a neural network introduced into the system even if they are freely mounted. In clinical use, users with insufficient knowledge about EMG signals are assumed, and it is considered difficult to mount electrodes on specific muscles. Therefore, such an adaptive function is considered effective.
In the experiment, neural network learning was performed with the pointer moving direction corresponding to the bending direction of the right wrist. However, in the case of amputees, they were instructed to operate the mouse based on the wrist movement image left in the brain.
The sampling frequency of data is 100 [Hz], the number of components and the number of states, which are parameters of R-LLGMN, are M = 1, K = 1, and the reference direction used for learning is from φ = 0 to π / 2 rad increments. 4 directions were set. In this experiment, the display size was 17 inches and the resolution was 640 × 480.
[0036]
[Operation example]
FIG. 4 and FIG. 5 show results when subject A draws a circle as an example of the operation. The time series length at the time of R-LLGMN learning was set to 20, and the time series length at the time of estimation was set to 5. FIG. 4 shows the locus of the pointer on the display, and a drag operation is performed in a section where the color of the pointer is dark. FIG. 5 shows the state of signal processing during control, and shows the EMG signal, the muscle activity level α (t), the click signal β (t), the moving speed of the pointer, and the moving direction in order from the top. From this figure, it can be seen that the pointer can be arbitrarily controlled using the EMG signal.
[0037]
Next, FIG. 6 and FIG. 7 show examples of operations by the subject F (upper limb amputee). Here, a task is executed in which several circles are randomly arranged on the display and the circles are clicked in order. The number of electrodes used was 7ch for the remaining muscles of the forearm section cut at 8ch, and 1ch for the click event was attached to the left forearm part (radial carpal extensor muscle). The time series length at the time of R-LLGMN learning was set to 20, and the time series length at the time of estimation was set to 5. 6 shows the locus of the pointer on the display, and FIG. 7 shows the state of signal processing during control. From the top, the EMG signal, the muscle activity level α (t), the click signal β (t), and the moving speed of the pointer are shown. , Shows the direction of movement. It can be seen that this system can provide smooth pointing even for an amputee.
[0038]
[Verification of direction accuracy]
In order to investigate the control characteristics of the developed system in all directions, the directional accuracy was verified. Each subject was instructed to move the wrist in 16 directions of π / 8 rad for 2 seconds in each direction without feedback from the display. Then, an error between the pointer movement direction estimated from the measured EMG signal and the direction instructed to the operator was calculated as an evaluation index.
Fig. 8 shows the experimental results. FIG. 8A shows the result of the subject A, and FIG. 8B shows the result of the subject B.
An error average calculated by changing the time series length of the feature pattern input at the time of learning and estimation at 1600 from 1 to 40 in total is shown. The error average is an average value of errors for all directions. From the figure, it can be seen that the time series length at the time of learning is long for each subject, and the error decreases as the time series length at the time of estimation becomes shorter.
[0039]
Here, there was a difference in the direction accuracy between the learning direction and the unlearned direction. Therefore, the experimental results in Fig. 8 were recalculated separately for the 4 directions learned and the 12 unlearned directions. 9 and 10 are averages of errors in the learning direction and the unlearned direction, respectively. (A) is the result of the subject A, and (b) is the result of the subject B. From this figure, it can be seen that as the time series length at the time of learning is longer and the time series length at the time of estimation becomes shorter, the error increases in the learned direction and decreases in the unlearned direction. This is because the time series length at the time of estimation is shorter than that at the time of learning, the information amount is reduced, and the output of the posterior probability from the R-LLGMN is ambiguous. From this result, it can be seen that, in order to improve the accuracy in all directions on the average, the error in the learned direction increases, but the estimated signal should be shorter. In the following, it is assumed that the operation is performed with the time series length at the time of learning set to 10 and the time series length at the time of estimation set to 1.
[0040]
Next, in order to show the effectiveness of R-LLGMN in this system, a comparison experiment with a system using LLGMN and Back Propagation Neural Network (hereinafter referred to as BPNN) was performed. LLGMN introduces a mixed normal distribution model as a structure, and corresponds to a case where recurrent coupling is removed from R-LLGMN. The subject was instructed to move the pointer in the specified direction with feedback from the display, and the direction estimation accuracy at that time was obtained.
In this experiment, the number of components of LLGMN, the learning time, learning rate, and sampling time which are parameters of the terminal attractor were set to the same number as those of R-LLGMN. Regarding the number of BPNN units, the input layer 6, the second layer 7, the third layer 6, and the output layer 4 were determined by trial and error so that the direction estimation accuracy was increased. In the pointer movement direction estimation process, normalization was performed so that the sum of BPNN outputs was 1, and direction estimation was performed using the same algorithm as in the case of R-LLGMN. The learning evaluation function uses a square error with the teacher signal, and the learning rate, inertia coefficient, and learning end error threshold parameters are set to 0.01, 0.001, and 0.0003, respectively. However, if the value of the evaluation function does not fall below the error threshold after 50,000 iterations, the learning is terminated because the convergence is impossible.
FIG. 11 shows the experimental results of subject B. Each radar chart shows the mean and standard deviation of error in 2 seconds after showing each designated direction,
(A) is the result of LLGMN, (b) is the result of BPNN, and (c) is the result of R-LLGMN. From this figure, it can be seen that R-LLGMN has no variation in direction accuracy compared to BPNN or LLGMN, and is correctly estimated in the indicated direction.
[0041]
[Verification of operability]
Finally, an experiment was conducted to evaluate the operability of the device of the present invention. A screen presented to the subject during the experiment is shown in FIG. In the figure, the size of the pointer, the size of the target, the distance between the pointer and the target, and the ratio of each are the same as those displayed on the display. As shown in this figure, the subject performed a task of reciprocating between two points displayed on the display. The distance between the two points is 400 pixels, the direction of movement of the pointer is six directions of π / 6 rad, and the integrated value of the operation time in the reciprocating operation and the deviation from the shortest trajectory is calculated. The operation for all six directions was taken as one session, and the average value and standard deviation of the data of five sessions were obtained. The settings of R-LLGMN, LLGMN, and BPNN were the same as those in [Verification of direction accuracy].
Shows the experimental results.
The experimental results are shown in FIGS. FIG. 13 shows the average time of reciprocating scanning, and FIG. 14 shows the integrated value of deviation from the shortest activation. Further, the results of BPNN and LLGMN were tested based on the results of R-LLGMN in FIG. 14 (one-sided test). Here, the mark is shown in the table for each result judged to have a significant difference at a risk rate of 5.1%. Comparing each method, R-LLGMN tended to be smaller in both average time and orbit error, and it was confirmed that operability was the highest.
[0042]
Although the preferred embodiment has been described above, the present invention is not limited to this, and appropriate modifications can be made without departing from the scope of the present invention. For example, in the embodiment, the EMG signal is used as the biological signal, but signals such as brain waves, limb movement (acceleration), and contact pressure can be used instead of or in combination with this signal.
[0043]
【The invention's effect】
As described above, the pointing device of the present invention estimates the moving direction of the pointer using a neural network having a recurrent structure, and therefore can accurately cope with a rapidly changing biological signal. In addition, it is possible to provide an omnidirectional pointing device that can smoothly move in a direction other than the reference direction that is not learned.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an outline of a system of the present invention.
FIG. 2 is an operation explanatory diagram of a neural network (R-LLGMN) used in the pointing device of the present invention.
FIG. 3 is a diagram showing a relationship between a reference direction and a teacher signal for explaining a learning process in the pointing device of the present invention.
FIG. 4 is a diagram showing a result of using the pointing device of the present invention (part 1);
5 is a signal waveform diagram of each part when the result of FIG. 4 is obtained.
FIG. 6 is a diagram showing a result of using the pointing device of the present invention (part 2);
7 is a signal waveform diagram of each part when the result of FIG. 6 is obtained.
FIG. 8 is a graph showing a directional error using a time series length during learning and estimation as a parameter (part 1).
FIG. 9 is a graph showing a directional error using a time series length during learning and estimation as a parameter (part 2).
FIG. 10 is a graph showing a direction error using a time series length at the time of learning and estimation as a parameter (part 3).
FIG. 11 is a diagram showing the accuracy of the estimated direction when R-LLGMN is used for the neural network unit and when another method is used.
FIG. 12 is a diagram showing target points on a display used for verifying the operability of the pointing device of the present invention.
FIG. 13 is a chart showing the average time of reciprocating operations when R-LLGMN is used for the neural network unit and when another method is used.
FIG. 14 is a chart showing an integral value of deviation from the shortest trajectory when R-LLGMN is used for the neural network unit and when another method is used.

Claims (14)

A biological signal collecting unit that collects biological signals, a biological signal processing unit that extracts a feature pattern from the biological signal, a neural network unit that calculates a probability of a pointer movement direction intended by a user from the feature pattern, and a neural network A moving direction determining unit that determines a moving direction of the pointer from the probability calculated by the net unit, and a pointer moving that calculates a moving amount of the pointer based on a physical model based on the moving direction of the pointer and an average intensity of the biological signal A logarithmic linearized mixed normal distribution network R− in which the neural network unit has a recurrent structure in a pointing device including a pointer control unit having a quantity calculation unit and a parameter storage unit that stores parameters used by each unit in a readable manner. LLGMN (Recurrent Log-Linear A pointing device which is characterized in that utilizes a zed Gaussian Mixture Network). 2. The pointing device according to claim 1, wherein the biological signal collection unit can select and combine myoelectric potential, electroencephalogram, limb movement (acceleration), and contact pressure as inputs. The pointing device according to claim 1, wherein rectification processing, filtering processing, normalization processing, and peak detection processing are performed in the biological signal processing unit. The pointing device according to claim 1, wherein rectification processing, filtering processing, frequency analysis processing, normalization processing, and peak detection processing are performed in the biological signal processing unit. The pointing device according to claim 1, wherein the neural network unit includes a learning mechanism using a neural network in order to adapt to individual differences included in a biological signal to be used. The pointing device according to claim 1, wherein the neural network unit calculates a probability that the pointer moves in a plurality of reference directions such as left and right and up and down with respect to the obtained feature pattern. 7. The pointing device according to claim 6, wherein the moving direction determining unit determines the moving direction based on the pointer moving probability with respect to each reference direction calculated by the neural network unit. The pointer control unit calculates a movement amount of the pointer according to a physical model composed of inertia and viscosity by calculating a force intended by the user from a movement direction of the pointer and an average intensity of the biological signal. Item 10. The pointing device according to Item 1. The pointer control unit calculates a force intended by the user from a moving direction of the pointer and an average intensity of the biological signal, and calculates a moving amount of the pointer according to a physical model including inertia, viscosity, and elasticity. The pointing device according to claim 1. The pointing device according to claim 1, further comprising a feedback unit that presents a processing status of each unit during pointer control to a user. 11. The feedback unit graphically presents at least one of the measured biological signal, the feature pattern unit, the output of the neural network unit, the movement direction estimation result, and the movement amount to the user. The pointing device described. The data stored in the parameter storage unit is related to the type of signal selected in the biological signal collection unit, the cutoff frequency of the filter used when extracting the feature pattern in the biological signal processing unit, and the peak value of the signal. 2. The pointing according to claim 1, wherein one or more of parameters relating to a neural network such as a threshold value and a weighting coefficient in the neural network unit, and inertia and viscosity parameters in the pointer control unit are included. device. The pointing device according to claim 1, wherein the parameter used in the neural network unit stored in the parameter storage unit is stored for each user. The pointing device according to claim 1, wherein the entire system is configured to be wearable.

Images