JP2004502470A - System and method for detecting head movement and generating output responsive thereto - Google Patents

Abstract

How to generate control signals. The method includes the steps of sensing a pneumatic pattern within or near a human ear, wherein the pneumatic pattern results from spontaneous physical movement of the human; And converting the air pressure pattern into an electrical output signal. A control system is also described. The control system includes a pneumatic sensor and a processing circuit. The air pressure sensor detects an air pressure pattern near a human ear when the human is performing spontaneous physical movement, and generates an electric output signal corresponding to the detected air pressure pattern. The processing circuit identifies a pattern from the electrical output signal.
[Selection diagram] Fig. 1

Description

ここで、Ｆ（ｆ（ｔ））は、従来のフーリエ変換である。当業者に周知であるように、高速フーリエ変換は離散フーリエ変換を計算するための効率的なアルゴリズムであるので、高速フーリエ変換は、従来のフーリエ変換に関連している。デジタル信号データは、高速フーリエ変換を介して周波数ドメインに変換された後、ステップ１７２において、舌動作に関連するデータをノイズデータと区別するように周波数ドメインデータを処理する。当業者に周知なように、データノイズからの分離は、しばしば、周波数ドメインにおいて単純化される。なぜなら、ノイズと異なり、データ信号はある物理特性を有するからである。時間ドメインにおけるデータ信号はノイズより小さい振幅を有するが、周波数ドメインにおけるデータ信号はノイズより大きい振幅を有するからである。したがって、高速フーリエ変換は、ノイズを分離するための一般的な方法である。
【００３９】
デジタル信号データのデータ処理の周辺の詳細は、当業者に周知なさまざまなデータ処理技術を介して為され得、任意のデータ処理方法論は、本発明の範囲内にあると企図される。多くの異なるデータ処理方法論が使用され得るが、好適な方法論は、以下の方法およびシステムに関連して下記に示す。
【００４０】
図１０を参照して、本発明のシステム２００および方法を示す。１つ以上のセンサ１２はユーザの耳（単数または複数）から空気圧を検知する。センサ（単数または複数）からのアナログ信号（単数または複数）は、データ獲得カード、アナログ‐デジタルコンバータおよびバッファ１６／１７を介してプロセッサ１８に供給される。舌が動いている間、センサから受信した信号は、少なくともそれぞれの時間周期に対して実質的に連続的である。したがって、バッファ１７はプロセッサ１８による解析のための信号の十分な部分を格納可能である。
【００４１】
プロセッサ１８は、バンドパスフィルタ２０２と、パワースペクトルフィルタ２０４と、信号検出器２０６と、特徴抽出部２０８と、神経網２１０とを含む。システム２００は、また、所望な出力周辺機器２０に結合され得る出力部２１２を含む。
【００４２】
システム２００全体の動作を要約すると、センサ（１２）およびデータ獲得、Ａ／Ｄおよびバッファ１６／１７は、バンドバスフィルタ２０２に信号を供給する。バンドパスフィルタ２０２に供給される生データまたは生信号の例を、図１１のグラフの２２０で示す。生信号は、バンドパスフィルタに提供され、バンドパスフィルタによって高速フーリエ変換技術を用いて周波数ドメインに操作される。高速フーリエ変換はデジタル信号を周波数ドメインに変換する。バンドパスフィルタでは、舌動作を表す周波数範囲の上または下の周波数が、例えば、信号から取り除かれる。対象の周波数範囲より高い周波数および対象の周波数範囲より低い周波数は、一般的に、ノイズの原因となる。バンドパスフィルタ化によってフィルタリングされた後の信号の例を図１２のグラフの２２２で図示する。さまざまなバンドパスフィルタを用いて、所望な周波数範囲より高い信号周波数および所望な周波数範囲より低い信号周波数を取り除くようにバンドパスフィルタ化を行い得る。
【００４３】
２２２で表される信号の一部（例えば、部分２２２ａおよび部分２２２ｂ）は、舌動作を表さないかもしれない。例えば、それらは比較的パワーが低く、したがって、依然としてノイズかもしれない。したがって、パワースペクトルフィルタ２０４は、信号のさまざまな部分のパワーに従ってさらにフィルタ化され、舌動作に起因し、したがって、さらなる解析を考慮すべき部分を決定する。パワーフィルタ化は、図１２の信号を、図１３の２２４ａ、２２４ｂ、．．．２２４ｉで表されるような複数のセグメントまたはウィンドウにスライス化することによって実行され得る。図１２および図１３のグラフは異なるスケールであることが認識される。したがって、ウィンドウ２２４ａ〜２２４ｉは、時間ｔ_ａと時間ｔ_ｂとの間の時間フレームをスライスする。ここで、時間ｔ_ａと時間ｔ_ｂとの間の同じ時間フレームを図１２の信号部分２２２ｃに対して図示する。
【００４４】
パワーフィルタ化は、信号のパワーが所定の閾値または値を越えず、それにより、恐らくノイズを表す周波数を抑制するか、または、除去する。ノイズ成分をを除去することによって、舌動作を表す信号が見出され得る。パワーフィルタ化を行うために、例えば、信号のパワースペクトルを以下の公式を用いて計算する。
【００４５】
Ｐ_ｆ＝Ｍａｇ_ｆ ^２＋Ｐｈａｓｅ_ｆ ^２
ここで、ｆ＝Ｆ_０．．．Ｆ_ｌ、および［Ｆ_０；Ｆ_ｌ］は舌動作から生じる信号が期待される周波数範囲を表す。
【００４６】
言い換えると、パワーは、大きさの２乗と位相の２乗との和の関数である。それぞれのウィンドウ２２４ａ、２２４ｂ．．．２２４ｉに対して上のように計算されたパワーは、図１４に示されるグラフ２２６を作成するように再構築される。グラフ２２６は、上記の計算からの値２２６ａ、２２６ｂ．．．２２６ｉを示す。曲線２２６は比較的滑らかであり、舌動作によって生じる信号が図１２のグラフ２２２内に見出され得るパワースペクトルを表す。舌動作によって生じる信号は、通常、比較的高いパワーであり、グラフ２２６のほぼ真中の時間に、例えば、図１３に示される時間フレームＴに対して生じる。時間内において信号２２２が舌動作を表している場合に、信号を検出し、または位置付けるのは、この時間フレームが生じる時間およびその時間フレームの期間である。したがって、実際には、システム２００の信号検出器構成要素２０６は、図１２および図１３のグラフとここで記載された解析とを組みあわせて、信号検出機能が提供される。時間フレームＴは、また、図１２のグラフに示され、それにより、舌動作によって生成されると考慮されるべき信号２２２ｃの（図示されるブロックウィンドウ２２２ｅにおける）部分２２２ｄを指定する。
【００４７】
次いで、信号部分２２２ｄ、または、少なくともその信号の１つの特徴は、神経網２１０に供給され得る。特徴抽出原理を用いて、信号部分２２２ｄの１つの特徴は、神経網に供給される。神経網２１０において、その特徴は、神経網が慣らされた他の特徴と比較される。神経網の慣れは周知の手順である。神経網は、入力特徴（例えば、信号部２２２ｄの代表的なもの）の許容可能に整合するか、または、神経網の慣れに許容可能に相関するかどうかを判定し得、そうである場合、神経網は出力２１２の生成を生じさせる。出力２１２は、コンピュータマウス、ジョイステックまたは他のデバイスによって生成される信号のタイプに類似した電気信号であり得る。したがって、口の前、下、左から、口の後ろ、上、右へユーザの舌を移動させることによる舌動作は、マウスパッドの左下から右上へのマウスの移動に類似した出力信号を生成し得る。別の例として、口の左前からマウスの右前までの舌の移動により、マウスを左から右へ移動したのと類似する信号を生じ得る。他も同様である。
【００４８】
例えば、信号２２２から抽出される特徴は、ある特性を示す特徴、または、他のものからの信号を表す曲線を区別する特徴であり得る。周知のさまざまな特徴が、特徴抽出方法において使用される。本発明において、例示の特徴は、信号部分２２２ｄの滑らかなパワースペクトルである。他の特徴は、パワースペクトルの対数、パワースペクトルそれ自体などであり得る。
【００４９】
神経網２１０は、一般的な形態で慣らされた従来の神経網であり得る。神経網２１０は、信号部分２２２ｄの特徴を、その慣らされたものと比較し、その特徴がある慣らされた特徴または別の慣らされた特徴により近いかどうかを判定し得る。信号部分２２２ｄの入力特徴がある慣らされた特徴に十分に近い場合、神経網は、上述したように、例えば、その出力において適切な信号を生成させる。入力特徴は、慣らされた特徴の任意のものと十分には似ていない場合、または、２つ以上の慣らされた特徴と比較的類似している場合、神経網は出力信号を生成させるための出力を生じさない。さもなければ、不確かな、望ましくない出力信号を生成される。
【００５０】
上述したように、２つのセンサ１２を使用すれば、システム２００はそれぞれのセンサからの各々の信号について順次的に動作し得るか、または、パラレル処理機能を含み得、それにより、両方のセンサからの信号は実質的に同時に処理され得る。出力部２１２は、両方のセンサから生じる信号の信号解析の結果を反映し得、出力部２１２において、実際には３次元データを反映する出力信号を供給するために用い得る。
【００５１】
特定の好適な実施形態または特定の好適な複数の実施形態に関して本発明を示し、説明してきたが、本明細書および添付の図面を読み、理解した他の当業者が、等価な代替および改変を想起することは明らかである。特に、上記の構成要素（アセンブリ、デバイス、回路など）によって実行された様々な機能に関して、そのような構成要素を説明するために使用された用語（「手段」と参照されるものを含む）は、本発明の例示的な実施形態を図示した本明細書における機能を実行する開示された構造に構造的に等価でなくても、記載された構成要素の特定の機能を実行する任意の構成要素（すなわち、機能的に等価なもの）に対応するものを意図する。さらに、本発明の特定の特徴は、いくつかの実施形態の１つのみに関して示されているかもしれないが、そのような特徴は、任意の所与の用途または特定の用途に対して所望であり、利点を有し得る場合、他の実施形態の１つ以上の他の特徴と組み合わされ得る。
【図面の簡単な説明】
【図１】
図１は、本発明による、本明細書中ではしばしば舌の動作とも呼ばれる口の動作が検知され、その口の動きを使って、検知された口の動作に対応する制御機能を提供するシステムを示すブロックレベル図である。
【図２ａ】
図２ａは、本発明による、表示と連結されたプロセッサと電気的な通信を行う耳の近くに設置された圧力センサを有するユーザの環境図である。
【図２ｂ】
図２ｂは、本発明の１側面による、耳の近くの空気圧を監視する圧力センサをより詳細に示す図２ａの一部の拡大図である。
【図２ｃ】
図２ｃは、本発明による、空気内の波の映像であり、複数の縦の分子的密度変化を表す図である。
【図２ｄ】
図２ｄは、本発明による、圧力センサに対して局所的な位置の空気圧の変化を示す圧力センサと連結されたトランスデューサによって生成される、模範的な電気信号を示すグラフである。
【図３】
図３は、本発明による、口の動作を検知し、検知された口の動作に反応して制御指示を提供する方法を示すフローチャートである。
【図４】
図４は、本発明による、空気圧を監視する方法を示すフローチャートである。
【図５】
図５は、思考を検知するために空気圧に対応する電気信号を処理する方法を示すフローチャートである。
【図６】
図６は、本発明による、アナログ電気信号のデジタル信号データへの変換を示すグラフである。
【図７】
図７は、本発明による、デジタル信号データを解析する方法を示すフロー−チャートである。
【図８】
図８は、本発明による、図７のフローチャートとともにデジタル信号データを解析する方法を示すグラフである。
【図９】
図９は、本発明による、デジタル信号データを処理する方法を示すフローチャートである。
【図１０】
図１０は、本発明の部分および機能を含むシステムを示すシステム図である。
【図１１】
図１１は、口の動作を表す空気圧を検知するのに本発明において使用されるセンサによって検知される、模範的な未処理信号情報を表すグラフである。
【図１２】
図１２は、図１２の周波数フィルタリングされた信号情報のグラフであり、例えば、周波数をフィルタリングして騒音を除去したグラフである。
【図１３】
図１３は、頭の動作を表す信号を遅れずに位置付けするのに使用する図１２の曲線のパワースペクトルのウィンドウを表すグラフである。
【図１４】
図１４は、このようなパワースペクトルのグラフである。[0001]
(Cross-reference with related applications)
This application is filed on Nov. 9, 1999, US Provisional Patent Application Serial No. 60/60, Serial No. 60/61, filed from "System @ Method @ Method @ Detecting @ an @ Action @ of @ Head @ and @ Generating @ an @ Output @ in @ Response @ Thereto". And the above disclosure is hereby incorporated by reference in its entirety.
[0002]
(Field of the Invention)
The present invention generally uses sensors to detect human head movements, such as muscle burning, movement, and tongue movements, related to the oral cavity, and responds to such detections, e.g., outputs such as system control functions. Regarding the offer. More specifically, the present invention analyzes the change in air pressure near the ear in humans and processes the collected data to determine the correct control functionality corresponding to the movement of the tongue in the oral cavity. , Position, and motion.
[0003]
(Background of the Invention)
An apparatus and method for detecting thoughts is disclosed in U.S. Patent No. 6,024,700, the entire disclosure of which is incorporated by reference. Such systems and methods may require convergence to consistently develop the same thought and detect it consistently. Training may be necessary to achieve the desired concentration and consistency. Such concentration can be relatively cumbersome. Thought can result in air pressure or sound that can be detected in one or both ears of a user using such systems and methods, and such air pressure can be detected by a detector such as a microphone. Exemplary outputs from such systems and methods of detecting thoughts may be one or more independent outputs, such as independent outputs indicating alphanumeric, direction, etc., respectively.
[0004]
The sound produced by the human ear is sometimes called automatic sound. Auto-acoustic sounds have been used in the past, for example, for diagnostics to indicate ear function or ear condition. For example, it has been found that in response to sound input to the ear, the ear provides a detectable automatic sound output.
[0005]
Currently, there are available controls that enable disabled people who cannot use their arms or legs to operate an electric wheelchair. Such controls include a trackball or joystick-type device that is inserted into the mouth of the individual, and the individual operates the controls using movements of the mouth and / or tongue to provide mechanical input. Such devices are difficult to operate and are troublesome if they cause a feeling of falling off or irritating in the mouth.
[0006]
There are other controls that allow a disabled individual to perform functions with mechanical help. Operation of such a controller typically requires an incident electromagnetic signal, such as a signal such as infrared or ultraviolet light / radiation directed to the individual. Thereafter, depending on the identified action or activity, the individual may modulate the incident electromagnetic signal, for example, by blinking, moving his head, or by any such action. The need for an incident electromagnetic signal is inconvenient for several reasons. The reason is that one special device is needed to provide the signal, the signal must be correctly oriented and aligned, the detector and the modulated signal must be correctly aligned , The signal must not adversely affect the user (eg, cause eye injury), and so on.
[0007]
(Summary of the Invention)
According to one aspect of the invention, the invention is a method for generating a control signal. The method includes sensing a pneumatic pattern in or near a person's ear, a pneumatic pattern resulting from a person's spontaneous physical movement, and converting the pneumatic pattern into an output signal.
[0008]
According to another aspect of the invention, the invention is a method for detecting spontaneous movement of a person by monitoring air pressure near a person's ear.
[0009]
According to another aspect of the invention, the invention is a control system. The control system detects a pneumatic pattern adjacent to the human ear during a person's spontaneous physical movement and generates an output signal corresponding to the detected pneumatic pattern, and an output signal thereof. And a processing circuit for identifying the pattern from
[0010]
(Detailed description of the present invention)
The following description sets forth a detailed description of the present invention in connection with the accompanying drawings, wherein like reference numerals are consistent with like elements.
[0011]
The present invention relates to an apparatus and a method for detecting movements of a human head, such as movements occurring in connection with the oral cavity, wherein these movements can provide only a distributed output, but rather an analog type signal. As in, an output range may be generated. As an example, movement of the tongue in the oral cavity, such as, for example, two-dimensional or three-dimensional movement, can be detected or sensed, providing a useful output. Exemplary outputs may be in the form of signals similar to those generated by a computer mouse, click buttons on a mouse, computer joystick, and the like.
[0012]
Using the invention as found is determined to be somewhat cumbersome and typically requires less concentration than the thought detection apparatus and method described above. Also, the present invention can provide an output having the same persistent characteristics as a signal generated, for example, by moving a computer mouse along a surface. Still further, because the tongue can move in three dimensions, the present invention provides a computer having up, down, left, right, back and forth signal outputs that includes the speed characteristics of the signals associated with the speed at which the joystick is moved in each position or direction. A three-dimensional type of signal output, such as a signal output generated by a joystick, can be provided.
[0013]
The invention can be used for several purposes. One exemplary use provides input to a computer or other electrical or electronic device. Although the exemplary input is an electrical signal, it is recognized that the input can be other than electrical, for example, optical, fluid, and the like. The input may control a computer or other device, or may be used for any other purpose. In order to simplify and avoid the complexity of the description herein, the present invention may be implemented, for example, on a personal computer or any other computer, whether digital or analog. It will be described in terms of providing electrical input and used to control the operation of the computer.
[0014]
One example of the use of the present invention in combination with a computer controlled with the present invention is to provide functions and capabilities for the disabled. For example, the present invention provides that a disabled or non-disabled person controls the operation of one machine in a manufacturing facility, controls the operation of a motorized wheelchair or similar device, controls the operation of a video game or the like, Can be used to enable playing video games. It is recognized that these are only examples and that the present invention can be used for many other purposes.
[0015]
In the present invention, an indeterminate electromagnetic signal requiring modification to produce a useful output is not required. Rather, in the present invention, the individual movements that occur in the individual's head are used, for example, with respect to the oral cavity, such as tongue movements, tongues against teeth or pallets, and the use of barometric pressure or pressure at one or both ears of the individual. Sounds may be generated and these may be detected or sensed and provided as outputs of the present invention as inputs to a computer or any other device. Thus, the present invention provides a signal generated by the action of an individual without requiring a non-irritating signal, e.g., an indeterminate input such as an indeterminate electromagnetic signal.
[0016]
According to embodiments of the present invention, the sensor senses or detects barometric pressure, or a change in barometric pressure, or sound produced by an individual's ear. For simplicity of the description herein, the event of the sensing or detection step is called the detection step and what is detected is called the barometric pressure. Further, to simplify the description herein, the act of raising a detectable barometric pressure is referred to as tongue movement in the oral cavity, and is sometimes referred to as "tongue movement". The present invention contemplates movement or movement of other muscles in or relative to the oral cavity, or even of the head itself, or movement of the tongue against teeth or pawls or tongue tapping, lung discharge or inhalation, nose movement That allow the detection of barometric pressure at one or both ears of an individual and can be used according to the present invention. Another movement of the detectable head may be to rotate the head or to tilt the head. An exemplary advantage of using tongue movement to effect barometric pressure detection is that tongue movement can be two-dimensional or three-dimensional (two-dimensional is up-down or left-right movement, and three-dimensional is back-and-forth , Left and right and up and down).
[0017]
The present invention relates to systems and methods for detecting motion of a human head, where in the following example the motion is tongue motion or other motion with respect to the oral cavity and an output useful in response to such detection I will provide a. The present invention provides for a substantially real-time detection and control system that monitors changes in air pressure near the human ear that occur almost instantaneously in response to tongue movement. In addition, monitoring of barometric pressure is passive, thus avoiding potential health and / or environmental concerns for the subject to electrical signals and electromagnetic radiation. In addition, changes in barometric pressure uniquely correspond to one of the various tongue movements (eg, having a unique signal signature), and multiple different tongue movements for systems that require multiple control functions. Allow it to be detected and identified.
[0018]
According to one aspect of the present invention, a system for detecting movement of a tongue and providing a corresponding output in response to the detection includes a pressure sensor located near a user's ear. The sensor is in electrical communication with the processing circuitry to sense a change in barometric pressure near the ear caused by movement of the user's tongue and convert the barometric pressure change into an electrical signal. This electrical signal is then processed by a processing circuit to detect in the electrical signal the presence and type of tongue movement and to respond to control commands, or tongue movements specific to the output peripheral for execution. Provide any other useful output, such as an output similar to that produced by a computer mouse or joystick. Exemplary control functions may include controlling a video game display, controlling one medical device such as a wheelchair, and controlling computer functions to execute a hand-free mouse, joystick, etc. However, it is not limited to these.
[0019]
According to another aspect of the present invention, there is provided a method of detecting tongue movement and providing a control command in response to the tongue movement. Tongue movement is monitored by monitoring barometric pressure, which changes near the ear in response to tongue movement and noise. The method includes converting barometric pressure data into an analog electrical signal, which is substantially converted to digital signal data for further processing. Data signal processing is performed to analyze the data and separate noise from tongue movement data, thereby detecting the presence of tongue movement. Further processing then determines the detected movement of the tongue and provides one or more useful outputs, i.e., control commands or any other output, to the output peripheral to perform the appropriate function. Used for
[0020]
In an embodiment of the present invention, the pressure sensor and the transducer include a microphone, and the data signal processing includes transforming the data segment into the frequency domain to distinguish data relating to tongue movement from noise. In addition, band signal filtering to isolate noise occurring at frequencies above and below the frequency band in which the desired or signal is expected, and signals generated with less power than the threshold, i.e., considered as noise There is power spectral filtering that removes the signal that is generated. If a tongue motion type signal is detected, further data processing includes feature extraction and analysis of its correlation with the stored features. If sufficient correlation has been obtained, then a corresponding output is provided. The output may, for example, be transmitted to an output peripheral for one or more control commands to execute the command, thereby providing a system control function. Neural nets can be used to perform the correlation function.
[0021]
Here, returning to the drawings, the detailed description of the present invention will be continued. FIG. 1 is a block-level diagram illustrating a system 10 for detecting tongue movement or other head movement and providing an output corresponding to the detected tongue movement. System 10 includes an analog-to-digital (A / D) converter 16 that converts analog signals to digital signal data, such as a PCI9118HG data acquisition card manufactured by Adlink Technology or a DAQi250 data acquisition card manufactured by Ines Company Gmbh, Germany. It includes a pressure sensor 12 coupled to the circuit section 14. Buffer 17 is provided so that these signals are relatively continuous and may reflect continuous movement of the tongue (eg, until filtration and signal detection may be performed as described further below). Then, a part of the signal from the A / D converter 16 is stored. Further, the processing circuit unit 14 receives digital signal data from the A / D converter 16 and the buffer 17 and executes various signal processing functions on the digital signal data to detect occurrence of tongue movement. And a processor 18 for determining the type of tongue movement. The system 10 also includes an output peripheral 20 coupled to the processor 18 for executing one or more control instructions provided by the processor 18 and corresponding to the detected views.
[0022]
The system 10 is shown in the example environment context of FIG. 2a. In FIG. 2 a, the user 22 provides the pressure sensor 12 near the ear 24 by providing the pressure sensor 12 in a pair of headphones 26. Headphones 26 preferably provide two functions. These two functions include: (1) providing the pressure sensor 12 at a relatively fixed location near the ear of the headphones 26; and (2) providing a small amount of external acoustic isolation of the headphones 26. 12 is a function of reducing the amount of external noise detected. If the user 22 moves, for example, by moving the tongue relative to the oral cavity, a change in barometric pressure occurs in or near the ear 24, and the change in barometric pressure uniquely identifies the tongue movement. The change in air pressure is detected by the pressure sensor 12 that converts the detected air pressure into an analog electronic signal so that the circuit unit 14 can continue processing. In FIG. 2a, the exemplary output peripheral device 20 is associated with a computer or video game that performs control instructions (eg, running, walking, flying, punching or kicking in a video game, etc.). Shown as a display (or exemplary output peripheral 20 may provide control functions such as for performing tasks in a mechanized wheelchair and manufacturing process). The direction and speed of movement displayed may correspond to the direction and speed of movement of the tongue, and the strength of the kick or punch may correspond, for example, to the strength of the tongue on the pallet or teeth.
[0023]
The positional relationship between the pressure sensor 12 and the ear 24 will be described in more detail with reference to FIG. Although the present invention can use one sensor 12 as shown in FIG. 2b, using two sensors as shown in FIG. 2a is advantageous in obtaining three-dimensional information. By using two sensors, each sensor collects different information regarding the strength of the oral cavity or teeth or tongue tapping as a function of the three-dimensional movement of the tongue. The detected signals may be correlated to provide an output having, in effect, three degrees of motion (eg, a three-dimensional effect). For example, intra-oral movement may be up and down, left and right, back and forth, and each combination of such movements may provide a desired three-dimensional result. This allows the sensor 12 at each ear to detect a different barometric pressure, which can be correlated with the three-dimensional movement of the joystick or a mouse that provides an output signal indicative of the three-dimensional movement.
[0024]
The pressure sensor 12 includes a housing 28 such as a headphone housing having a sensor such as a microphone 30 fixed on the sensor. The manner in which microphone 30 is attached to housing 28 may be varied to adjust the distance from ear 24 to microphone 30. The microphone 30 is near the ear 24 (eg, within about one to two inches), depending on the sensitivity of the microphone, but other distances can be used, and this is within the scope of the present invention. It is believed that there is. The microphone 30 may be provided comfortably in the ear 24 or as close as possible to the ear to receive a change in barometric pressure as the level of intensity increases. In FIG. 2b, neither the shape nor the position of the microphone 30 is shown in view of the scale, but it is only shown in this way for clarity of explanation.
[0025]
In one embodiment of the present invention, microphone 30 has a sensitivity of at least about 47 mV / Pa (millivolt per pascal), and in one embodiment, has a sensitivity of about 100 mV / Pa or more in a frequency range from about 10 Hz to about 800 Hz. Has sensitivity. A frequency range on the order of about 20 Hz is useful in one embodiment for detecting barometric pressure resulting from tongue movement as described herein. Further, it is again noted that the criteria for detecting barometric pressure may mean detecting barometric pressure, a difference in barometric pressure and / or sound. One exemplary microphone that can be used is a Model 4190 microphone manufactured by Bruel & Kjaer, Denmark. However, alternatively, other types of microphones or other types of pressure sensors can be used, and each such modification is considered to be within the scope of the present invention. With a Model 4190 microphone, the analog output signal is about 400 mV peak-to-peak. However, the amplitude of the output signal depends on the amplitude coefficient of the electronic components and the position of the microphone with respect to the ear, and may vary substantially.
[0026]
It is unclear which physical, chemical, or neural mechanisms cause or produce changes in air pressure in or near the ear in response to various muscle activations (eg, muscle contraction and release). Absent. However, to connect the oral cavity to the ear via an eustachian tube, the movement of the tongue creates a barometric pressure and airflow or ear to the ear that produces a barometric pressure or a detectable barometric pressure that can be detected by the sensor 12. Changes in airflow from the air may occur. Whether physical, chemical, or neural mechanisms are accurate or not, experimental tests show that tongue movements produce pressure changes in or near a person's ear, and changes in barometric pressure are substantially independent of each other. It has a unique signature, which ensures that it is substantially unique for each given tongue movement. As a result, changes in barometric pressure can be monitored near the ears and the change in barometric pressure can be used to detect movement of the user's tongue.
[0027]
The present invention uses the term "change in air pressure" near the ear in the broadest sense to characterize the parameter being measured. Alternatively, the change in air pressure can be characterized as a sound wave. As is well known to those skilled in the art, acoustic waves are longitudinal waves, in which a "push" is transmitted for each molecule in the medium (air in the preferred embodiment). The resilience of such waves is due to air pressure. Whenever the density of a molecule becomes higher than normal, the pressure also becomes higher than normal, pushing the molecule farther. FIG. 2c shows an exemplary acoustic wave 40 in air, consisting of alternating zones 42 of low molecular density 42a and high molecular density 42b, respectively. Variations in molecular density result in changes in air pressure having a particular frequency such that sound waves propagate. Further, as is well known to those skilled in the art, as a sound wave spreads from its source, its intensity drops. This is because the area of the wave widens greatly, while the total energy is constant. Therefore, the energy per unit area decreases with the reciprocal of the square of the distance. Consequently, it is desirable to have the microphone 30 sufficiently close to the ear 24 to increase the intensity level of the air pressure change, thereby making it easier to detect for any noise.
[0028]
The range of frequencies at which sound waves can be heard is from about 20 Hz to about 20 KHz, but the present invention does not concern whether changes in air pressure are audible. This is because the microphone 30 is sufficiently sensitive and has a sufficient frequency detection range to detect a change in air pressure at a high frequency or a low frequency. In embodiments of the present invention, the frequency range from about 10 Hz to about 800 Hz is concentrated. This is because it has been determined through empirical tests that sufficient data is available within the frequency range to detect and identify tongue movement. In another embodiment, the frequency range is about 20 Hz ± several Hz (eg, about ± 10 Hz), and thus is on the order of about 10 Hz to about 30 Hz. However, alternatively, any frequency range suitable for detecting tongue movement may be monitored, and such modifications are contemplated to be within the scope of the present invention.
[0029]
The pressure sensor 12 (preferably including the microphone 30) monitors changes in air pressure and converts the pressure data into an analog electrical signal 50, as shown in FIG. 2d. Note that in signal 50, there are at least two signal components, a high frequency component 52 and a low frequency component 54. Further, other frequencies may be present in the electrical signal 50, and the present invention preferably analyzes various signal frequencies in subsequent data processing performed by the processor 18 (described in detail below). .
[0030]
A method 100 for performing the present invention is shown in FIG. The method 100 includes detecting the tongue movement by monitoring the change in air pressure caused by the person's tongue movement in step 102. Upon detecting tongue movement in step 102, one or more control instructions or other output (s) corresponding to the detected tongue movement are provided to the output peripheral in step 104 to achieve the desired output function. .
[0031]
A method for detecting tongue movement (step 102) is illustrated in FIG. A pressure sensor as shown in connection with FIG. 1 and FIGS. 2a-2d is placed close to the user's ear that detects tongue movement in step 120. In accordance with an embodiment of the present invention, a change in air pressure near the ear occurs in response to tongue movement, and thus, in step 120, the pressure sensor is positioned near the ear. However, in an alternative embodiment of the present invention, the pressure sensor may be applied to other parts of the body because, alternatively or due to tongue movement, pressure changes may occur at or near other parts of the body. Alternatively, it may be contemplated to be located near other parts of the body, and any detection of tongue movement by analyzing changes in air pressure is contemplated as being within the scope of the present invention.
[0032]
At step 122, the air pressure near the ear is monitored by a sensor, and at step 124 it is converted to an electrical signal for later analysis. After being converted to an electrical signal in step 124, the electrical signal is analyzed in step 126 to detect tongue movement. It is contemplated that tongue movement may be detected at step 126 by simply analyzing the signal corresponding to the change in air pressure without processing further data, although the tongue movement detection process of step 126 may include a signal analysis. It is preferable to include data processing related to.
[0033]
A method for analyzing and processing the electrical signal corresponding to the monitored pressure is illustrated in FIG. The electrical signal (analog signal illustrated in FIG. 2d) is converted to a digital signal (illustrated in FIG. 6) in step 140. As is well known to those skilled in the art, an analog signal can be converted to a digital signal by sampling the analog signal at a selected frequency and identifying the signal amplitude at each of the sampling points. Each of the sampled data points is then stored as a digital word in memory and used for further analysis. FIG. 6 illustrates a sampled analog signal. In FIG. 6, the dashed line illustrates an exemplary analog signal for a particular time period, and the points on the dashed line represent sampled amplitude values stored in memory. Desirably, the sampling frequency is sufficient to capture enough data points to properly represent the analog signal. As a specific example, the sampling rate may be 32 KHz, and the total signal time length analyzed may be 2048 mSec. However, alternatively, other sampling rates and data acquisition time frames may be used, and such modifications are contemplated within the scope of the present invention.
[0034]
Once the analog signal is converted to digital signal data in step 140, the digital data is analyzed and processed by a signal processor, for example, in step 142, to detect the presence of tongue movement. Analysis and processing of the data may be performed in multiple segments, for example, as illustrated in FIGS. As shown in FIG. 7, the first data segment is analyzed at step 144, and then the second data segment is analyzed at step 146. Once the various data segments have been analyzed separately, step 148 analyzes the data segments together. In step 150, if all data segments have not been analyzed yet, the method 142 returns to step 146 to analyze the next data segment, and then all previous segments are analyzed together in step 148. Is done. The process continues until all data segments have been analyzed in step 150, so that in step 152, results can be generated using the analyzed data segments.
[0035]
The data segment analysis can be shown graphically in FIG. Here, the digital signal data 154 is shown continuously for the purpose of simplification. The total length of the data for analysis can be separated into 64 segments, each of which is 32 mSec in length. Note that signal 154 includes both high frequency components 155 and low frequency components 156. The data segments may be found in either component potentially, or the tongue movement data may span multiple data segments, so that the data segments are individually and together Can be parsed. Therefore, in step 144 of FIG. 7, the first data segment is analyzed (area 157), then in step 146 the second data segment is analyzed (area 158), and in step 148 both data segments are analyzed. Analyze together (region 159) and then continue the process for all data segments. As a result, the data analysis of the present invention analyzes both high and low frequency signals to detect tongue movement. This is because empirical tests generally indicate that the signal of interest is in the range of about 10 Hz to about 800 Hz, or in the range of about 20 Hz ± about 10 Hz.
[0036]
Returning to FIG. 5, if in step 142 data found to be tongue movement data is found in the pressure data, then in step 160, a subsequent analysis is performed to determine the type of tongue movement. Such analysis includes feature extraction and analysis using a neural network that is accustomed to recognizing each signal / feature of such a signal. If the neural network determines that the signal represents a particular characteristic tongue movement, it provides an output or generates its representative output for use as described herein. However, or alternatively, upon detecting tongue movement, other techniques for identifying the type of tongue movement may be used, and any such technique is contemplated to be within the scope of the present invention.
[0037]
One exemplary method of analyzing a digital signal into data segments is illustrated in FIG. For each of the 32 mSec data segments, at step 170, the data is transformed from the time domain to the frequency domain using, for example, a fast Fourier transform (FFT) as is well known to those skilled in the art. As is well known, the time domain signal f (t) is related to the frequency domain f (jω) according to the following equation:
[0038]
(Equation 1)

Here, F (f (t)) is a conventional Fourier transform. As is well known to those skilled in the art, the fast Fourier transform is related to the conventional Fourier transform because it is an efficient algorithm for computing the discrete Fourier transform. After the digital signal data has been converted to the frequency domain via a fast Fourier transform, the frequency domain data is processed at step 172 to distinguish data related to tongue movement from noise data. As is well known to those skilled in the art, isolation from data noise is often simplified in the frequency domain. This is because, unlike noise, data signals have certain physical characteristics. This is because the data signal in the time domain has a smaller amplitude than the noise, whereas the data signal in the frequency domain has a larger amplitude than the noise. Therefore, the fast Fourier transform is a common method for separating noise.
[0039]
Peripheral details of data processing of digital signal data may be made through various data processing techniques well known to those skilled in the art, and any data processing methodology is contemplated to be within the scope of the present invention. Although many different data processing methodologies can be used, the preferred methodologies are set forth below in connection with the following methods and systems.
[0040]
Referring to FIG. 10, a system 200 and method of the present invention is shown. One or more sensors 12 sense air pressure from the user's ear (s). The analog signal (s) from the sensor (s) is provided to processor 18 via a data acquisition card, analog-to-digital converter and buffers 16/17. During movement of the tongue, the signal received from the sensor is substantially continuous, at least for each time period. Thus, buffer 17 can store a sufficient portion of the signal for analysis by processor 18.
[0041]
The processor 18 includes a band pass filter 202, a power spectrum filter 204, a signal detector 206, a feature extraction unit 208, and a neural network 210. System 200 also includes an output 212 that can be coupled to a desired output peripheral 20.
[0042]
To summarize the operation of the overall system 200, the sensors (12) and data acquisition, A / D and buffers 16/17 provide signals to the bandpass filter 202. An example of raw data or a raw signal supplied to the bandpass filter 202 is shown by 220 in the graph of FIG. The raw signal is provided to a bandpass filter, which manipulates the frequency domain using a fast Fourier transform technique. Fast Fourier transform converts a digital signal into the frequency domain. In a bandpass filter, frequencies above or below the frequency range representing tongue movement are removed from the signal, for example. Frequencies above and below the frequency range of interest are generally sources of noise. An example of the signal after being filtered by the band-pass filtering is shown as 222 in the graph of FIG. Various bandpass filters may be used to perform bandpass filtering to remove signal frequencies above the desired frequency range and signal frequencies below the desired frequency range.
[0043]
Portions of the signal represented by 222 (eg, portions 222a and 222b) may not exhibit tongue movement. For example, they are relatively low power and may therefore still be noise. Thus, the power spectral filter 204 is further filtered according to the power of the various parts of the signal to determine those parts that are due to tongue movement and therefore should be considered for further analysis. Power filtering converts the signal of FIG. 12 to 224a, 224b,. . . This may be done by slicing into multiple segments or windows as represented by 224i. It is recognized that the graphs of FIGS. 12 and 13 are at different scales. Thus, windows 224a-224i are at time t_aAnd time t_bSlice the time frame between and. Here, time t_aAnd time t_bThe same time frame between is shown for signal portion 222c of FIG.
[0044]
Power filtering suppresses or eliminates frequencies where the power of the signal does not exceed a predetermined threshold or value, thereby possibly representing noise. By removing the noise component, a signal representative of tongue movement can be found. To perform power filtering, for example, the power spectrum of the signal is calculated using the following formula.
[0045]
P_f= Mag_f ²+ Phase_f ²
Where f = F₀. . . F_l, And [F₀F_l] Represents the frequency range in which the signal resulting from the tongue movement is expected.
[0046]
In other words, power is a function of the sum of magnitude squared and phase squared. Each window 224a, 224b. . . The power calculated above for 224i is reconstructed to create the graph 226 shown in FIG. Graph 226 shows values 226a, 226b. . . 226i. Curve 226 is relatively smooth and represents the power spectrum in which the signal resulting from tongue movement may be found in graph 222 of FIG. The signal produced by tongue movement is typically of relatively high power and occurs at about the middle of the graph 226, for example, for the time frame T shown in FIG. If the signal 222 represents tongue movement in time, it is the time at which this time frame occurs and the duration of that time frame that detect or locate the signal. Thus, in practice, the signal detector component 206 of the system 200 combines the graphs of FIGS. 12 and 13 with the analysis described herein to provide a signal detection function. The time frame T is also shown in the graph of FIG. 12, thereby specifying a portion 222d (in the illustrated block window 222e) of the signal 222c to be considered generated by the tongue movement.
[0047]
The signal portion 222d, or at least one feature of the signal, may then be provided to the neural network 210. Using the feature extraction principle, one feature of the signal portion 222d is provided to the neural network. In the neural network 210, its features are compared to other features to which the neural network has been accustomed. Nervous network familiarization is a well-known procedure. The neural network may determine whether the input features (e.g., representative of signal portion 222d) acceptably match, or acceptably correlate to, neural network familiarity, and if so, The neural network produces the output 212. Output 212 may be an electrical signal similar to the type of signal generated by a computer mouse, joystick or other device. Thus, tongue movement by moving the user's tongue from the front, bottom, and left to the back, top, and right of the mouth produces an output signal similar to moving the mouse from the bottom left to the top right of the mouse pad. obtain. As another example, movement of the tongue from the left front of the mouth to the right front of the mouse may result in a signal similar to moving the mouse from left to right. Others are the same.
[0048]
For example, features extracted from signal 222 may be features that exhibit certain characteristics or features that distinguish curves representing signals from others. Various known features are used in the feature extraction method. In the present invention, an exemplary feature is a smooth power spectrum of the signal portion 222d. Other features may be the log of the power spectrum, the power spectrum itself, etc.
[0049]
Neural network 210 may be a conventional neural network that has been accustomed to in a general form. The neural network 210 may compare the characteristics of the signal portion 222d with its familiarized ones and determine whether the characteristic is closer to one familiarized characteristic or another familiarized characteristic. If the input features of the signal portion 222d are close enough to a familiarized feature, the neural network causes an appropriate signal to be generated at its output, for example, as described above. If the input features are not sufficiently similar to any of the familiarized features, or are relatively similar to two or more familiarized features, the neural network will generate an output signal. Produces no output. Otherwise, uncertain and undesirable output signals are generated.
[0050]
As described above, using two sensors 12, system 200 may operate sequentially on each signal from each sensor, or may include a parallel processing function, whereby both sensors Can be processed substantially simultaneously. Output 212 may reflect the results of the signal analysis of the signals originating from both sensors, and may be used at output 212 to provide an output signal that actually reflects the three-dimensional data.
[0051]
While the invention has been shown and described with respect to certain preferred embodiments or specific preferred embodiments, those skilled in the art, upon reading and understanding this specification and the accompanying drawings, will recognize equivalent alternatives and modifications. It is clear to recall. In particular, with respect to various functions performed by the above components (assemblies, devices, circuits, etc.), the terms used to describe such components (including those referred to as "means") are: Any component that performs a particular function of the described component, even though it is not structurally equivalent to the disclosed structure that performs the function herein, illustrating an exemplary embodiment of the invention. (Ie, those that are functionally equivalent) are intended. Further, while particular features of the invention may be illustrated in terms of only one of the several embodiments, such features may be desirable for any given application or particular application. If so, and may have advantages, they may be combined with one or more other features of other embodiments.
[Brief description of the drawings]
FIG.
FIG. 1 illustrates a system in accordance with the present invention that detects mouth movement, often referred to herein as tongue movement, and uses the movement of the mouth to provide a control function corresponding to the sensed mouth movement. It is a block level diagram shown.
FIG. 2a
FIG. 2a is an environment diagram of a user having a pressure sensor located near an ear in electrical communication with a processor coupled to a display, in accordance with the present invention.
FIG. 2b
FIG. 2b is an enlarged view of a portion of FIG. 2a showing in more detail a pressure sensor for monitoring air pressure near the ear, according to one aspect of the present invention.
FIG. 2c
FIG. 2c is an image of a wave in air, showing multiple vertical molecular density changes, according to the present invention.
FIG. 2d
FIG. 2d is a graph illustrating an exemplary electrical signal generated by a transducer coupled to a pressure sensor that indicates a change in air pressure at a location local to the pressure sensor according to the present invention.
FIG. 3
FIG. 3 is a flowchart illustrating a method for detecting mouth movement and providing a control instruction in response to the detected mouth movement according to the present invention.
FIG. 4
FIG. 4 is a flowchart illustrating a method for monitoring air pressure according to the present invention.
FIG. 5
FIG. 5 is a flowchart illustrating a method of processing an electrical signal corresponding to air pressure to detect a thought.
FIG. 6
FIG. 6 is a graph illustrating the conversion of an analog electrical signal to digital signal data according to the present invention.
FIG. 7
FIG. 7 is a flow chart showing a method for analyzing digital signal data according to the present invention.
FIG. 8
FIG. 8 is a graph illustrating a method for analyzing digital signal data in accordance with the flowchart of FIG. 7 according to the present invention.
FIG. 9
FIG. 9 is a flowchart illustrating a method for processing digital signal data according to the present invention.
FIG. 10
FIG. 10 is a system diagram showing a system including parts and functions of the present invention.
FIG. 11
FIG. 11 is a graph illustrating exemplary raw signal information sensed by a sensor used in the present invention to sense air pressure indicative of mouth movement.
FIG.
FIG. 12 is a graph of the frequency-filtered signal information of FIG. 12, for example, a graph in which noise is removed by filtering the frequency.
FIG. 13
FIG. 13 is a graph showing a window of the power spectrum of the curve of FIG. 12 used to locate the signal representing head movement in time.
FIG. 14
FIG. 14 is a graph of such a power spectrum.

Claims (24)

A method for generating a control signal, comprising:
Sensing a pneumatic pattern generated in a human ear, wherein the pneumatic pattern is generated by the human's spontaneous physical movement;
Converting the pneumatic pattern into an electrical output signal;
Generating a control command from the electrical output signal, wherein the control command corresponds to the spontaneous physical movement;
Controlling the peripheral output device using the control instruction;
A method comprising: The method of claim 1, further comprising placing a microphone proximate to the ear, the microphone operative to convert the pneumatic pattern into the electrical output signal. The method of claim 1 or 2, wherein the control instructions are used to control a computer, machine or other type of equipment. Locating an air pressure sensor near the human ear to sense an air pressure pattern associated with each of the plurality and uniquely defined physical movements performed by the human;
Recording an output signal for each uniquely defined physical movement resulting from the sensing step;
Generating a series of control instructions, each control instruction corresponding to one of the recorded output signals;
The method of claim 1, further comprising: Dividing the electrical output signal into a plurality of data segments;
Analyzing the first data segment to detect a specific physical movement having a known electrical signal pattern;
Analyzing a second data segment to detect the particular physical movement;
Analyzing at least two consecutive data segments to detect the particular physical movement;
The method of claim 1, further comprising: (A) dividing the electrical output signal into a plurality of data segments;
(B) analyzing the first data segment to detect physical movement;
(C) analyzing the next data segment to detect physical movement;
(D) analyzing all of the analyzed data segments collectively to detect physical movement;
(E) repeating steps (c) and (d) until all of the data segments have been analyzed;
The method of claim 1, further comprising processing the electrical output signal by performing the following. 7. The method of claim 1, further comprising monitoring an air pressure pattern near each ear of the human to derive the expression of the spontaneous physical movement from an electrical output signal corresponding to the air pressure pattern sensed at each ear. The method according to any of the above. The method of claim 1, further comprising analyzing the electrical output signal to find a match with one of a series of predetermined output signal patterns. 9. The method of claim 1, wherein the spontaneous physical movement includes at least one of the following: movement of the human head, movement of the human oral cavity, movement of the human tongue, and respiration of the human. Crab method. The method according to any of the preceding claims, wherein the spontaneous physical movement is a movement without vocalization of the human tongue. The method of claim 1, further comprising determining, using a neural network, whether the electrical output signal correlates with one of a series of training patterns. An air pressure sensor that detects an air pressure pattern near the human ear when the human is performing spontaneous physical movement, and generates an electric output signal corresponding to the detected air pressure pattern,
A processing circuit electrically coupled to the pneumatic sensor for identifying a pattern from the electrical output signal and generating a control command corresponding to the spontaneous physical movement, the control command controlling a peripheral output device A processing circuit adapted to
A control system comprising: The control system of claim 12, wherein the processing circuit identifies the control command from a plurality of specific control commands from the electrical output signal. 14. The control system according to claim 12, further comprising a peripheral output device controlled by the control command. The control system according to claim 14, wherein the peripheral output device is selected from a computer, a display, a machine, a robot, a game device, and a wheelchair. The control system of claim 12, wherein the processing circuit analyzes the output signal to find one that matches one of a series of predetermined output signal patterns. The control system according to any one of claims 12 to 16, wherein said air pressure sensor is a microphone. 13. The control system of claim 12, wherein the processing circuit comprises a neural network for determining whether there is a correlation between the pneumatic pattern and one of a series of training patterns. An air pressure pattern associated with each of the plurality of physical movements performed by the human is detected, and the resulting electrical output signals for each of the physical movements are recorded by the processing circuit, the processing circuit comprising a series of control signals. The control system of claim 12, wherein the control signals each correspond to at least two of the plurality of output signals. The processing circuit includes:
Dividing the electrical signal into a plurality of data segments,
Analyzing the first data segment to detect a specific physical movement having a known electrical signal pattern;
Analyzing the second data segment to detect the particular physical movement;
Analyzing at least two consecutive data segments to detect the particular physical movement;
The control system according to claim 12. The processing circuit includes:
(A) dividing the electrical output signal into a plurality of data segments;
(B) analyzing the first data segment to detect physical movement;
(C) analyzing the next data segment to detect physical movement,
(D) collectively analyzing the analyzed data segments to detect physical movement,
(E) repeating steps (c) and (d) until all the data segments have been analyzed;
The control system according to claim 12. A second air pressure sensor monitors an air pressure pattern near the second ear of the human, and the processing circuit derives a three-dimensional representation of the physical movement from output signals from both of the air pressure sensors. The control system according to any one of claims 12 to 21, which performs the control. 23. The method of any of claims 12 to 22, wherein the physical movement comprises at least one of movement of the human head, movement of the human oral cavity, movement of the human tongue, and respiratory activity of the human. Control system. 23. The control system according to claim 12, wherein the spontaneous physical movement is a movement without vocalization of the human tongue.

Images