JPH0485680A - Head skin electric potential distribution pattern recognizing type syllable input device - Google Patents

Abstract

PURPOSE:To use the title device even by a user unable to execute the voluntary movement of actual vocalization and so on by providing a presenting part to present the king of a syllable that the vocalization is to be intended by the user and the timing of the reading by heart of it in learning the pattern of a recognizing part and constituting the recognizing part with a neuronetwork. CONSTITUTION:The title device possesses a skin surface electrode 1, a detecting part 2 to detect head skin electric potential based on electric potential detected by the electrode 1, a storage part 3 to store the multichannel time series distribution pattern of the head skin electric potential and a recognizing pat 4 to identify and to output a corresponding vocalized syllable to the stored multichannel time series distribution pattern. Then, a presenting part 7 to present the kind of the syllable that vocalization is intended by a user and the timing of the reading by heart of it in learning the pattern of the recognizing part 4 is provided and the recognizing part 4 is constituted with a neuronetwork. Thus, the input device, for which the actual vocalization is unnecessary, applicable even to a person unable to execute any voluntary movement can be realized.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention is applicable to people with disabilities who are unable to make any voluntary movements such as vocalization, eye movement, or keyboard operation due to diseases such as lateral sclerosis, and those with excessive Scalp potential distribution pattern used by users who are unable to make voluntary movements of their arms or fingers due to exposure to gravitational acceleration! ! ! ! Regarding the cognition syllable input device (hereinafter also simply referred to as "input device"), in particular,
The present invention relates to an input device that recognizes a syllable intended to be uttered from a potential distribution pattern detected from a user's scalp as described above.

[Conventional technology]

Traditionally, people with disabilities who are unable to make sufficient voluntary movements as described above,
As an input device for users who are unable to make voluntary movements of their arms or fingers due to exposure to excessive gravitational acceleration, for example, it can be used as an input device that recognizes and analyzes readiness potential distribution patterns using electroencephalogram topography. What is based on this is being researched.

Regarding the method of recognizing and analyzing the preparedness potential distribution pattern applying the above-mentioned electroencephalogram topography, please refer to Tsukahara's ``Brain Information Processing J (1964), p. 180, and Kaneko's 1 Recent Brain Science J (1988, The description on page 79 of the Japanese Patent Application No. 1999-1-1 (Japanese Patent Application No. No. 52872 "Syllable recognition device using electroencephalogram topography".

The device was also proposed by the present inventors in Japanese Patent Application No. 2-37456 [Operation device using scalp P$ potential pattern],
There is a device.

[Goma to be Solved by the Invention] All of the devices described in the above-mentioned prior art section detect changes in electric potential distribution from the scalp and ( The preparedness potential pattern detected in step 7 is used to recognize and identify the vocal syllables and hand movements intended by the user using a neural network (neural network) with excellent pattern learning and recognition abilities. It is something. In these devices, the neural network constituting the recognition unit learns the relationship between the scalp readiness potential pattern and the intended voluntary movement; In fact, it was a time-series pattern obtained by identifying the starting point of the voluntary movement, and using the observation window set from that starting point as a reference point.

Therefore, 1., the neural network of the recognition part mentioned above) is Pi
1. It is essential for the user to have the physical strength to perform some voluntary movements, and for people who are unable to perform any voluntary movements due to illness etc., this device can be used without manual effort. The present invention has been made in view of the above circumstances, and the purpose of the present invention is to solve the problems such as the door description in the conventional technology, It is an object of the present invention to provide an input device that does not require voluntary movements such as actual vocalization and is applicable even to people who cannot perform any voluntary movements due to illness or the like.

[Means for solving the problem] The above-mentioned object of the present invention is to provide a detection unit that detects a scalp potential based on the potential detected by the electrodes, such as a plurality of skin surface 1fi electrodes; A storage unit that stores channel time-series distribution patterns and performs necessary preprocessing; and a recognition unit that identifies and outputs uttered syllables corresponding to the multi-channel time-series distribution patterns stored and preprocessed by the storage unit. In the input device, the recognition unit is provided with a presentation unit that presents the type of syllable that the user intends to utter and the timing at which it must be read when the recognition unit learns the pattern, and the recognition unit is formed of m units. This is achieved by an input device characterized by being constructed of a neural network consisting of weighted links connecting them.

[Function] In the input device according to the present invention, the input device recognizes the syllable that the user intends to pronounce from the potential distribution pattern detected from the user's scalp, and the input device allows the user to know the type of syllable that the user intends to pronounce and the type of syllable that the user intends to pronounce. By presenting the timing of the must-read on the presentation unit and making the user intend to utter, etc., the need for voluntary movements such as actual utterance is eliminated even when learning recognition and part bata.2. This allows even users who are unable to exercise to use it.

In addition, in this specification, "must read" refers to reading the letters syllable by syllable without actually saying them. , to say that you are conscious of uttering each syllable but do not actually utter it. In addition, in the present invention, "syllable"
The word a includes everything from so-called monosyllable units to syllables in which a plurality of monosyllables are uttered consecutively (combined), for example, the alphabet or
4. [Embodiments] Hereinafter, embodiments of the present invention will be described in detail in 1.7 based on the drawings.

FIG. 1 is a diagram showing the overall configuration of an input device that is an embodiment of the present invention. In the figure, 1 is a large number of skin surface electrodes, 2 is a detection unit that detects the electric potential detected from the scalp of a person based on the potential detected from the skin surface electrode 1, and 3 is the scalp of the above-mentioned person. A storage unit 4 stores the time-series pattern of potentials detected from the storage unit 3, receives as input the multi-channel time-series potential pattern stored in the storage unit 3, recognizes the multi-channel potential pattern, and generates the corresponding syllable. The output recognition unit 5 indicates a presentation unit that presents the syllables recognized by the recognition unit 4. Further, 6 is a presentation signal generation unit that generates the type of syllable that the user intends to utter and generates a syllable presentation signal when the recognition unit 4 learns the pattern; 7 is a presentation signal generation unit that generates a syllable presentation signal; A presentation section 8 is a control section that controls the operation of each section by timing the above-mentioned sections.

The meanings of each signal aj in FIG. 1 are as follows. a is a scalp potential time series signal detected by the skin surface electrode 1, b is a scalp potential time series signal processed by the detection unit 2, and C is a digitized scalp potential time series processed by the storage unit 3. signal, d is a syllable signal identified by the recognition unit 4, e+Lg and b are control signals that activate and control the operation of the control unit 8 and each part, 1 is the syllable signal that the user intends to utter when the recognition unit 4 learns the pattern. A signal for presenting the type of syllable to be used as teacher data to the recognition section 4, J indicates a signal generated by the presentation signal generation section 6 to be presented to the presentation section 7.

FIG. 2 is a configuration diagram showing details of the above-mentioned recognition unit 4,
Reference numeral 4-1 indicates a neural network, and reference numeral 4-2 indicates a weight update control unit. Here, the neural network 4-1 is connected in layers, and 3
The case is shown in which the weight update control unit 4-2 is configured with layers.
is activated by the control unit 8 when the recognition unit 4 is in the learning mode, and the error detection unit 4-28 detects the difference between the signal from the presentation signal generation unit 6 and the output of the neural network 4-1, and based on this, Then, signal processing for updating the load is performed by the load update value calculation 4-2b.

The operation of this embodiment configured as described above will be explained below. In addition, in the following explanation, learning mode A of the neural network 4-1 of the recognition unit 4 and a change in the scalp potential distribution when the user intends to utter a certain syllable and visualizes the utterance will be explained. Regarding recognition mode B, which recognizes and identifies the emerging readiness potential pattern and recognizes the syllable intended to be uttered,
I will explain them one by one.

First, learning mode A in which the recognition unit 4 learns the relationship between the scalp potential distribution pattern and the syllable intended to be uttered will be described.

Learning mode A is a mode in which the neural network 4-1 of the recognition unit 4 learns the relationship between the potential distribution pattern detected from the user's scalp and the uttered syllable intended by the user. By presenting the type of syllable and the timing at which it must be read on the display device, and instructing the user to intend to pronounce it accordingly, learning pattern data can be identified without actually making voluntary movements. This eliminates the need for actual voluntary movements, making it possible for users who are unable to perform any voluntary movements to use it.

A timing chart of this operation is shown in FIG. 4, and a flowchart is shown in FIG.

The learning mode A is roughly divided into a period in which the user is trained to read the required reading (steps 72 to 79), and a period in which the user is not conscious of the required reading (steps 80 to 83).

In learning mode A, first, the user is instructed to read the presented syllables intentionally, and the user selects the presentation section 7.
The user must read the syllables that are presented, the scalp potential distribution including the readiness potential generated on the user's scalp due to the must-read is input to the recognition unit 4, and the type of the presented syllable is output, and the neural network of the recognition unit 4 is The circuit network 4-1 learns the relationship between input and output.

Now, when the presentation of syllables to the presentation section 7 targets the alphabet as shown in FIG. At the time of flashing, only the characters of the syllables that encourage utterance are flashed with the background color inverted. This flashing of characters is performed alphabetically and at equal intervals in time. It is assumed that the user must read the syllables displayed on the display device with the intention of uttering them, while checking the timing of the equal intervals.

Once the user gets used to the timing of the flash on the presentation unit 7, the user will be aware of preparing to speak the next must-read character immediately before the flash stimulation, and will be able to read the must-read character at the timing of the flash. At this time, the scalp potential obtained from the skin surface electrode attached to the user's scalp is detected as a signal by the detection unit 2, and is stored in the storage unit 3 immediately before the flash of the presentation unit 7, from 1 second before the flash. It is processed as a scalp potential time-series signal of the previous time, converted into a numerical matrix as shown in FIG. Given as a cover turn.

At this time, the output layer 4-1 of the neural network 4-1 of the recognition unit 4
0 is given the type of syllable presented as a teacher signal. FIG. 6 shows a case where the input layer 4-1a is given a numerical matrix of scalp potential time-series signals obtained just before the must-read syllable "b", and as the teacher signal, as shown in the figure, “1” is applied only to unit “b” of output layer 4-10.
”, and “0” is given to other units.

After repeating the above-mentioned required reading multiple times for all syllables, the learning of the recognition unit 4 through the required reading is completed. after that,
In this state, the recognition unit 4 is given a scalp potential time series signal that the user is not aware of as a must-read, and the output layer 4-
1c is taught no input syllable, and this is repeated multiple times. When learning of the scalp potential pattern without input syllables is completed, learning mode A is ended.

Here, various methods that have already been proposed can be used for the learning method of the neural network 4-1 of the recognition unit 4. For example, in the case of the error backpropagation method, in the learning of the recognition unit 4,
In learning one pattern, the neural network repeatedly learns the pattern relationship given to the input and output until there is no error or until an error tolerance is reached.

Next, recognition mode B will be explained based on the flowchart shown in FIG. When the learning mode of the recognition unit 4 is finished, the acquisition of the user's scalp potential is started (step 91). Thereafter, the user must read the syllables that he/she intends to input one after another, and visualizes how to pronounce them. Here, as shown in Fig. 8, an observation window of the same length (here, 1 second) as opened in learning mode A is opened for the time-series scalp potential signal, and the signal within the observation window is observed in learning mode A. Similarly to the case of , the neural network 4-1 of the recognition unit 4 is converted into a numerical matrix
input layer 4-18 (steps 92-93).

Each unit of the output layer 4-1c corresponding to each syllable outputs a numerical matrix as a given input cover pattern, but among the units of this output layer, -
The unit with the highest written output value is set as the recognized syllable (step 94). For example, among the output values of the output layer unit,
If the output value of the unit e'' is the largest value, it is assumed that the syllable that the user intends to utter is e''. Assuming that the sampling rate is Δt, steps 95 to 96) in which the time windows are sequentially shifted as shown in FIG. 8 for the continuous scalp potential time series signals, calculate the scalp potential time series signals within each time window. The recognition unit 4 then performs recognition. Through the above-described processing, a string of characters intended to be uttered by the user is formed from the output syllable string.

According to the above embodiment, a neural network having a learning ability recognizes and identifies patterns of change in readiness potential that are mixed in the background brain waves and appear in conjunction with must-read or intended vocalization or actual vocalization, In addition, when learning the neural network, the user responds to the signals presented to the user,
Since the required reading timings are synchronized, you can input the syllable string you intend even if you are unable to make any voluntary movements.

In addition, in the above description, the alphabet was explained as an example of target syllables, but as mentioned above, it is also possible to target Japanese syllables or syllables of other languages. In addition, in the timing chart shown in Figure 4, the interval of flash stimulation of the presentation section is set to 1.6~
Although we have shown an example of 1.9 seconds, changing the interval of flash stimulation and changing the duration of flash will reduce the influence of the evoked potential P300 observed in the user's scalp due to flash stimulation within the observation window. Each of these is possible as long as it does not fall within the range and within the range that the user can recognize.

Furthermore, if the neural network of the recognition unit has completed sufficient learning, in other words, it learns the relationship between the scalp potential time-series patterns uttered by a sufficient number of unspecified users and the syllables uttered by the users. If the weight values of the neural network of the recognized recognition unit can be used, the minority category, e.g.
Regarding the discrimination of several syllables, since the neural network can absorb the aforementioned individual differences in the scalp potential time series pattern, the presentation signal generation section 6 and the presentation section 1 can be connected to each other from the input device shown in the above embodiment. As a configuration in which the above is removed, it is also possible to provide this device as a dedicated machine for recognition mode B even to unspecified users who have not gone through learning mode A.

That is, a plurality of skin surface electrodes, a detection unit that detects scalp potential based on the potential detected by the electrodes, and a storage unit that stores the multichannel time-series distribution pattern of the scalp potential and performs necessary preprocessing. and the storage unit stores and
A scalp potential distribution pattern recognition type syllable input device comprising a recognition unit that identifies and outputs a corresponding uttered syllable with respect to a preprocessed multi-channel time-series distribution pattern. It is also possible to realize a scalp potential distribution pattern recognition type syllable input device comprising a neural network consisting of weighted links, and characterized in that the recognition section has completed sufficient learning.

〔Effect of the invention〕

As described in detail above, according to the present invention, there are provided a plurality of skin surface electrodes, a detection unit that detects a scalp potential based on the potential detected by the electrodes, and a multichannel time-series distribution pattern of the scalp potential. and a recognition section that identifies and outputs a corresponding uttered syllable with respect to the multi-channel time-series distribution pattern stored and preprocessed by the storage section. In the distribution pattern recognition type syllable input device, a presentation section is provided that presents the type of syllable that the user intends to utter and the timing at which it must be read when the recognition section learns the pattern, and the recognition R section is configured to include a plurality of units. Since it is constructed with a neural network consisting of weighted links connecting them, it can be applied to people who are unable to perform any voluntary movements due to illness, etc., and does not require voluntary movements such as actual vocalization. This has the remarkable effect that it is possible to realize an input device that does not require any input.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the overall configuration of an input device that is an embodiment of the present invention, FIG. 2 is a configuration diagram showing details of the recognition section 4, and FIG. 3 is a diagram showing an example of the presentation screen of the presentation section. FIG. 4 is a timing chart in learning mode A, FIG. 5 is an explanatory diagram of the process of converting a time-series scalp potential signal into a numerical matrix based on the timing chart of FIG. 4, and FIG. 6 is a recognition unit , and the corresponding syllable is taught to the output layer as a teacher signal. Figure 7 is an operation flowchart when entering the learning mode. Figure 8 is a diagram explaining the situation in which the recognition unit outputs the time-series scalp potential after the learning mode is completed. FIG. 9, which is a diagram showing the situation in which signals are recognized, is an operation flowchart of recognition mode B. 1. Skin surface electrode, 2: Detection section, : Recognition section, 4-1. Neural network, 4 4-1b: Hidden layer, 4-1c: Output layer, New system section, 4-
2a: error detection, 4 value calculation, 5 recognition content presentation section, 6 section, 7: presentation section, 8: control section. 38 storage unit, 4 18 input layer, 4-2: Weight update 2b: Load update: Presentation signal generation

Claims (1)

[Claims] (1) A plurality of skin surface electrodes, a detection unit that detects scalp potential based on the potential detected by the electrodes, and a memory that stores the multichannel time-series distribution pattern of the scalp potential and performs necessary preprocessing. and the multi-channel time-series distribution pattern stored and preprocessed by the storage unit,
In a scalp potential distribution pattern recognition type syllable input device having a recognition unit that identifies and outputs a corresponding uttered syllable, when the recognition unit learns the pattern, the type of syllable that the user intends to utter and the timing at which it must be read are presented. 1. A scalp potential distribution pattern recognition type syllable input device, characterized in that a presentation section is provided, and the recognition section is constituted by a neural network consisting of a plurality of units and weighted links connecting them.