KR20180123365A - Apparatus and method for context recognizable brain-machine interface - Google Patents

Abstract

The present invention includes a memory in which a program for recognizing the intention of a measurement target based on an EEG signal is stored, and a processor for executing the program stored in the memory. Then, the processor recognizes the EEG signal of the measurement target with attention to the position of a flickering visual stimulus corresponding to the shape of a syllable associated in the arrangement of the plurality of flickering visual stimuli that flicker at individual unique frequencies, extracts a plurality of characteristics from the recognized EEG signal, and detects the intention of the measurement object based on the plurality of characteristics extracted and a generated intention recognition model, as the program is executed. In this case, the intention is one of words, phrases, clauses, and sentences including one or more syllables. The intention recognition model for arranging the characteristics of the individual syllables in a time sequential manner is generated based on sequential EEG signals collected to correspond to a machine learning algorithm and a plurality of predefined intentions. It is possible to recognize the intention of the measurement object based on the interface between a brain and a machine.

Description

[0001] APPARATUS AND METHOD FOR CONTEXT RECOGNIZABLE BRAIN-MACHINE INTERFACE [0002]

The present invention relates to context-aware brain-mechanical interface devices and methods.

Techniques have been developed to process brain-to-machine interface by analyzing parameters according to cognitive properties of EEG signals. For example, such techniques include a brain-machine interface (BMI) or a brain-computer interface (BCI). A brain-machine interface or a brain-computer interface can control a machine or a computer according to a person's intention using an EEG signal.

Generally, when the EEG signal is used as an interface control signal with the machine, the EEG signal related to the proposed stimulus is repeatedly measured to calculate the average EEG potential of the EEG segments. In this way, the cumulative electroencephalogram (EEG) associated with the presented stimulus or event is called the event-related potential (ERP). The component obtained by the time-axis analysis of EEG is 'Steady State Visual Evoked Potential (SSVEP)'. The SSVEP component is an EEG signal that responds to repetitive visual stimuli. For example, if a person is watching a flickering stimulus, an EEG having the same frequency as the flicker frequency of the stimulus is physically induced.

In this regard, Korean Patent Registration No. 10-1585150 (entitled " Multimode Brain-Computer Interface System Based on Brain Connectivity) " is disclosed.

An embodiment of the present invention is to provide a context-aware brain-machine interface apparatus and method capable of recognizing an intent of a measurement object based on an interface between a brain and a machine.

It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a context-aware brain-mechanical interface device including a memory for storing a program for recognizing an intention of a measurement object based on an EEG signal of a measurement object, And a processor for executing the program stored in the memory. At this time, the processor recognizes the EEG signal of the measurement subject by executing the program, extracts a plurality of predetermined characteristics from the recognized EEG signal, and outputs the extracted plurality of characteristics and the generated intention recognition model The intention of the measurement object is detected. Also, the intention may be one of words, phrases, clauses, and sentences including one or more syllables, and the intention recognition model may be generated based on a brain wave signal collected to correspond to a predetermined machine learning algorithm and a plurality of labeled intents .

According to a second aspect of the present invention, there is provided a method of recognizing a context in a brain-mechanical interface device, comprising: recognizing an EEG signal to be measured; Extracting a plurality of predetermined characteristics from the recognized EEG signal; And detecting the intention of the measurement object based on the extracted plurality of characteristics and the generated intention recognition model. Here, the intention may be one of words, phrases, clauses, and sentences including one or more syllables, and the intention recognition model may be generated based on an EEG signal collected so as to correspond to a predetermined machine learning algorithm and a plurality of labeled intents .

The present invention is based on a machine learning algorithm such as a deep learning, and based on a context including words, phrases, phrases and sentences including one or more syllables reminiscent of a subject to be measured from an EEG signal to be measured, The intention can be detected and judged.

1 is a block diagram of a brain-mechanical interface system in accordance with an embodiment of the present invention.
2 is an exemplary diagram of visual stimulation according to an embodiment of the present invention.
3 is an exemplary diagram of a topology corresponding to an intention recognition model according to an embodiment of the present invention.
4 is a flowchart of a context recognition method in a brain-mechanical interface apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as "including " an element, it does not exclude other elements unless specifically stated otherwise.

Hereinafter, a context-aware brain-machine interface apparatus and method according to an embodiment of the present invention will be described with reference to the drawings.

1 is a block diagram of a brain-mechanical interface system 100 in accordance with an embodiment of the present invention.

The brain-machine interface system 100 can determine the intent of the measurement object through the context-aware brain-mechanical interface device 110 based on the brain wave signal of the measurement object collected through the brain wave measuring device 150 have. At this time, the brain-machine interface device 110 may include a brain-wave measuring device 150 for measuring an EEG signal to be measured, or may be connected to an external EEG device 150.

The EEG detector 150 can measure an EEG signal from a plurality of regions on the head of the subject to be measured (e.g., frontal lobe position). The brain wave measuring device 150 transmits the measured brain wave signal to the brain-mechanical interface device 110.

The brain-machine interface device 110 includes a display module 120, a memory 130, and a processor 140.

The display module 120 displays a stimulation array including a plurality of visual stimuli flickering at a preset frequency. At this time, each visual stimulus flickers at different frequencies and induces a steady state visual evoked potential (SSVEP) among the EEG signals to be measured. When the measurement subject watches the visual stimulation, an EEG signal (including a frequency attribute of a blinking time stimulus corresponding to an attention area (i.e., a viewing area) corresponding to the intention of the measurement object (e.g., character or sentence) That is, SSVEP).

The display module 120 may output a visual stimulus set such that a plurality of colors are alternated at specific time intervals. In this way, consecutive syllables on a word (e.g., word and sentence) included in an EEG signal measured from a measurement object can be recognized in a time sequential manner. At this time, the display module 120 may change the color of the entire visual stimulus in a predetermined unit of time (e.g., 1 second unit) and display it. For example, when the visual stimuli of the first and second colors are alternately output, the first color may be white and the second color may be yellow, and the types and the number of colors of the visual stimuli are not limited thereto. In this way, the subject can perceive the predetermined unit of time so as to remind the intention of the syllable unit, so that the syllable included in his / her intention can be considered for each unit time, and attention can be given to the corresponding position on the visual stimulus.

2 is an exemplary diagram of visual stimulation according to an embodiment of the present invention.

Referring to FIG. 2, the display module 120 may display and display visual stimuli, for example, 5 x 5. The number and arrangement of the visual stimuli displayed by the display module 120 are not limited to those shown in FIG. 2, and may be implemented to correspond to various letter shapes. For example, the number of visual stimuli on the array and the size of the array can be further increased for accurate intention determination.

Each visual stimulus may blink at a different frequency according to a predetermined frequency. For example, in Fig. 2, the frequency of f1 may be set to 5 Hz, the frequency of f2 to 6 Hz, the frequency of f3 to 7 Hz, and so on, a total of 25 different frequencies may be provided as visual stimuli. The frequency of each visual stimulus can be set so that the resonance frequency is excluded from each other.

The memory 130 stores a program for recognizing the intent of the measurement object based on the EEG signal measured from the measurement object. At this time, the program may include an algorithm that recognizes the intent of the measurement object based on the context awareness-aware intention recognition model through machine learning.

The processor 140 executes the program stored in the memory 130. [

At this time, the processor 140 displays a visual stimulus for inducing an EEG signal according to a plurality of intentions of the measurement object through the display module 120 according to the execution of the program. At this time, the intention of the measurement object may be a word, a phrase, a section and a sentence including one or more syllables. For example, the intent may be text having a predetermined meaning such as "Hello", "I love you", "Thank you", "Happy" and "Cool" The intention may be information corresponding to control commands such as TV channels such as "KBS1 "," KBS2 ", "MBC ", and" SBS "

The processor 140 sequentially collects EEG signals induced by visual stimulation corresponding to the shape of each syllable included in the intention of the measurement object. In other words, the collected EEG signal is a combination of 'steady state visual evoked potentials (SSVEP)' in which the frequencies of visual stimuli at positions in accordance with the shape of the character in all the visual stimuli are induced. The characteristic extracted from the EEG signal may be a power spectrum value for a frequency corresponding to each visual stimulus, but is not limited thereto. For reference, the power spectrum value may be a fast Fourier transform (FFT) value.

Specifically, the processor 140 controls the display module 120 to display blinking visual stimuli at the set frequency. At this time, the processor 140 may control the output of the visual stimulus through the display module 120 based on a predetermined unit of time (i.e., a unit time set so that the measurement object remembers one syllable). The processor 140 receives the EEG signal measured through the EEG 150 and observing the visual stimulus displayed through the display module 120.

At this time, the measurement subject looks at the visual stimuli displayed through the display module 120 and sequentially remembers the syllables included in the intention of the user for a predetermined time unit. In other words, referring to FIG. 2, in a state in which each visual stimulus is blinking at 25 frequencies irrespective of a user's intention, when the user considers the syllable according to the intention of " Attention can be paid to the position of the syllable according to the shape of the syllable during stimulation. At the same time, the frequency characteristics of visual stimuli according to the intended intention are included in the measured EEG. That is, the measured EEG signals may include frequency attributes corresponding to f1, f2, f3, f8, f13, f17, f21, f4, f9, f14, f19, f24 and f15. For the sake of convenience of explanation, FIG. 2 explains the positions of the visual stimuli to which the measurement object is focused, in order to explain the positions of the visual stimuli in FIGS. 2, 3, 4, 5, 6, 7, 8, 9, It does not mean that the corresponding visual stimulus is displayed separately, and that some visual stimuli are flickered to be distinguished from each other in all the visual stimuli. In addition, the kind of the intentional characteristic included in the EEG signal is not limited to the frequency property.

When the measurement object is intended to be a word, a phrase, a section or a sentence including a plurality of syllables, the processor 140 sets a unit time (For example, one second / syllable) is set so that notification can be performed through a preset output device (not shown). That is, the processor 140 may display a plurality of visual stimuli through the display module 120 such that two or more different colors are alternated every unit time. In addition, the processor 140 may notify the elapsed time of the unit time through a separate output device (such as a speaker), and the unit time elapse notification method is not limited thereto.

Processor 140 may collect a plurality of EEG signals for one intent. At this time, the processor 140 may generate some of the collected EEG signals as training data and another part as testing data. Specifically, the processor 140 models the intention recognition model by learning the learning data through predetermined machine learning. After the intention recognition model is established through the preceding machine learning, the EEG signals collected in real time are used as test data So that the intention of the measurement object can be decoded.

Specifically, when the brain wave signal corresponding to the intention of the labeled measurement object (e.g., a word formed by sequentially recognized syllables and a sentence in which the words are sequentially recognized, etc.) is collected, An intention recognition model can be generated based on a machine learning algorithm such as deep learning. At this time, the processor 140 extracts predetermined features from the collected EEG signals, and uses the extracted characteristics as learning data for machine learning. Processor 140 may select, but is not limited to, some of the collected EEG signals based on feature extraction techniques.

3 is an exemplary diagram of a topology corresponding to an intention recognition model according to an embodiment of the present invention.

The processor 140 may generate a topology corresponding to the intention recognition model based on the plurality of labeled intentions and characteristics extracted from the EEG signal of the measurement object. At this time, the processor 140 can generate the topology such that the characteristics of the EEG correspond to the input layer of the intention recognition model and the plurality of intentions correspond to the output layer. The processor 140 may then generate a topology to include a plurality of hidden layers.

The processor 140 mechanically learns the intention recognition model based on the topology and learning data corresponding to the generated intention recognition model.

At this time, the processor 140 may generate an intention recognition model based on a deep learning-based machine learning algorithm (e.g., a long short-term memory (LSTM) recurrent neural network (RNN) And the RNN is suitable for data with time dependency to the signal, that is, the LSTM RNN processes the sequentially generated EEG signals Processor 140 may be based on any one of an attention mechanism, reinforcement learning, and an external memory network method. An intention recognition model may be generated, but is not limited thereto.

Further, when the learning of the intention recognition model is completed, the processor 140 can evaluate the intention recognition model through the test data.

Meanwhile, when the generation of the intention recognition model is completed, the processor 140 recognizes and determines the intention of the person to be measured from the EEG signal of the measurement object through the intention recognition model.

Specifically, the processor 140 recognizes an EEG signal from the measurement object through the EEG 150.

The processor 140 extracts a plurality of predetermined characteristics from the recognized EEG signal. At this time, the extracted characteristic may be a characteristic of the same kind as the characteristic extracted to generate the intention recognition model.

In addition, the processor 140 detects the intention of the measurement object by passing the extracted features through the learned intention recognition model. At this time, the plurality of characteristics extracted from the recognized EEG signal can be arranged in time sequence in units of syllables based on the unit time.

Next, a context recognition method in the brain-machine interface device 110 according to an embodiment of the present invention will be described with reference to FIG.

4 is a flowchart of a context recognition method in a brain-mechanical interface device 110 according to an embodiment of the present invention.

In a state in which the brain-machine interface device 110 displays a visual stimulus for inducing an EEG (i.e., SSVEP) of an object to be measured through the display module 120 in step S400, Attention is focused on the blinking time stimulus corresponding locally to the character shape for a defined time (i.e., a predetermined unit time).

In this state, the brain-mechanical interface device 110 recognizes an EEG signal to be measured (S410).

At this time, the brain-machine interface device 110 receives the EEG signal corresponding to the shape of the syllable according to the intention of the measurement object (i.e., a combination of SSVEPs caused by the flicker frequency of the attention- Brain wave signal).

Then, the brain-mechanical interface device 110 extracts a plurality of predetermined characteristics from the recognized EEG signal (S420).

At this time, the brain-machine interface device 110 can sequentially extract and arrange the characteristics of each syllable nested on the recognized EEG signal. Further, the frequency component can be extracted as the predetermined characteristic.

Next, the brain-mechanical interface device 110 detects the intention of the measurement object based on the plurality of characteristics extracted from the recognized EEG signal and the generated intention recognition model (S430).

At this time, the intention may be any one of words, phrases, clauses, and sentences including one or more syllables. And the intention recognition model may be generated based on a deep learning based machine learning algorithm and an EEG signal collected corresponding to a plurality of labeled intents. That is, the brain-machine interface device 110 collects a plurality of corresponding brain wave signals for each predetermined intention from the measurement object, and generates an intention recognition model based on the collected brain wave signal and a deep learning-based machine learning algorithm have.

As described above, the context-aware brain-machine interface device 110 and method according to an embodiment of the present invention may be configured to perform a brain-machine interface based on a brain- It is possible to judge intentions including words, phrases, phrases or sentences composed of one or more syllables.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: Brain-machine interface system
110: Brain-machine interface device
120: Display module
130: memory
140: Processor
150: Electroencephalograph

Claims (10)

A context-aware brain-mechanical interface device,
A memory for storing a program for recognizing an intention of the measurement object based on an EEG signal of a measurement object;
And a processor for executing a program stored in the memory,
Wherein the processor is configured to recognize the EEG signal of the measurement subject by executing the program, to extract a plurality of predetermined characteristics from the recognized EEG signal, and to generate the EEG signal based on the extracted plurality of characteristics and the pre- Detecting an intention of the measurement object,
The intention is any one of words, phrases, phrases and sentences including one or more syllables,
Wherein the intention recognition model is generated based on a predetermined machine learning algorithm and an EEG signal collected so as to correspond to each of a plurality of labeled intentions.
The method according to claim 1,
The processor comprising:
Collecting an EEG signal corresponding to a plurality of predetermined intentions from the measurement object,
And generates the intention recognition model based on the collected EEG signal and a machine learning algorithm based on deep learning.
3. The method of claim 2,
Further comprising a display module,
The processor comprising:
Displaying through the display module an array comprising a plurality of visual stimuli that each flash at a unique frequency,
Recognizing an EEG signal of the measurement subject which has been attentively focused on a position of a visual stimulus corresponding to a shape of one or more syllables included in the intention for a predetermined unit time for each syllable,
Extracting the characteristics of the recognized EEG signal in syllable units, arranging the characteristics of each syllable in time sequence, detecting the intention based on the intention recognition model,
Wherein the plurality of visual stimuli are flickered at different frequencies, and are alternately blinking in units of two or more colors different from each other.
3. The method of claim 2,
The processor comprising:
Extracting the predetermined plurality of characteristics from the EEG signals corresponding to the plurality of predefined intentions,
Generating a topology corresponding to the intention recognition model such that an input layer corresponding to the extracted characteristics and an output layer corresponding to the plurality of predetermined intentions are included,
And learns the intention recognition model based on the extracted characteristics and a topology corresponding to the intention recognition model.
3. The method of claim 2,
Wherein the deep learning-based machine learning algorithm is based on a long short-term memory (LSTM) recurrent neural network (RNN).
The method according to claim 1,
Further comprising an electroencephalogram measurer for measuring an electroencephalogram signal from a plurality of regions preset in the head of the measurement subject,
Wherein the processor receives and recognizes the brain wave signal of the measurement object collected through the brain wave measuring device.
A context-aware method of a context-aware brain-mechanical interface device,
Recognizing an EEG signal to be measured;
Extracting a plurality of predetermined characteristics from the recognized EEG signal; And
Detecting an intention of the measurement object based on the extracted plurality of characteristics and the generated intention recognition model,
The intention is any one of words, phrases, phrases and sentences including one or more syllables,
Wherein the intention recognition model is generated based on a predetermined machine learning algorithm and an EEG signal collected so as to correspond to each of a plurality of labeled intentions.
8. The method of claim 7,
Before the step of detecting the intention of the measurement object,
Collecting an EEG signal corresponding to a plurality of predetermined intentions from the measurement object; And
Further comprising generating the intention recognition model based on the collected EEG signal and a machine learning algorithm based on deep learning.
8. The method of claim 7,
Before the step of recognizing the EEG signal of the measurement object,
Further comprising displaying through the display module an array comprising a plurality of visual stimuli that each flash at a unique frequency,
Recognizing the EEG signal of the measurement subject that has been attentively focused on the position of the visual stimulus corresponding to the shape of the one or more syllables included in the intention for a predetermined unit time for each syllable,
Wherein the intention is detected based on the intention recognition model by extracting the characteristics of the recognized EEG signal in syllable units and arranging the characteristics of each syllable in a time sequential manner, Context Awareness Method.
A computer-readable recording medium recording a program for performing the method according to any one of claims 7 to 9 on a computer.

Images