CN111338482A - Brain-controlled character spelling recognition method and system based on supervised self-encoding - Google Patents

Brain-controlled character spelling recognition method and system based on supervised self-encoding Download PDF

Info

Publication number
CN111338482A
CN111338482A CN202010144039.6A CN202010144039A CN111338482A CN 111338482 A CN111338482 A CN 111338482A CN 202010144039 A CN202010144039 A CN 202010144039A CN 111338482 A CN111338482 A CN 111338482A
Authority
CN
China
Prior art keywords
self
encoder
level
brain
weight
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010144039.6A
Other languages
Chinese (zh)
Other versions
CN111338482B (en
Inventor
陈桂军
张雪英
焦江丽
郭柳君
李凤莲
黄丽霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Taiyuan University of Technology
Original Assignee
Taiyuan University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Taiyuan University of Technology filed Critical Taiyuan University of Technology
Priority to CN202010144039.6A priority Critical patent/CN111338482B/en
Publication of CN111338482A publication Critical patent/CN111338482A/en
Application granted granted Critical
Publication of CN111338482B publication Critical patent/CN111338482B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Physics & Mathematics (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Signal Processing (AREA)
  • Psychiatry (AREA)
  • Public Health (AREA)
  • Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physiology (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Neurosurgery (AREA)
  • Neurology (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Dermatology (AREA)
  • Psychology (AREA)

Abstract

The invention relates to a brain-controlled character spelling recognition method and system based on supervised self-coding. The method comprises the steps of acquiring an electroencephalogram signal of a user; preprocessing the electroencephalogram signals; extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain the characteristics of the electroencephalogram signal source; and recognizing the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled. The brain-controlled character spelling recognition method and system based on the supervised self-coding improve the accuracy and efficiency of brain-controlled character spelling.

Description

Brain-controlled character spelling recognition method and system based on supervised self-encoding
Technical Field
The invention relates to the field of human-computer interaction, in particular to a brain-controlled character spelling recognition method and system based on supervised self-coding.
Background
Spelling based on electroencephalogram signals is a brand-new information interaction mode, information exchange and control between the brain and the external environment can be achieved without depending on peripheral nerves and muscle systems of a human body, interaction experience of handicapped people or certain specific limited applications can be greatly improved through the interaction mode, and the method has wide application prospects in the fields of medical rehabilitation, education and teaching, game and entertainment, smart home, military and the like.
The existing brain-controlled spelling system adopting the motor vision evoked potential has the problems that ① coded character instructions are few, the input speed is low, ② recognition and decoding are mainly realized by selecting the optimal electrode leads or characteristics and then using a classification algorithm for recognition and decoding, and the electroencephalogram is a superposed signal of an activation cortex at an electrode, so that the brain activity information of spelling intention cannot be intuitively reflected by a scalp, and the decoding precision and efficiency are not high.
Disclosure of Invention
The invention aims to provide a brain-controlled character spelling recognition method and system based on supervised self-coding, and the accuracy and efficiency of brain-controlled character spelling are improved.
In order to achieve the purpose, the invention provides the following scheme:
a brain-controlled character spelling recognition method based on supervised self-coding comprises the following steps:
acquiring an electroencephalogram signal of a user;
preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain the characteristics of the electroencephalogram signal source; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output;
and recognizing the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled.
Optionally, the method includes extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain an electroencephalogram signal source characteristic, and the method includes:
constructing a three-level self-encoder with output category feedback;
and training the self-encoder to obtain the trained self-encoder.
Optionally, the constructing a three-level self-encoder with output category feedback specifically includes:
taking the preprocessed EEG signal X as input, extracting a first-stage source feature h1＝φ(W1X+b1) Constructing a first-level self-encoder; so that
Figure BDA0002400096490000021
W1、
Figure BDA0002400096490000022
As the first level codec weight, b1、
Figure BDA0002400096490000023
For the first level of codec bias terms,
Figure BDA0002400096490000024
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
by first-order source signature h1As input, second-level source features h are extracted2＝φ(W2h1+b2) Constructing a two-stage auto-encoder to obtain a reconstructed input signal
Figure BDA0002400096490000025
So that
Figure BDA0002400096490000026
W2、
Figure BDA0002400096490000027
As a second level of coding and decoding weight, b2、
Figure BDA0002400096490000028
For the second level of codec bias terms,
Figure BDA0002400096490000029
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
random initialization
Figure BDA00024000964900000210
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure BDA00024000964900000211
h0The updated weight and bias term are used as the weight and bias term of the next input signal, and all signals are subjected to coder pre-training in sequence;
by second-level source signature h2As input, correspond to
Figure BDA00024000964900000212
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure BDA00024000964900000213
Introducing output class y calculation error
Figure BDA00024000964900000214
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure BDA0002400096490000031
Optionally, the recognizing spelling of the brain-controlled character according to the characteristics of the electroencephalogram signal source to obtain a character to be spelled, and then further comprising:
sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, and stimulation interfaces of commas, periods, spaces and returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged according to the periphery of the squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku.
A brain-controlled character spelling recognition system based on supervised self-encoding, comprising:
the electroencephalogram signal acquisition module is used for acquiring an electroencephalogram signal of a user;
the preprocessing module is used for preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module is used for extracting the preprocessed electroencephalogram signals by utilizing the trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output;
and the character determination module to be spelled is used for identifying the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled.
Optionally, the method further includes:
the self-encoder building module is used for building a self-encoder with three-level output category feedback;
and the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder.
Optionally, the self-encoder building module with the three-level output category feedback specifically includes:
a first-stage self-encoder constructing unit for constructing a first-stage self-encoder by taking the preprocessed electroencephalogram signal X as input, and extracting a first-stage source characteristic h1＝φ(W1X+b1) Then reconstructing the input signal
Figure BDA0002400096490000032
So that
Figure BDA0002400096490000033
W1、
Figure BDA0002400096490000034
As the first level codec weight, b1、
Figure BDA0002400096490000035
For the first level of codec bias terms,
Figure BDA0002400096490000041
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
a two-level self-encoder building unit for constructing a first-level source signature h1As input, a two-stage self-encoder is constructed, and a second-stage source feature h is extracted2＝φ(W2h1+b2) Then reconstructing the input signal
Figure BDA0002400096490000042
So that
Figure BDA0002400096490000043
W2、
Figure BDA0002400096490000044
As a second level of coding and decoding weight, b2、
Figure BDA0002400096490000045
For the second level of codec bias terms,
Figure BDA0002400096490000046
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
a pre-training unit for random initialization
Figure BDA0002400096490000047
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure BDA0002400096490000048
Figure BDA0002400096490000049
h0The updated weight and bias term are used as the weight and bias term of the next input signal, and all signals are subjected to coder pre-training in sequence;
a self-encoder building unit with three-level output class feedback for using the second-level source characteristics h2As input, correspond to
Figure BDA00024000964900000410
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure BDA00024000964900000411
Introducing output class y calculation error
Figure BDA00024000964900000412
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure BDA00024000964900000413
Optionally, the method further includes:
the spelling correction module is used for sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, and stimulation interfaces of commas, periods, spaces and returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged according to the periphery of the squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
according to the brain-controlled character spelling recognition method and system based on the supervised self-coding, provided by the invention, electroencephalogram signals are preprocessed to obtain the motor vision evoked potentials, and because the stimulation presentation of each character is carried out simultaneously, the conversion efficiency from intentions to the electroencephalogram signals can be improved; and extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain the characteristics of the electroencephalogram signal source. And recognizing the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled. The method is characterized in that a motion visual evoked potential signal is approximately mapped to a cortex source current density signal, a target category is introduced in the self-coding model building process, cortex source current density characteristics related to the target category as far as possible are extracted, and the accuracy of target identification is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a schematic flow chart of a brain-controlled character spelling recognition method based on supervised self-encoding according to the present invention;
FIG. 2 is a schematic diagram of a system corresponding to a brain-controlled character spelling recognition method based on supervised self-encoding according to the present invention;
FIG. 3 is a schematic view of a stimulation interface;
FIG. 4 is a schematic diagram of a three-level self-encoder with output class feedback;
fig. 5 is a schematic structural diagram of a brain-controlled character spelling recognition system based on supervised self-encoding according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a brain-controlled character spelling recognition method and system based on supervised self-coding, and the accuracy and efficiency of brain-controlled character spelling are improved.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Fig. 1 is a schematic flow chart of a brain-controlled character spelling recognition method based on supervised self-encoding, as shown in fig. 1, the brain-controlled character spelling recognition method based on supervised self-encoding includes:
s101, acquiring electroencephalogram signals of a user. The user sits right in front of the display, looks at the screen with eyes, after the spelling of characters is started, as shown in fig. 2, the 64-lead Ag/AgCl electrode is arranged on the scalp of the user according to an international 10-20 system mode to obtain the electroencephalogram signals of the user.
After acquiring the electroencephalogram of the user, the acquired electrical signal is amplified, and as shown in fig. 2, the acquisition and amplification device acquires and amplifies the electroencephalogram.
Constructing a three-level self-encoder with output category feedback, and as shown in FIG. 4;
taking the preprocessed EEG signal X as input, extracting a first-stage source feature h1＝φ(W1X+b1) Constructing a first-level self-encoder; so that
Figure BDA0002400096490000061
W1、
Figure BDA0002400096490000062
As the first level codec weight, b1、
Figure BDA0002400096490000063
For the first level of codec bias terms,
Figure BDA0002400096490000064
for the neuron activation function, the weight and the bias term can be obtained through pre-training.
By first-order source signature h1As input, second-level source features h are extracted2＝φ(W2h1+b2) Constructing a two-stage auto-encoder to obtain a reconstructed input signal
Figure BDA0002400096490000065
So that
Figure BDA0002400096490000066
W2、
Figure BDA0002400096490000067
As a second level of coding and decoding weight, b2、
Figure BDA0002400096490000068
For the second level of codec bias terms,
Figure BDA0002400096490000069
for the neuron activation function, the weight and the bias term can be obtained through pre-training.
By second-level source signature h2As an input to the process, the process may,correspond to
Figure BDA00024000964900000610
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure BDA00024000964900000611
Introducing output class y calculation error
Figure BDA00024000964900000612
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure BDA00024000964900000613
And pre-training the self-encoder to obtain the trained self-encoder.
Random initialization
Figure BDA00024000964900000614
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure BDA00024000964900000615
h0And (2) sequentially pre-training all signals by using the updated weight and offset term as the weight and offset term of the next input signal, wherein j is 1,2 and 3, and epsilon is a learning rate.
S102, preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulation presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference.
S103, extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output.
And S104, recognizing spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled.
Sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, a comma, a period, a blank and a stimulation interface returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged on the periphery of a squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku and is shown in figure 3.
2-level presentation is carried out through a stimulation interface, the stimulation time of one movement is 400ms, a quick movement square block is marked as 1 in the current stimulation test, a quick movement square block is not marked as 0, and the quick movement stimulations of i times can be marked as 2 at the same timeiAnd outputting each target, namely identifying a block code and a character code by 2 steps according to the intention of the user for selecting the characters, wherein the block code adopts 3 bits to realize 5 groups of output, the character code adopts 3 bits to realize 8 characters of output, each character selection process refers to fig. 3 to perform identification output, one-time quick movement stimulation is finished, all movement blocks disappear for 100ms, and the movement blocks are used as the starting mark of the next stimulation test, and a stimulation presentation program can be written by psychtolbo.
Stimulating the interface display and brain electricity data acquisition, the user sits in the display dead ahead, eyes watch the screen, after the spelling of the characters starts, the sliding square in the stimulating presentation interface presents according to the group code rule shown in table one, then presents according to the character code rule, 6 times of moving process from left to right square can mark a character to be selected, and the user marks the character to be selected and outputs, taking the character N as an example, the sliding square of the group code sequentially slides to the right, is static and is static in 3 presentation stages, the group code marks are 1, 0 and 0, the sliding square of the character code sequentially slides, is static and slides in 3 presentation stages, and the character code marks are 1, 0 and 1. Table one is shown below:
watch 1
Figure BDA0002400096490000081
Fig. 5 is a schematic structural diagram of a brain-controlled character spelling recognition system based on supervised self-encoding, as shown in fig. 5, the brain-controlled character spelling recognition system based on supervised self-encoding provided by the present invention includes: an electroencephalogram signal acquisition module 501, a preprocessing module 502, an electroencephalogram signal source characteristic determining module 503 and a character determining module 504 to be spelled,
the electroencephalogram signal acquisition module 501 is used for acquiring an electroencephalogram signal of a user;
the preprocessing module 502 is used for preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module 503 is configured to extract the preprocessed electroencephalogram signal by using the trained self-encoder to obtain an electroencephalogram signal source characteristic; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output;
the to-be-spelled character determination module 504 is configured to recognize the spelling of the brain-controlled character according to the characteristics of the electroencephalogram signal source, so as to obtain the to-be-spelled character.
The brain-controlled character spelling recognition system based on the supervised self-encoding provided by the invention further comprises: the system comprises a three-level self-encoder building module with output category feedback, a self-encoder training module and a spelling correction module.
The self-encoder building module with the three-level output category feedback is used for building a self-encoder with the three-level output category feedback;
and the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder.
The spelling correction module is used for sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, and stimulation interfaces of commas, periods, spaces and returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged according to the periphery of the squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku.
The self-encoder building module with the three-level output category feedback specifically comprises: the device comprises a first-level self-encoder building unit, a second-level self-encoder building unit, a pre-training unit and a third-level self-encoder building unit with output category feedback.
The primary self-coding construction unit is used for taking the preprocessed electroencephalogram signal X as input and extracting a primary source characteristic h1＝φ(W1X+b1) Then reconstructing the input signal
Figure BDA0002400096490000091
So that
Figure BDA0002400096490000092
W1、
Figure BDA0002400096490000093
As the first level codec weight, b1、
Figure BDA0002400096490000094
For the first level of codec bias terms,
Figure BDA0002400096490000095
for the neuron activation function, the weight and the bias term can be obtained through pre-training.
The two-level self-coding construction unit is used for constructing a first-level source characteristic h1As input, second-level source features h are extracted2＝φ(W2h1+b2) Then reconstructing the input signal
Figure BDA0002400096490000096
So that
Figure BDA0002400096490000097
W2、
Figure BDA0002400096490000098
As a second level of coding and decoding weight, b2、
Figure BDA0002400096490000099
For the second level of codec bias terms,
Figure BDA00024000964900000910
for the neuron activation function, the weight and the bias term can be obtained through pre-training.
Pre-training unit random initialization
Figure BDA00024000964900000911
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure BDA00024000964900000912
Figure BDA00024000964900000913
h0And (2) sequentially pre-training all signals by using the updated weight and offset term as the weight and offset term of the next input signal, wherein j is 1,2 and 3, and epsilon is a learning rate.
The self-encoder building unit with three-level output class feedback is used for utilizing a second-level source characteristic h2As input, correspond to
Figure BDA00024000964900000914
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure BDA00024000964900000915
Introducing output class y calculation error
Figure BDA00024000964900000916
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure BDA00024000964900000917
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principle and the implementation mode of the invention are explained by applying a specific example, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.

Claims (8)

1. A brain-controlled character spelling recognition method based on supervised self-coding is characterized by comprising the following steps:
acquiring an electroencephalogram signal of a user;
preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
extracting the preprocessed electroencephalogram signal by using a trained self-encoder to obtain the characteristics of the electroencephalogram signal source; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output;
and recognizing the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled.
2. The brain-controlled character spelling recognition method based on supervised self-encoding as recited in claim 1, wherein the extraction of the preprocessed electroencephalogram signal by using the trained self-encoder to obtain the characteristics of the electroencephalogram signal source further comprises:
constructing a three-level self-encoder with output category feedback;
and training the self-encoder to obtain the trained self-encoder.
3. The brain-controlled character spelling recognition method based on supervised self-encoding as recited in claim 1, wherein the constructing of the three-level self-encoder with output class feedback specifically includes:
taking the preprocessed EEG signal X as input, extracting a first-stage source feature h1＝φ(W1X+b1) Constructing a first-level self-encoder; so that
Figure FDA0002400096480000011
W1、
Figure FDA0002400096480000012
As the first level codec weight, b1、
Figure FDA0002400096480000013
For the first level of codec bias terms,
Figure FDA0002400096480000014
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
by first-order source signature h1As input, second-level source features h are extracted2＝φ(W2h1+b2) Constructing a two-stage auto-encoder to obtain a reconstructed input signal
Figure FDA0002400096480000015
So that
Figure FDA0002400096480000016
W2、
Figure FDA0002400096480000017
As a second level of coding and decoding weight, b2、
Figure FDA0002400096480000018
For the second level of codec bias terms,
Figure FDA0002400096480000019
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
random initialization
Figure FDA00024000964800000110
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure FDA00024000964800000111
h0The updated weight and bias term are used as the weight and bias term of the next input signal, and all signals are subjected to coder pre-training in sequence;
by second-level source signature h2As input, correspond to
Figure FDA0002400096480000021
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure FDA0002400096480000022
Introducing output class y calculation error
Figure FDA0002400096480000023
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure FDA0002400096480000024
4. The brain-controlled character spelling recognition method based on supervised self-coding as recited in claim 1, wherein the recognition of the brain-controlled character spelling is performed according to the electroencephalogram signal source characteristics, so as to obtain a character to be spelled, and then further comprising:
sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, and stimulation interfaces of commas, periods, spaces and returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged according to the periphery of the squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku.
5. A brain-controlled character spelling recognition system based on supervised self-encoding, comprising:
the electroencephalogram signal acquisition module is used for acquiring an electroencephalogram signal of a user;
the preprocessing module is used for preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module is used for extracting the preprocessed electroencephalogram signals by utilizing the trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signal as input and takes the characteristics of the electroencephalogram signal source as output;
and the character determination module to be spelled is used for identifying the spelling of the brain-controlled characters according to the characteristics of the electroencephalogram signal source to obtain the characters to be spelled.
6. The brain-controlled character spelling recognition system based on supervised self-encoding as recited in claim 5, further comprising:
the self-encoder building module is used for building a self-encoder with three-level output category feedback;
and the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder.
7. The brain-controlled character spelling recognition system based on supervised self-encoding as recited in claim 5, wherein the self-encoder building block with output class feedback comprises:
a first-stage self-encoder constructing unit for constructing a first-stage self-encoder by taking the preprocessed electroencephalogram signal X as input, and extracting a first-stage source characteristic h1＝φ(W1X+b1) Then reconstructing the input signal
Figure FDA0002400096480000031
So that
Figure FDA0002400096480000032
W1、
Figure FDA0002400096480000033
As the first level codec weight, b1、
Figure FDA0002400096480000034
For the first level of codec bias terms,
Figure FDA0002400096480000035
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
a two-level self-encoder building unit for constructing a first-level source signature h1AsInputting, constructing a secondary self-encoder, and extracting a secondary source feature h2＝φ(W2h1+b2) Then reconstructing the input signal
Figure FDA0002400096480000036
So that
Figure FDA0002400096480000037
W2、
Figure FDA0002400096480000038
As a second level of coding and decoding weight, b2、
Figure FDA0002400096480000039
For the second level of codec bias terms,
Figure FDA00024000964800000310
for the neuron activation function, the weight and the bias item can be obtained through pre-training;
a pre-training unit for random initialization
Figure FDA00024000964800000311
T represents transposition, and then an update amount is calculated from each input signal and the reconstructed signal
Figure FDA00024000964800000312
Figure FDA00024000964800000313
h0The updated weight and bias term are used as the weight and bias term of the next input signal, and all signals are subjected to coder pre-training in sequence;
a self-encoder building unit with three-level output class feedback for using the second-level source characteristics h2As input, correspond to
Figure FDA00024000964800000314
As the output category, [10 ]]Represents a non-target, [01 ]]Representing the target, extracting a third-level source feature h3＝φ(W3h2+b3) Constructing a three-level self-encoder with output category feedback; the third level source signature h3Predicting output by softmax layer
Figure FDA00024000964800000315
Introducing output class y calculation error
Figure FDA00024000964800000316
Fine tuning the weight and the offset item layer by using a gradient descent method, wherein each level of the fine tuning update quantity of the weight and the offset item
Figure FDA00024000964800000317
8. The brain-controlled character spelling recognition system based on supervised self-encoding as recited in claim 5, further comprising:
the spelling correction module is used for sending the character to be spelled to a stimulation interface for spelling correction; the stimulation interfaces comprise 26 English characters from A to Z, 10 numeric characters from 0 to 9, and stimulation interfaces of commas, periods, spaces and returning 4 functional characters, and are divided into 5 groups, 8 characters in each group are arranged according to the periphery of the squared Sudoku, and a square moving rapidly from left to right is arranged below each character and in the center of the squared Sudoku.
CN202010144039.6A 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding Active CN111338482B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010144039.6A CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010144039.6A CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Publications (2)

Publication Number Publication Date
CN111338482A true CN111338482A (en) 2020-06-26
CN111338482B CN111338482B (en) 2023-04-25

Family

ID=71182048

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010144039.6A Active CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Country Status (1)

Country Link
CN (1) CN111338482B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114167989A (en) * 2021-12-09 2022-03-11 太原理工大学 Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding

Citations (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050017870A1 (en) * 2003-06-05 2005-01-27 Allison Brendan Z. Communication methods based on brain computer interfaces
US20050261803A1 (en) * 2004-04-15 2005-11-24 Neurosciences Research Foundation, Inc. Mobile brain-based device for use in a real world environment
US20070091265A1 (en) * 2002-01-18 2007-04-26 Kardon Randy H System and method for optical imaging of human retinal function
JP2008009632A (en) * 2006-06-28 2008-01-17 Denso Corp Character input device
EP1975809A1 (en) * 2007-03-30 2008-10-01 Research In Motion Limited Use of multiple data sources for spell check function, and associated handheld electronic device
US20100145214A1 (en) * 2007-02-09 2010-06-10 Agency For Science, Technology And Research system and method for processing brain signals in a bci system
US20110289030A1 (en) * 2008-05-26 2011-11-24 Shijian Lu Method and system for classifying brain signals in a bci
US20120323967A1 (en) * 2011-06-14 2012-12-20 Microsoft Corporation Spelling Using a Fuzzy Pattern Search
KR20130002590A (en) * 2011-06-29 2013-01-08 한양대학교 산학협력단 Word input interface apparatus and method with a qwerty type using steady state visual evoked potential
CN103488297A (en) * 2013-09-30 2014-01-01 华南理工大学 Online semi-supervising character input system and method based on brain-computer interface
JP2015148945A (en) * 2014-02-06 2015-08-20 国立大学法人名古屋大学 Character input device, and program
CN104978035A (en) * 2015-07-27 2015-10-14 中国医学科学院生物医学工程研究所 Brain computer interface system evoking P300 based on somatosensory electrical stimulation and implementation method thereof
WO2015192610A1 (en) * 2014-06-17 2015-12-23 华南理工大学 Intelligent wheel chair control method based on brain computer interface and automatic driving technology
CN105512681A (en) * 2015-12-07 2016-04-20 北京信息科技大学 Method and system for acquiring target category picture
CN105512534A (en) * 2015-12-31 2016-04-20 湖南人文科技学院 Self-destroyable identity recognition system based on combined authentication of brain wave, iris and human face recognition
WO2017084416A1 (en) * 2015-11-17 2017-05-26 天津大学 Feedback system based on motor imagery brain-computer interface
US20170206004A1 (en) * 2014-07-15 2017-07-20 Amar Y Servir Input of characters of a symbol-based written language
CN107038421A (en) * 2017-04-17 2017-08-11 杭州电子科技大学 Modulation Types recognition methods based on sparse storehouse own coding
CN107122050A (en) * 2017-04-26 2017-09-01 西安交通大学 Stable state of motion VEP brain-machine interface method based on CSFL GDBN
CN107481359A (en) * 2017-07-14 2017-12-15 昆明理工大学 Intelligent door lock system and its control method based on P300 and Mental imagery
CN108537133A (en) * 2018-03-16 2018-09-14 江苏经贸职业技术学院 A kind of face reconstructing method based on supervised learning depth self-encoding encoder
CN108921141A (en) * 2018-08-16 2018-11-30 广东工业大学 A kind of EEG signals EEG feature extracting method encoding neural network certainly based on depth
CN109222966A (en) * 2018-10-09 2019-01-18 山东大学 A kind of EEG signals sensibility classification method based on variation self-encoding encoder
CN109472194A (en) * 2018-09-26 2019-03-15 重庆邮电大学 A kind of Mental imagery EEG signals characteristic recognition method based on CBLSTM algorithm model
CN110139597A (en) * 2016-12-21 2019-08-16 英乐爱有限公司 The system and method for being iterated classification using neuro-physiological signals
CN110123313A (en) * 2019-04-17 2019-08-16 中国科学院深圳先进技术研究院 A kind of self-training brain machine interface system and related training method
CN110262658A (en) * 2019-06-14 2019-09-20 长春理工大学 A kind of brain-computer interface character input system and implementation method based on reinforcing attention
CN110347242A (en) * 2019-05-29 2019-10-18 长春理工大学 Audio visual brain-computer interface spelling system and its method based on space and semantic congruence
CN110364255A (en) * 2019-06-06 2019-10-22 江苏大学 A kind of hepatopathy appraisal procedure based on self-encoding encoder
CN110353672A (en) * 2019-07-15 2019-10-22 西安邮电大学 Eye artefact removal system and minimizing technology in a kind of EEG signals
RU2704497C1 (en) * 2019-03-05 2019-10-29 Общество с ограниченной ответственностью "Нейроботикс" Method for forming brain-computer control system
CN110688013A (en) * 2019-10-11 2020-01-14 南京邮电大学 English keyboard spelling system and method based on SSVEP

Patent Citations (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070091265A1 (en) * 2002-01-18 2007-04-26 Kardon Randy H System and method for optical imaging of human retinal function
US20050017870A1 (en) * 2003-06-05 2005-01-27 Allison Brendan Z. Communication methods based on brain computer interfaces
US20050261803A1 (en) * 2004-04-15 2005-11-24 Neurosciences Research Foundation, Inc. Mobile brain-based device for use in a real world environment
JP2008009632A (en) * 2006-06-28 2008-01-17 Denso Corp Character input device
US20100145214A1 (en) * 2007-02-09 2010-06-10 Agency For Science, Technology And Research system and method for processing brain signals in a bci system
EP1975809A1 (en) * 2007-03-30 2008-10-01 Research In Motion Limited Use of multiple data sources for spell check function, and associated handheld electronic device
US20110289030A1 (en) * 2008-05-26 2011-11-24 Shijian Lu Method and system for classifying brain signals in a bci
US20120323967A1 (en) * 2011-06-14 2012-12-20 Microsoft Corporation Spelling Using a Fuzzy Pattern Search
KR20130002590A (en) * 2011-06-29 2013-01-08 한양대학교 산학협력단 Word input interface apparatus and method with a qwerty type using steady state visual evoked potential
CN103488297A (en) * 2013-09-30 2014-01-01 华南理工大学 Online semi-supervising character input system and method based on brain-computer interface
JP2015148945A (en) * 2014-02-06 2015-08-20 国立大学法人名古屋大学 Character input device, and program
WO2015192610A1 (en) * 2014-06-17 2015-12-23 华南理工大学 Intelligent wheel chair control method based on brain computer interface and automatic driving technology
US20170206004A1 (en) * 2014-07-15 2017-07-20 Amar Y Servir Input of characters of a symbol-based written language
CN104978035A (en) * 2015-07-27 2015-10-14 中国医学科学院生物医学工程研究所 Brain computer interface system evoking P300 based on somatosensory electrical stimulation and implementation method thereof
WO2017084416A1 (en) * 2015-11-17 2017-05-26 天津大学 Feedback system based on motor imagery brain-computer interface
CN105512681A (en) * 2015-12-07 2016-04-20 北京信息科技大学 Method and system for acquiring target category picture
CN105512534A (en) * 2015-12-31 2016-04-20 湖南人文科技学院 Self-destroyable identity recognition system based on combined authentication of brain wave, iris and human face recognition
CN110139597A (en) * 2016-12-21 2019-08-16 英乐爱有限公司 The system and method for being iterated classification using neuro-physiological signals
CN107038421A (en) * 2017-04-17 2017-08-11 杭州电子科技大学 Modulation Types recognition methods based on sparse storehouse own coding
CN107122050A (en) * 2017-04-26 2017-09-01 西安交通大学 Stable state of motion VEP brain-machine interface method based on CSFL GDBN
CN107481359A (en) * 2017-07-14 2017-12-15 昆明理工大学 Intelligent door lock system and its control method based on P300 and Mental imagery
CN108537133A (en) * 2018-03-16 2018-09-14 江苏经贸职业技术学院 A kind of face reconstructing method based on supervised learning depth self-encoding encoder
CN108921141A (en) * 2018-08-16 2018-11-30 广东工业大学 A kind of EEG signals EEG feature extracting method encoding neural network certainly based on depth
CN109472194A (en) * 2018-09-26 2019-03-15 重庆邮电大学 A kind of Mental imagery EEG signals characteristic recognition method based on CBLSTM algorithm model
CN109222966A (en) * 2018-10-09 2019-01-18 山东大学 A kind of EEG signals sensibility classification method based on variation self-encoding encoder
RU2704497C1 (en) * 2019-03-05 2019-10-29 Общество с ограниченной ответственностью "Нейроботикс" Method for forming brain-computer control system
CN110123313A (en) * 2019-04-17 2019-08-16 中国科学院深圳先进技术研究院 A kind of self-training brain machine interface system and related training method
CN110347242A (en) * 2019-05-29 2019-10-18 长春理工大学 Audio visual brain-computer interface spelling system and its method based on space and semantic congruence
CN110364255A (en) * 2019-06-06 2019-10-22 江苏大学 A kind of hepatopathy appraisal procedure based on self-encoding encoder
CN110262658A (en) * 2019-06-14 2019-09-20 长春理工大学 A kind of brain-computer interface character input system and implementation method based on reinforcing attention
CN110353672A (en) * 2019-07-15 2019-10-22 西安邮电大学 Eye artefact removal system and minimizing technology in a kind of EEG signals
CN110688013A (en) * 2019-10-11 2020-01-14 南京邮电大学 English keyboard spelling system and method based on SSVEP

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
HONG BO: "N200-speller using motion-onset visual response", 《CLINICAL NEUROPHYSIOLOGY》 *
佘玉梅等: "《人工智能原理及应用》", 31 December 2018 *
张赛等: "基于监督学习自动编码器图像重构", 《江苏科技信息》 *
杨保杰等: "基于深度自动编码器的脑网络状态观测矩阵降维方法", 《传感器与微系统》 *
韦鹏程等: "《大数据巨量分析与机器学习的整合与开发》", 31 May 2017 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114167989A (en) * 2021-12-09 2022-03-11 太原理工大学 Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding

Also Published As

Publication number Publication date
CN111338482B (en) 2023-04-25

Similar Documents

Publication Publication Date Title
CN113288183B (en) Silent voice recognition method based on facial neck surface myoelectricity
CN106569604B (en) Audiovisual bimodal semantic matches and semantic mismatch collaboration stimulation brain-machine interface method
CN101464729B (en) Independent desire expression method based on auditory sense cognition neural signal
CN109841122A (en) A kind of intelligent robot tutoring system and student's learning method
CN107981997B (en) A kind of method for controlling intelligent wheelchair and system based on human brain motion intention
EP3235540A1 (en) Auxiliary device for training and auxiliary method for training
CN115590515A (en) Emotion recognition method and system based on generative self-supervision learning and electroencephalogram signals
CN105280044A (en) Intelligent teaching system for ASD (Autism Spectrum Disorder) children
CN105824418A (en) Brain-computer interface communication system based on asymmetric visual evoked potential
CN104978035A (en) Brain computer interface system evoking P300 based on somatosensory electrical stimulation and implementation method thereof
CN111783942A (en) Brain cognition process simulation method based on convolution cyclic neural network
CN102200833A (en) Speller brain-computer interface (SCI) system and control method thereof
Birkhoff Mathematics and psychology
CN111338482A (en) Brain-controlled character spelling recognition method and system based on supervised self-encoding
CN111772629A (en) Brain cognitive skill transplantation method
CN111443799A (en) Auxiliary learning method based on brain-computer interface, terminal and computer storage medium
CN117608402A (en) Hidden Chinese language processing system and method based on Chinese character writing imagination
Yu et al. A P300-based brain–computer interface for Chinese character input
Du et al. InceptionSSVEP: a multi-scale convolutional neural network for steady-state visual evoked potential classification
CN117608399A (en) Track fitting method and device based on Chinese character strokes
Hild et al. An ERP-based brain-computer interface for text entry using rapid serial visual presentation and language modeling
CN109801513A (en) A kind of passive accumulating method and wearable device based on tactile
CN114167989B (en) Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding
CN114997229A (en) Electroencephalogram signal-based motor imagery classification method and system
Li et al. MyoTac: Real‐Time Recognition of Tactical Sign Language Based on Lightweight Deep Neural Network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant