CN111338482B - Brain-controlled character spelling recognition method and system based on supervision self-coding - Google Patents

Brain-controlled character spelling recognition method and system based on supervision self-coding Download PDF

Info

Publication number
CN111338482B
CN111338482B CN202010144039.6A CN202010144039A CN111338482B CN 111338482 B CN111338482 B CN 111338482B CN 202010144039 A CN202010144039 A CN 202010144039A CN 111338482 B CN111338482 B CN 111338482B
Authority
CN
China
Prior art keywords
self
encoder
level
brain
character
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010144039.6A
Other languages
Chinese (zh)
Other versions
CN111338482A (en
Inventor
陈桂军
张雪英
焦江丽
郭柳君
李凤莲
黄丽霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Taiyuan University of Technology
Original Assignee
Taiyuan University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Taiyuan University of Technology filed Critical Taiyuan University of Technology
Priority to CN202010144039.6A priority Critical patent/CN111338482B/en
Publication of CN111338482A publication Critical patent/CN111338482A/en
Application granted granted Critical
Publication of CN111338482B publication Critical patent/CN111338482B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns

Abstract

The invention relates to a brain-controlled character spelling recognition method and system based on supervision self-coding. The method comprises the steps of acquiring an electroencephalogram signal of a user; preprocessing the electroencephalogram signals; extracting the preprocessed electroencephalogram signals by using a trained self-encoder to obtain electroencephalogram signal source characteristics; and carrying out brain control character spelling recognition according to the brain electrical signal source characteristics to obtain the character to be spelled. The brain-controlled character spelling recognition method and system based on the supervision self-coding provided by the invention can be used for improving the accuracy and efficiency of brain-controlled character spelling.

Description

Brain-controlled character spelling recognition method and system based on supervision self-coding
Technical Field
The invention relates to the field of man-machine interaction, in particular to a brain-controlled character spelling recognition method and system based on supervision self-coding.
Background
The spelling based on the electroencephalogram signals is a brand-new information interaction mode, information exchange and control between the brain and the external environment can be realized without depending on peripheral nerves and muscle systems of human bodies, and the interaction mode can greatly improve interaction experience of handicapped people or some specific limited applications, and has wide application prospects in the fields of medical rehabilitation, education and teaching, game entertainment, intelligent home, military and the like.
The current character spelling system based on the electroencephalogram signals is mainly carried out by using Event Related Potential (ERP) or steady-state visual evoked potential (SSVEP), the ERP is serially evoked in time sequence, and when the number of targets is large, the time required for outputting one target is long; although the SSVEP has high information transmission rate, the flickering of the characters can cause visual fatigue of users, and the SSVEP is not suitable for long-term use, so that the spelling speed of the characters can be improved by adopting the sports visual evoked potential, and the visual fatigue of the users is reduced. The existing brain-controlled spelling adopting the motion visual evoked potential has the problems that: (1) the coding character instruction is less, the input speed is slow, (2) the identification decoding is carried out mainly by selecting the optimal electrode lead or characteristic and then carrying out the identification decoding by using a classification algorithm, and as the brain electricity is the superposition signal of the skin activation source at the scalp electrode, the spelling intention brain activity information can not be intuitively reflected, and the decoding precision and efficiency are not high.
Disclosure of Invention
The invention aims to provide a brain-controlled character spelling recognition method and system based on supervision self-coding, which can improve the accuracy and efficiency of brain-controlled character spelling.
In order to achieve the above object, the present invention provides the following solutions:
a brain-controlled character spelling recognition method based on supervision self-coding comprises the following steps:
acquiring an electroencephalogram signal of a user;
preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifacts removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
extracting the preprocessed electroencephalogram signals by using a trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signals as input and takes the electroencephalogram signal source characteristics as output;
and carrying out brain control character spelling recognition according to the brain electrical signal source characteristics to obtain the character to be spelled.
Optionally, the extracting the preprocessed electroencephalogram signal by using the trained self-encoder to obtain the electroencephalogram signal source feature further includes:
constructing a three-level self-encoder with output category feedback;
and training the self-encoder to obtain the trained self-encoder.
Optionally, the constructing a three-level self-encoder with output class feedback specifically includes:
taking the preprocessed electroencephalogram signal X as input to extract first-stage source characteristics h 1 ＝φ(W 1 X+b 1 ) Constructing a primary self-encoder; so that
Figure BDA0002400096490000021
W 1 、/>
Figure BDA0002400096490000022
B is the first level coding and decoding weight value 1 、/>
Figure BDA0002400096490000023
For the first level codec bias term, +.>
Figure BDA0002400096490000024
The weights and bias terms can be obtained through pre-training for the neuron activation function;
with first-level source characteristics h 1 As input, extract the second level source features h 2 ＝φ(W 2 h 1 +b 2 ) Constructing a two-stage self-encoder to obtain a reconstructed input signal
Figure BDA0002400096490000025
Make->
Figure BDA0002400096490000026
W 2 、/>
Figure BDA0002400096490000027
B is the second level coding and decoding weight 2 、/>
Figure BDA0002400096490000028
For the second level codec bias term, +.>
Figure BDA0002400096490000029
Weights and bias terms can be used to activate the function for neuronsPre-training to obtain;
random initialization
Figure BDA00024000964900000210
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure BDA00024000964900000211
h 0 The method comprises the steps that (1) X, j=1, 2,3 and epsilon are learning rates, updated weights and bias items are used as weights and bias items of the next input signals, and encoder pre-training is carried out on all signals in sequence;
with second-level source characteristics h 2 As input, correspond to
Figure BDA00024000964900000212
For outputting category [10 ]]Representing non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output by softmax layer +.>
Figure BDA00024000964900000213
Introducing output class y calculation errors
Figure BDA00024000964900000214
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount
Figure BDA0002400096490000031
Optionally, the identifying of the spelling of the brain-controlled character according to the electroencephalogram signal source feature, to obtain the character to be spelled, and then further includes:
sending the character to be spelled to a stimulus interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square which moves rapidly from left to right is arranged below each character and in the center of the nine square.
A supervised self-encoding based brain-controlled character spelling recognition system, comprising:
the electroencephalogram signal acquisition module is used for acquiring electroencephalogram signals of a user;
the preprocessing module is used for preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifacts removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module is used for extracting the preprocessed electroencephalogram signals by using the trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signals as input and takes the electroencephalogram signal source characteristics as output;
and the character determining module to be spelled is used for carrying out recognition of brain-controlled character spelling according to the brain-controlled character source characteristics to obtain the character to be spelled.
Optionally, the method further comprises:
the three-level self-encoder construction module with output type feedback is used for constructing the three-level self-encoder with output type feedback;
and the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder.
Optionally, the three-stage self-encoder construction module with output type feedback specifically includes:
a first-stage self-encoder construction unit for constructing a first-stage self-encoder by taking the preprocessed electroencephalogram signal X as an input, and extracting a first-stage source characteristic h 1 ＝φ(W 1 X+b 1 ) Then reconstruct the input signal
Figure BDA0002400096490000032
Make->
Figure BDA0002400096490000033
W 1 、
Figure BDA0002400096490000034
B is the first level coding and decoding weight value 1 、/>
Figure BDA0002400096490000035
For the first level codec bias term, +.>
Figure BDA0002400096490000041
The weights and bias terms can be obtained through pre-training for the neuron activation function;
a two-level self-encoder construction unit for generating a first-level source characteristic h 1 As input, a two-level self-encoder is constructed, and a second-level source characteristic h is extracted 2 ＝φ(W 2 h 1 +b 2 ) Then reconstruct the input signal
Figure BDA0002400096490000042
Make->
Figure BDA0002400096490000043
W 2 、/>
Figure BDA0002400096490000044
B is the second level coding and decoding weight 2 、/>
Figure BDA0002400096490000045
For the second level codec bias term, +.>
Figure BDA0002400096490000046
The weights and bias terms can be obtained through pre-training for the neuron activation function;
a pre-training unit for randomly initializing
Figure BDA0002400096490000047
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure BDA0002400096490000048
Figure BDA0002400096490000049
h 0 The method comprises the steps that (1) X, j=1, 2,3 and epsilon are learning rates, updated weights and bias items are used as weights and bias items of the next input signals, and encoder pre-training is carried out on all signals in sequence;
three-level self-encoder building block with output class feedback for generating a second level source signature h 2 As input, correspond to
Figure BDA00024000964900000410
For outputting category [10]Representing a non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output through softmax layers
Figure BDA00024000964900000411
Introducing an output class y calculation error->
Figure BDA00024000964900000412
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount +.>
Figure BDA00024000964900000413
Optionally, the method further comprises:
the spelling correction module is used for sending the character to be spelled to a stimulus interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square which moves rapidly from left to right is arranged below each character and in the center of the nine square.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
according to the brain-controlled character spelling recognition method and system based on supervision self-coding, the brain-controlled character spelling recognition method and system based on supervision self-coding are characterized in that brain-controlled signals are preprocessed to obtain the motion visual evoked potential, and as each character stimulus presentation is performed simultaneously, the conversion efficiency from intention to brain-controlled signals can be improved; and extracting the preprocessed brain electrical signals by using a trained self-encoder to obtain brain electrical signal source characteristics. And carrying out brain control character spelling recognition according to the brain electrical signal source characteristics to obtain the character to be spelled. The motion visual evoked potential signals are approximately mapped into the cortex source current density signals, the target class is introduced in the construction process of the self-coding model, the cortex source current density characteristics which are related to the target class as much as possible are extracted, and the accuracy of target identification is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a brain-controlled character spelling recognition method based on supervised self-coding;
FIG. 2 is a schematic diagram of a system corresponding to a brain-controlled character spelling recognition method based on supervised self-coding;
FIG. 3 is a schematic illustration of a stimulus interface;
FIG. 4 is a schematic diagram of a three-level self-encoder with output class feedback;
fig. 5 is a schematic diagram of a brain-controlled character spelling recognition system based on supervised self-coding.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a brain-controlled character spelling recognition method and system based on supervision self-coding, which can improve the accuracy and efficiency of brain-controlled character spelling.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Fig. 1 is a schematic flow chart of a brain-controlled character spelling recognition method based on supervised self-coding, as shown in fig. 1, and the brain-controlled character spelling recognition method based on supervised self-coding provided by the invention comprises the following steps:
s101, acquiring an electroencephalogram signal of a user. The user sits right in front of the display, the eyes look at the screen, after the spelling of the characters is started, and as shown in figure 2, 64 Ag/AgCl electrodes are distributed on the scalp of the user according to the international 10-20 system mode to acquire the brain electrical signals of the user.
After acquiring the brain electrical signals of the user, amplifying the acquired electrical signals, as shown in fig. 2, by an acquisition and amplification device, acquiring and amplifying the brain electrical signals.
Constructing a three-level self-encoder with output class feedback, as shown in fig. 4;
taking the preprocessed electroencephalogram signal X as input to extract first-stage source characteristics h 1 ＝φ(W 1 X+b 1 ) Constructing a primary self-encoder; so that
Figure BDA0002400096490000061
W 1 、/>
Figure BDA0002400096490000062
B is the first level coding and decoding weight value 1 、/>
Figure BDA0002400096490000063
For the first level codec bias term,
Figure BDA0002400096490000064
weights and bias terms can be obtained by pre-training for the neuron activation function.
With first-level source characteristics h 1 As input, extract the second level source features h 2 ＝φ(W 2 h 1 +b 2 ) Constructing a two-stage self-encoder to obtain a reconstructed input signal
Figure BDA0002400096490000065
Make->
Figure BDA0002400096490000066
W 2 、/>
Figure BDA0002400096490000067
B is the second level coding and decoding weight 2 、/>
Figure BDA0002400096490000068
For the second level codec bias term, +.>
Figure BDA0002400096490000069
Weights and bias terms can be obtained by pre-training for the neuron activation function.
With second-level source characteristics h 2 As input, correspond to
Figure BDA00024000964900000610
For outputting category [10 ]]Representing non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output by softmax layer +.>
Figure BDA00024000964900000611
Introducing output class y calculation errors
Figure BDA00024000964900000612
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount
Figure BDA00024000964900000613
And pre-training the self-encoder to obtain a trained self-encoder.
Random initialization
Figure BDA00024000964900000614
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure BDA00024000964900000615
h 0 And (3) taking the =x, j=1, 2,3 and epsilon as the learning rate, taking the updated weight and bias term as the weight and bias term of the next input signal, and sequentially performing encoder pre-training on all signals.
S102, preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifact removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference.
S103, extracting the preprocessed electroencephalogram signals by using a trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed brain electrical signals as input and takes the brain electrical signal source characteristics as output.
S104, performing brain control character spelling recognition according to the brain electrical signal source characteristics to obtain characters to be spelled.
Sending the character to be spelled to a stimulus interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square block which moves rapidly from left to right is arranged below each character and in the center of the nine square, and is shown in figure 3.
2-level presentation is carried out through a stimulation interface, the time of one-time motion stimulation is 400ms, the rapid motion square is marked as 1 in the current stimulation test time, the rapid motion square is marked as 0 in the no rapid motion square, and then i times of rapid motion stimulation can be marked as 2 at the same time i The target output is carried out, the intention of a user to select a character is divided into 2 steps for identification, namely, a block code is firstly identified, then a character code is identified, wherein the block code adopts 3 bits to realize 5-group output, the character code adopts 3 bits to realize 8-character output, each character selection process carries out identification output by referring to FIG. 3, one-time quick motion stimulation is ended, all motion blocks disappear for 100ms, the next stimulation test start mark is used, and a stimulation presentation program can be written through psychrolbox.
The method comprises the steps that a user sits right in front of a display, eyes look at a screen, after character spelling is started, sliding blocks in the stimulated display interface are displayed according to a block code rule shown in a table, then the character code rule is displayed, a character to be selected can be marked in the moving process of the block from left to right for 6 times, the user marks the character to be selected and outputs, taking a character N as an example, the block code sliding blocks slide rightwards, are static and static in sequence in 3 display stages, the block codes are marked as 1, 0 and 0, the character code sliding blocks slide, are static and slide in sequence in 3 display stages, and the character codes are marked as 1, 0 and 1. The table one is as follows:
list one
Figure BDA0002400096490000081
Fig. 5 is a schematic structural diagram of a brain-controlled character spelling recognition system based on supervised self-coding, as shown in fig. 5, the brain-controlled character spelling recognition system based on supervised self-coding provided by the invention comprises: an electroencephalogram signal acquisition module 501, a preprocessing module 502, an electroencephalogram signal source characteristic determination module 503 and a character determination module 504 to be spelled,
the electroencephalogram signal acquisition module 501 is used for acquiring an electroencephalogram signal of a user;
the preprocessing module 502 is configured to preprocess the electroencephalogram signal; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifacts removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module 503 is used for extracting the preprocessed electroencephalogram signal by using the trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signals as input and takes the electroencephalogram signal source characteristics as output;
the to-be-spelled character determining module 504 is configured to perform recognition of brain-controlled character spelling according to the electroencephalogram signal source characteristic, so as to obtain a to-be-spelled character.
The invention provides a brain-controlled character spelling recognition system based on supervision self-coding, which further comprises: the system comprises a three-stage self-encoder construction module with output category feedback, a self-encoder training module and a spelling correction module.
The three-level self-encoder construction module with output type feedback is used for constructing the three-level self-encoder with output type feedback;
the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder.
The spelling correction module is used for sending the character to be spelled to the stimulation interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square which moves rapidly from left to right is arranged below each character and in the center of the nine square.
The three-stage self-encoder construction module with output type feedback specifically comprises: the device comprises a primary self-encoder building unit, a secondary self-encoder building unit, a pre-training unit and a tertiary self-encoder building unit with output category feedback.
First-order self-coding constructionThe unit is used for taking the preprocessed electroencephalogram signal X as input to extract the first-stage source characteristic h 1 ＝φ(W 1 X+b 1 ) Then reconstruct the input signal
Figure BDA0002400096490000091
Make->
Figure BDA0002400096490000092
W 1 、/>
Figure BDA0002400096490000093
B is the first level coding and decoding weight value 1 、/>
Figure BDA0002400096490000094
For the first level codec bias term, +.>
Figure BDA0002400096490000095
Weights and bias terms can be obtained by pre-training for the neuron activation function.
The secondary self-coding construction unit is used for generating a first-level source characteristic h 1 As input, extract the second level source features h 2 ＝φ(W 2 h 1 +b 2 ) Then reconstruct the input signal
Figure BDA0002400096490000096
Make->
Figure BDA0002400096490000097
W 2 、/>
Figure BDA0002400096490000098
B is the second level coding and decoding weight 2 、
Figure BDA0002400096490000099
For the second level codec bias term, +.>
Figure BDA00024000964900000910
For neuron activation function, weights andthe bias term may be obtained through pre-training.
Pre-training unit random initialization
Figure BDA00024000964900000911
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure BDA00024000964900000912
Figure BDA00024000964900000913
h 0 And (3) taking the =x, j=1, 2,3 and epsilon as the learning rate, taking the updated weight and bias term as the weight and bias term of the next input signal, and sequentially performing encoder pre-training on all signals.
Three-level self-encoder building block with output class feedback for generating a second level source signature h 2 As input, correspond to
Figure BDA00024000964900000914
For outputting category [10 ]]Representing non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output through softmax layers
Figure BDA00024000964900000915
Introducing an output class y calculation error->
Figure BDA00024000964900000916
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount
Figure BDA00024000964900000917
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims (4)

1. A brain-controlled character spelling recognition method based on supervised self-encoding, comprising:
acquiring an electroencephalogram signal of a user;
preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifacts removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
extracting the preprocessed electroencephalogram signals by using a trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signals as input and takes the electroencephalogram signal source characteristics as output;
performing brain control character spelling recognition according to the brain electrical signal source characteristics to obtain characters to be spelled;
the method comprises the steps of extracting the preprocessed electroencephalogram signals by using a trained self-encoder to obtain electroencephalogram signal source characteristics, and further comprises the following steps:
constructing a three-level self-encoder with output category feedback;
training the self-encoder to obtain a trained self-encoder;
the self-encoder for constructing three-level output type feedback comprises the following specific steps:
to be pre-arrangedThe processed electroencephalogram signal X is taken as input to extract first-stage source characteristics h 1 ＝φ(W 1 X+b 1 ) Constructing a primary self-encoder; so that
Figure FDA0004082569860000011
W 1 、/>
Figure FDA0004082569860000012
B is the first level coding and decoding weight value 1 、/>
Figure FDA0004082569860000013
For the first level codec bias term,
Figure FDA0004082569860000014
the weights and bias terms can be obtained through pre-training for the neuron activation function;
with first-level source characteristics h 1 As input, extract the second level source features h 2 ＝φ(W 2 h 1 +b 2 ) Constructing a two-stage self-encoder to obtain a reconstructed input signal
Figure FDA0004082569860000015
Make->
Figure FDA0004082569860000016
W 2 、/>
Figure FDA0004082569860000017
B is the second level coding and decoding weight 2 、/>
Figure FDA0004082569860000018
For the second level codec bias term, +.>
Figure FDA0004082569860000019
The weights and bias terms can be obtained through pre-training for the neuron activation function;
random initialization
Figure FDA00040825698600000110
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure FDA00040825698600000111
Figure FDA0004082569860000021
h 0 The method comprises the steps that (1) X, j=1, 2,3 and epsilon are learning rates, updated weights and bias items are used as weights and bias items of the next input signals, and encoder pre-training is carried out on all signals in sequence;
with second-level source characteristics h 2 As input, correspond to
Figure FDA0004082569860000022
For outputting category [10 ]]Representing non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output by softmax layer +.>
Figure FDA0004082569860000023
Introducing output class y calculation errors
Figure FDA0004082569860000024
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount
Figure FDA0004082569860000025
2. The brain-controlled character spelling recognition method based on supervised self coding according to claim 1, wherein the brain-controlled character spelling recognition according to the brain-controlled signal source characteristics is performed to obtain the character to be spelled, and then the method further comprises:
sending the character to be spelled to a stimulus interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square which moves rapidly from left to right is arranged below each character and in the center of the nine square.
3. A supervised self-encoding based brain-controlled character spelling recognition system, comprising:
the electroencephalogram signal acquisition module is used for acquiring electroencephalogram signals of a user;
the preprocessing module is used for preprocessing the electroencephalogram signals; the pretreatment comprises bilateral mastoid reference electrode conversion, 0.5-40Hz band-pass filtering and ocular artifacts removal, segmentation is carried out according to 0 ms-400 ms of stimulus presentation time, and baseline correction is carried out by taking-100 ms-0 ms as a reference;
the electroencephalogram signal source characteristic determining module is used for extracting the preprocessed electroencephalogram signals by using the trained self-encoder to obtain electroencephalogram signal source characteristics; the trained self-encoder takes the preprocessed electroencephalogram signals as input and takes the electroencephalogram signal source characteristics as output;
the character determining module to be spelled is used for carrying out recognition of brain-controlled character spelling according to the brain-controlled character source characteristics to obtain characters to be spelled;
the three-level self-encoder construction module with output type feedback is used for constructing the three-level self-encoder with output type feedback;
the self-encoder training module is used for training the self-encoder to obtain a trained self-encoder;
the three-stage self-encoder construction module with output type feedback specifically comprises:
a first-stage self-encoder construction unit for constructing a first-stage self-encoder by taking the preprocessed electroencephalogram signal X as an input, and extracting a firstStage source signature h 1 ＝φ(W 1 X+b 1 ) Then reconstruct the input signal
Figure FDA0004082569860000031
Make->
Figure FDA0004082569860000032
W 1 、/>
Figure FDA0004082569860000033
B is the first level coding and decoding weight value 1 、/>
Figure FDA0004082569860000034
For the first level codec bias term, +.>
Figure FDA0004082569860000035
The weights and bias terms can be obtained through pre-training for the neuron activation function;
a two-level self-encoder construction unit for generating a first-level source characteristic h 1 As input, a two-level self-encoder is constructed, and a second-level source characteristic h is extracted 2 ＝φ(W 2 h 1 +b 2 ) Then reconstruct the input signal
Figure FDA0004082569860000036
Make->
Figure FDA0004082569860000037
W 2 、/>
Figure FDA0004082569860000038
B is the second level coding and decoding weight 2 、/>
Figure FDA0004082569860000039
For the second level codec bias term, +.>
Figure FDA00040825698600000310
The weights and bias terms can be obtained through pre-training for the neuron activation function;
a pre-training unit for randomly initializing
Figure FDA00040825698600000311
T represents the transpose, and then calculates the update amount from each input signal and the reconstruction signal>
Figure FDA00040825698600000312
Figure FDA00040825698600000313
h 0 The method comprises the steps that (1) X, j=1, 2,3 and epsilon are learning rates, updated weights and bias items are used as weights and bias items of the next input signals, and encoder pre-training is carried out on all signals in sequence;
three-level self-encoder building block with output class feedback for generating a second level source signature h 2 As input, correspond to
Figure FDA00040825698600000314
For outputting category [10 ]]Representing non-target, [01 ]]Representative target, extracting third-level source characteristic h 3 ＝φ(W 3 h 2 +b 3 ) Constructing a three-level self-encoder with output type feedback; the third level source characteristic h 3 Predicting output through softmax layers
Figure FDA00040825698600000315
Introducing an output class y calculation error->
Figure FDA00040825698600000316
Layer-by-layer fine tuning of weights and bias terms using gradient descent, each level of weights and bias terms fine tuning the update amount +.>
Figure FDA00040825698600000317
4. A supervised self coding based brain controlled character spelling recognition system of claim 3 further comprising:
the spelling correction module is used for sending the character to be spelled to a stimulus interface for spelling correction; the stimulation interface comprises 26 English characters of A-Z, 10 digital characters of 0-9, commas, periods, spaces and stimulation interfaces for returning 4 functional characters, wherein the stimulation interfaces are divided into 5 groups, 8 characters in each group are arranged according to the square periphery of a nine square, and a square which moves rapidly from left to right is arranged below each character and in the center of the nine square.
CN202010144039.6A 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding Active CN111338482B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010144039.6A CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010144039.6A CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Publications (2)

Publication Number Publication Date
CN111338482A CN111338482A (en) 2020-06-26
CN111338482B true CN111338482B (en) 2023-04-25

Family

ID=71182048

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010144039.6A Active CN111338482B (en) 2020-03-04 2020-03-04 Brain-controlled character spelling recognition method and system based on supervision self-coding

Country Status (1)

Country Link
CN (1) CN111338482B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114167989B (en) * 2021-12-09 2023-04-07 太原理工大学 Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008009632A (en) * 2006-06-28 2008-01-17 Denso Corp Character input device
CN103488297A (en) * 2013-09-30 2014-01-01 华南理工大学 Online semi-supervising character input system and method based on brain-computer interface
CN104978035A (en) * 2015-07-27 2015-10-14 中国医学科学院生物医学工程研究所 Brain computer interface system evoking P300 based on somatosensory electrical stimulation and implementation method thereof
WO2015192610A1 (en) * 2014-06-17 2015-12-23 华南理工大学 Intelligent wheel chair control method based on brain computer interface and automatic driving technology
CN105512681A (en) * 2015-12-07 2016-04-20 北京信息科技大学 Method and system for acquiring target category picture
CN108537133A (en) * 2018-03-16 2018-09-14 江苏经贸职业技术学院 A kind of face reconstructing method based on supervised learning depth self-encoding encoder
CN108921141A (en) * 2018-08-16 2018-11-30 广东工业大学 A kind of EEG signals EEG feature extracting method encoding neural network certainly based on depth
CN109222966A (en) * 2018-10-09 2019-01-18 山东大学 A kind of EEG signals sensibility classification method based on variation self-encoding encoder
CN110353672A (en) * 2019-07-15 2019-10-22 西安邮电大学 Eye artefact removal system and minimizing technology in a kind of EEG signals

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070091265A1 (en) * 2002-01-18 2007-04-26 Kardon Randy H System and method for optical imaging of human retinal function
US7546158B2 (en) * 2003-06-05 2009-06-09 The Regents Of The University Of California Communication methods based on brain computer interfaces
US7519452B2 (en) * 2004-04-15 2009-04-14 Neurosciences Research Foundation, Inc. Mobile brain-based device for use in a real world environment
AU2008212131A1 (en) * 2007-02-09 2008-08-14 Agency For Science, Technology And Research A system and method for processing brain signals in a BCI system
EP1975809A1 (en) * 2007-03-30 2008-10-01 Research In Motion Limited Use of multiple data sources for spell check function, and associated handheld electronic device
US8849727B2 (en) * 2008-05-26 2014-09-30 Agency For Science, Technology And Research Method and system for classifying brain signals in a BCI using a subject-specific model
US20120323967A1 (en) * 2011-06-14 2012-12-20 Microsoft Corporation Spelling Using a Fuzzy Pattern Search
KR101293863B1 (en) * 2011-06-29 2013-08-16 한양대학교 산학협력단 Word input interface apparatus and method with a qwerty type using steady state visual evoked potential
JP2015148945A (en) * 2014-02-06 2015-08-20 国立大学法人名古屋大学 Character input device, and program
CN106716396A (en) * 2014-07-15 2017-05-24 艾玛尔·Y·塞维尔公司 Input of characters of a symbol-based written language
CN105468143A (en) * 2015-11-17 2016-04-06 天津大学 Feedback system based on motor imagery brain-computer interface
CN105512534A (en) * 2015-12-31 2016-04-20 湖南人文科技学院 Self-destroyable identity recognition system based on combined authentication of brain wave, iris and human face recognition
WO2018116248A1 (en) * 2016-12-21 2018-06-28 Innereye Ltd. System and method for iterative classification using neurophysiological signals
CN107038421A (en) * 2017-04-17 2017-08-11 杭州电子科技大学 Modulation Types recognition methods based on sparse storehouse own coding
CN107122050B (en) * 2017-04-26 2019-05-21 西安交通大学 Stable state of motion visual evoked potential brain-computer interface method based on CSFL-GDBN
CN107481359A (en) * 2017-07-14 2017-12-15 昆明理工大学 Intelligent door lock system and its control method based on P300 and Mental imagery
CN109472194B (en) * 2018-09-26 2022-02-11 重庆邮电大学 Motor imagery electroencephalogram signal feature identification method based on CBLSTM algorithm model
RU2704497C1 (en) * 2019-03-05 2019-10-29 Общество с ограниченной ответственностью "Нейроботикс" Method for forming brain-computer control system
CN110123313B (en) * 2019-04-17 2022-02-08 中国科学院深圳先进技术研究院 Self-training brain-computer interface system and related training method
CN110347242A (en) * 2019-05-29 2019-10-18 长春理工大学 Audio visual brain-computer interface spelling system and its method based on space and semantic congruence
CN110364255B (en) * 2019-06-06 2024-03-22 江苏大学 Liver disease assessment method based on self-encoder
CN110262658B (en) * 2019-06-14 2023-02-03 长春理工大学 Brain-computer interface character input system based on enhanced attention and implementation method
CN110688013A (en) * 2019-10-11 2020-01-14 南京邮电大学 English keyboard spelling system and method based on SSVEP

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008009632A (en) * 2006-06-28 2008-01-17 Denso Corp Character input device
CN103488297A (en) * 2013-09-30 2014-01-01 华南理工大学 Online semi-supervising character input system and method based on brain-computer interface
WO2015192610A1 (en) * 2014-06-17 2015-12-23 华南理工大学 Intelligent wheel chair control method based on brain computer interface and automatic driving technology
CN104978035A (en) * 2015-07-27 2015-10-14 中国医学科学院生物医学工程研究所 Brain computer interface system evoking P300 based on somatosensory electrical stimulation and implementation method thereof
CN105512681A (en) * 2015-12-07 2016-04-20 北京信息科技大学 Method and system for acquiring target category picture
CN108537133A (en) * 2018-03-16 2018-09-14 江苏经贸职业技术学院 A kind of face reconstructing method based on supervised learning depth self-encoding encoder
CN108921141A (en) * 2018-08-16 2018-11-30 广东工业大学 A kind of EEG signals EEG feature extracting method encoding neural network certainly based on depth
CN109222966A (en) * 2018-10-09 2019-01-18 山东大学 A kind of EEG signals sensibility classification method based on variation self-encoding encoder
CN110353672A (en) * 2019-07-15 2019-10-22 西安邮电大学 Eye artefact removal system and minimizing technology in a kind of EEG signals

Also Published As

Publication number Publication date
CN111338482A (en) 2020-06-26

Similar Documents

Publication Publication Date Title
West et al. Complex webs: anticipating the improbable
CN108804715A (en) Merge multitask coordinated recognition methods and the system of audiovisual perception
CN107194426A (en) A kind of image-recognizing method based on Spiking neutral nets
CN109086869A (en) A kind of human action prediction technique based on attention mechanism
CN106447042B (en) Psychological analysis method and device based on drawing projection
Phelps et al. Neural networks predict response biases of female tungara frogs
CN111338482B (en) Brain-controlled character spelling recognition method and system based on supervision self-coding
CN114145745B (en) Graph-based multitasking self-supervision emotion recognition method
CN113128353B (en) Emotion perception method and system oriented to natural man-machine interaction
Case et al. Differentiation, integration, and covariance mapping as fundamental processes in cognitive and neurological growth
CN116596046A (en) Method for reconstructing image by utilizing electroencephalogram signals and visual features
Han et al. An eighty-target high-speed Chinese BCI speller
Du et al. InceptionSSVEP: a multi-scale convolutional neural network for steady-state visual evoked potential classification
Carpenter Some neglected contributions of Wilhelm Wundt to the psychology of memory
Bhattacharya et al. Optimization of EEG-based imaginary motion classification using majority-voting
CN113408444A (en) Event-related potential signal classification method based on CNN-SVM
Zhou et al. CyberPsychological computation on social community of ubiquitous learning
Benaroch Contribution to the understanding of mental task BCI performances using predictive computational models
CN114167989B (en) Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding
Dotan Syntax-driven number reading: The identification of digits is dominated by the number’s syntactic structure
KR101959049B1 (en) Smart Keyboard Using in BCI System and Input Method for the Same
CN117575746B (en) Virtual try-on method and device, electronic equipment and storage medium
Pacheco et al. Evaluation of different parents selection methods in a genetic algorithm wrapper for P300 BCI
Bai et al. GA-XGBoost Driving Distraction State Classification Model Based on Non-Driving Related Tasks
Park et al. Neurocomputational principles underlying the number sense

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant