CN106951081B - implementation method of brain-controlled speech generator based on P300 - Google Patents

Abstract

the invention relates to a method for realizing a P300-based brain-controlled speech sounder, which is used for decoding a sentence spelled by a P300Speller and playing the sentence by a speech sounder to realize the process that a user directly completes speaking through the brain; the method mainly comprises the following steps: the user spells the character sequence in sequence through the P300Speller, and certain minor characters can be omitted in the spelling process until a complete sentence is spelled; correcting the spelled character sequence by using a decoding algorithm to obtain a correct sentence; the correct sentence is then transmitted to the speech generator. The method provided by the invention can improve the speed of spelling sentences by the P300Speller and realize the function of directly speaking by the brain.

Description

Implementation method of brain-controlled speech generator based on P300

Technical Field

the invention belongs to the application of combining a brain-computer interface and natural language processing, and relates to a method for spelling sentences based on P300 and realizing brain speaking through a speech device.

background

The brain-computer interface provides a way for some patients with motor nerve damage and brain function damage to communicate with the outside world, wherein the P300Speller analyzes brain electrical signals through a series of stimulations to the brain, and identifies characters which the user wants to spell to achieve communication with the outside world. Currently, the spelling of a sentence by the P300Speller can only be spelled one by one for characters, and the user can only correct the spelling by himself when an error occurs. There are problems in that it takes a long time to spell a sentence, users are fatigued easily, and the spelling effect is not good.

Disclosure of Invention

Accordingly, the present invention is directed to increasing the speed of spelling a sentence by a user using a P300Speller and increasing the efficiency of communication between the user and the outside. In the invention, a user can omit certain minor characters in the spelling process and does not correct errors by himself, the spelling character sequence is corrected by a decoding algorithm, and the obtained correct sentence is transmitted to a voice generator.

The invention is realized by adopting the following scheme: a realization method of a brain-controlled speech generator based on P300 comprises the following steps:

Step S1: user spells the Sentence sequence c by the P300 spelling matrix₁c₂,…,c_nThe P300 spelling matrix comprises letters A-Z, 36 characters in total from 0 to 9, c_iI is 1, … n is the character in the P300 spelling matrix;

step S2: correcting the sequence, inserting the character which is missed to be input into the sequence, and correcting the error character to obtain a new Sentence C _ sequence;

step S3: and transmitting the C _ Sennce to a voice generator and playing.

further, the step S2 specifically includes the following steps:

step S21: setting structural variables Cur, cur.sen ═ sequence, cur.loc ═ 1, cur.len ═ length (sequence); initializing a stack S, listing a table L, and pressing Cur into the stack S;

Step S22: if the stack S is not empty, popping the stack to update Cur, and turning to the next step; otherwise, go to step S26;

Step S23: judging whether a character is to be inserted into the Cur.loc position; if so, ins.sen ═ Insert (cur.sen, cur.loc), ins.loc ═ cur.loc +1, ins.len ═ cur.len +1, Ins is pushed into the stack S;

Step S24: correcting a character at a position of cur.loc, cur.sen ═ modification (cur.sen, cur.loc); cor. loc ═ cur. loc + 1;

step S25: if Cur.loc is larger than Cur.len, inserting Cur into a table L, otherwise, pressing Cur into a stack S; proceed to step S22;

step S26: the probabilities of all sentences in the table L are calculated using the word language model, and the Sentence C _ sequence with the highest probability is output.

Further, the specific method for determining whether to Insert a character and an Insert (cur.sen, cur.loc) in the cur.loc position in step S23 is as follows:

Taking the Cur.loc position as the center, taking a character subsequence from Cur.senIs denoted by c₁c₂…c_k(ii) a At c₁c₂…c_kInserting character c at the position corresponding to Cur_i,c_ie C, C contains the space character and all the characters in the P300 spelling matrix, resulting in C₁c₂…c_i...c_k+1(ii) a Computing c with a 5-gram character language model₁c₂...c_kAnd c₁c₂…c_i...c_k+1,c_iE probability of C, from C₁c₂…c_i…c_k+1,c_ie C, selecting the character sequence with the highest probability, and comparing it with C₁c₂...c_kIf the probability is higher, inserting the character;

when a character is to be inserted, Insert (cur. sen, cur. loc) is inserted at the cur.loc position of the cur.sen character sequence so that c.sen₁c₂...c_i...c_k+1,c_ie C the one with the highest probability of C_i。

further, the specific method for correcting the character at the cut.loc position, modify (cut.sen, cut.loc) in step S24 is as follows:

Selecting a plurality of characters with high possibility to be input actually to form a character set according to the character of Cur.sen at the Cur.loc position and the P300 spelling matrix probability modelLet i ═ curWherein c is_lsen in the original character of the l position, c_l'is a character corrected in the l position by Cur.sen, P (c'_l|c_l) Taken from the P300 spelling matrix probability model, if c_lis an inserted space, then P (c'_l|c_l) Taking 1; c. C₁c₂...c_i...c_nSen or its corrected result, P (c)₁c₂...c_i...c_n) According to 5-gram character languageFor model calculation, α is a scale factor; is calculated to obtain c_bBy c_breplacement of c in Cur_ias output of modify (cur. sen, cur. loc).

Further, the specific method for calculating the probability of the sentence in step S26 is as follows:

sen in the sentence of the table L are read, the space is used as a separator to separate the words, and the words are sequentially stored in the w_iI 1.. m, then the probability of the sentence is calculated using a 3-gram word language model, the formula is as follows,

Wherein C (w)_i-2w_i-1w_i) And C (w)_i-2w_i-1) Are respectively words w_i-2w_{i- 1}w_iAnd w_i-2w_i-1Number of occurrences in the corpus.

Further, according to the character of Cur.sen at the Cur.loc position and the probability model of the P300 spelling matrix, selecting a plurality of characters with high possibility to be input actually to form a character setand P (c'_l|c_l) The specific method is taken from a P300 spelling matrix probability model and comprises the following steps:

The user carries out P300 spelling training before using the user, and a P300 spelling matrix probability model is obtained through calculation and is represented as a matrix A; element a in A_ij＝P(c_j|c_i)，c_iSpelling the resulting character for the user, c_jFor the character to be spelled actually, P (c)_j|c_i) The character obtained when spelling is c_iwhen the character actually intended to be spelled is c_jthe probability of (a) of (b) being,c_i,c_j∈{'A','B',...,'Z','0',...,'9'}，i＝1,2,...,36,j＝1,2,..36；

for the character of Cur.sen at the Cur.loc position, querying the row corresponding to the matrix A to obtain the characters with high possibility of actually spelling;

P(c'_l|c_l) C in (1)_land c_l' are characters, which correspond to the rows and columns of matrix A, respectively, and the corresponding probabilities are extracted from A.

Further, the specific method for calculating by using the 5-gram character language model comprises the following steps:

5-gram character language model calculates any character sequence c₁c₂...c_nThe probability of (a) is determined by using,Wherein the content of the first and second substances,C(c₁...c_i-1c_i) And C (C)₁...c_i-1) Are respectively a character c₁...c_i-1c_iAnd c₁...c_i-1in the number of times the corpus is present,C(c_i-4...c_i-1c_i) And C (C)_i-4...c_i-1) Are respectively a character c_i-4...c_i-1c_iAnd c_i-4...c_i-1Number of occurrences in the corpus.

It is important to understand the needs and conditions of a patient with impaired motor and intact brain function, which requires a long time to spell a sentence with the P300 Speller. Therefore, compared with the prior art, the invention has the following advantages:

1. The invention can enable the user to omit some characters in the spelling process, reduce the work load of spelling and improve the spelling efficiency.

2. The invention adopts the decoding algorithm to correct the sentence spelled by the user, and improves the spelling rate of the sentence, thereby improving the communication speed with the outside.

3. the invention connects the spelled sentences through the voice equipment, more directly connects the user with the outside, and has strong practical application significance.

Drawings

FIG. 1 is a schematic illustration of the method flow of the present invention.

Detailed Description

The invention is further explained below with reference to the drawings and the embodiments.

The embodiment provides a method for implementing a brain-controlled speech generator based on P300, as shown in fig. 1, comprising the following steps:

Step S1: user spells the Sentence sequence c by the P300 spelling matrix₁c₂,…,c_nThe P300 spelling matrix comprises letters A-Z, 36 characters in total from 0 to 9, c_ii is 1, … n is the character in the P300 spelling matrix;

step S2: correcting the sequence, inserting the character which is missed to be input into the sequence, and correcting the error character to obtain a new Sentence C _ sequence;

Step S3: and transmitting the C _ Sennce to a voice generator and playing.

In this embodiment, step S2 specifically includes the following steps:

step S21: setting structural variables Cur, cur.sen ═ sequence, cur.loc ═ 1, cur.len ═ length (sequence); initializing a stack S, listing a table L, and pressing Cur into the stack S;

step S22: if the stack S is not empty, popping the stack to update Cur, and turning to the next step; otherwise, go to step S26;

step S23: judging whether a character is to be inserted into the Cur.loc position; if so, ins.sen ═ Insert (cur.sen, cur.loc), ins.loc ═ cur.loc +1, ins.len ═ cur.len +1, Ins is pushed into the stack S;

Step S24: correcting a character at a position of cur.loc, cur.sen ═ modification (cur.sen, cur.loc); cor. loc ═ cur. loc + 1;

step S25: if Cur.loc is larger than Cur.len, inserting Cur into a table L, otherwise, pressing Cur into a stack S; proceed to step S22;

step S26: the probabilities of all sentences in the table L are calculated using the word language model, and the Sentence C _ sequence with the highest probability is output.

In this embodiment, the specific method for determining whether to Insert a character and an Insert (cur.sen, cur.loc) in the cur.loc position in step S23 is as follows:

Taking the Cur.loc position as the center, taking a character subsequence out of Cur.sen and marking as c₁c₂…c_k(ii) a At c₁c₂...c_kInserting character c at the position corresponding to Cur_i,c_iE C, C contains the space character and all the characters in the P300 spelling matrix, resulting in C₁c₂…c_i…c_k+1(ii) a Computing c with a 5-gram character language model₁c₂...c_kAnd c₁c₂...c_i...c_k+1,c_ie probability of C, from C₁c₂...c_i...c_k+1,c_iE C, selecting the character sequence with the highest probability, and comparing it with C₁c₂...c_kIf the probability is higher, inserting the character;

When a character is to be inserted, Insert (cur. sen, cur. loc) is inserted at the cur.loc position of the cur.sen character sequence so that c.sen₁c₂...c_i...c_k+1,c_iE C the one with the highest probability of C_i。

In this embodiment, the specific method for correcting the character at the cut.loc position, modify (cut.sen, cut.loc) in step S24 is as follows:

Selecting a plurality of characters with high possibility to be input actually to form a character set according to the character of Cur.sen at the Cur.loc position and the P300 spelling matrix probability modelLet i ═ curWherein c is_lsen in the original character of the l position, c_l'is a character corrected in the l position by Cur.sen, P (c'_l|c_l) Taken from the P300 spelling matrix probability model, if c_lIs an inserted space, then P (c'_l|c_l) Taking 1; c. C₁c₂...c_i...c_nSen or its corrected result, P (c)₁c₂...c_i...c_n) Calculating according to a 5-gram character language model, wherein alpha is a scale factor; is calculated to obtain c_bBy c_bReplacement of c in Cur_iAs output of modify (cur. sen, cur. loc).

in this embodiment, the specific method for calculating the probability of the sentence in step S26 is as follows:

Sen in the sentence of the table L are read, the space is used as a separator to separate the words, and the words are sequentially stored in the w_iI 1.. m, then the probability of the sentence is calculated using a 3-gram word language model, the formula is as follows,

Wherein C (w)_i-2w_i-1w_i) And C (w)_i-2w_i-1) Are respectively words w_i-2w_{i- 1}w_iand w_i-2w_i-1Number of occurrences in the corpus.

In this embodiment, the characters with high possibility to be actually input are selected to form a character set according to the character of Cur.sen at the Cur.loc position and the probability model of the P300 spelling matrixAnd P (c'_l|c_l) The specific method is taken from a P300 spelling matrix probability model and comprises the following steps:

The user carries out P300 spelling training before using the user, and a P300 spelling matrix probability model is obtained through calculation and is represented as a matrix A; element a in A_ij＝P(c_j|c_i)，c_iSpelling the resulting character for the user, c_jFor the character to be spelled actually, P (c)_j|c_i) The character obtained when spelling is c_iWhen the character actually intended to be spelled is c_jThe probability of (a) of (b) being,c_i,c_j∈{'A','B',...,'Z','0',...,'9'}，i＝1,2,...,36,j＝1,2,..36；

For the character of Cur.sen at the Cur.loc position, querying the row corresponding to the matrix A to obtain the characters with high possibility of actually spelling;

P(c'_l|c_l) C in (1)_land c_l' are characters, which correspond to the rows and columns of matrix A, respectively, and the corresponding probabilities are extracted from A.

In this embodiment, the specific method for performing calculation by using the 5-gram character language model includes:

5-gram character language model calculates any character sequence c₁c₂...c_nThe probability of (a) is determined by using,Wherein the content of the first and second substances,C(c₁...c_i-1c_i) And C (C)₁...c_i-1) Are respectively a character c₁...c_i-1c_iAnd c₁...c_i-1In the number of times the corpus is present,C(c_i-4...c_i-1c_i) And C (C)_i-4...c_i-1) Are respectively a character c_i-4...c_i-1c_iAnd c_i-4...c_i-1Number of occurrences in the corpus.

In this embodiment, the specific method of step S3 is as follows:

The corrected Sentence C _ sequence is transmitted to the command line execution file espeak of the speech generator espeak.

In this embodiment, the P300 spelling matrix is adjustable, and its size and the included characters are not the core content of this patent.

In the present embodiment, the size of the matrix a in the P300 spelling matrix probability model is determined according to the size of the P300 spelling matrix.

The above description is only a preferred embodiment of the present invention, and all equivalent changes and modifications made in accordance with the claims of the present invention should be covered by the present invention.

Claims (6)

1. A realization method of a brain-controlled speech generator based on P300 is characterized in that: the method comprises the following steps:

Step S1: user spells the Sentence sequence c by the P300 spelling matrix₁c₂,…,c_nthe P300 spelling matrix comprises letters A-Z, 36 characters in total from 0 to 9, c_iI is 1, … n is the character in the P300 spelling matrix;

Step S2: correcting the sequence, inserting the character which is missed to be input into the sequence, and correcting the error character to obtain a new Sentence C _ sequence;

Step S3: transmitting the C _ Sennce to a voice generator and playing;

Wherein, the step S2 specifically includes the following steps:

Step S21: setting structural variables Cur, cur.sen ═ sequence, cur.loc ═ 1, cur.len ═ length (sequence); initializing a stack S, listing a table L, and pressing Cur into the stack S;

Step S22: if the stack S is not empty, popping the stack to update Cur, and turning to the next step; otherwise, go to step S26;

Step S23: judging whether a character is to be inserted into the Cur.loc position; if so, ins.sen ═ Insert (cur.sen, cur.loc), ins.loc ═ cur.loc +1, ins.len ═ cur.len +1, Ins is pushed into the stack S;

Step S24: correcting a character at a position of cur.loc, cur.sen ═ modification (cur.sen, cur.loc); cor. loc ═ cur. loc + 1;

Step S25: if Cur.loc is larger than Cur.len, inserting Cur into a table L, otherwise, pressing Cur into a stack S; proceed to step S22;

Step S26: the probabilities of all sentences in the table L are calculated using the word language model, and the Sentence C _ sequence with the highest probability is output.

2. The implementation method of the P300-based brain-controlled speech sound generator according to claim 1, wherein: the specific method for judging whether to Insert a character and an Insert (cur.sen, cur.loc) in the cur.loc position in step S23 includes:

Taking the Cur.loc position as the center, taking a character subsequence out of Cur.sen and marking as c₁c₂...c_k(ii) a At c₁c₂...c_kInserting character c at the position corresponding to Cur_i,c_iE C, C contains the space character and all the characters in the P300 spelling matrix, resulting in C₁c₂...c_i...c_k+1(ii) a Computing c with a 5-gram character language model₁c₂...c_kAnd c₁c₂...c_i...c_k+1,c_ie probability of C, from C₁c₂...c_i...c_k+1,c_iE C, selecting the character sequence with the highest probability, and comparing it with C₁c₂...c_kIf the probability is higher, inserting the character;

When a character is to be inserted, Insert (cur. sen, cur. loc) is inserted at the cur.loc position of the cur.sen character sequence so that c.sen₁c₂...c_i...c_k+1,c_iE C the one with the highest probability of C_i。

3. the implementation method of the P300-based brain-controlled speech sound generator according to claim 1, wherein: the specific method for correcting the character at the cur.loc position, modify (cur.sen, cur.loc) described in step S24 is as follows:

Selecting a plurality of characters with high possibility to be input actually to form a character set according to the character of Cur.sen at the Cur.loc position and the P300 spelling matrix probability modelLet i ═ curwherein c is_lSen in the original character of the l position, c_l'is a character corrected in the l position by Cur.sen, P (c'_l|c_l) Taken from the P300 spelling matrix probability model, if c_lIs an inserted space, then P (c'_l|c_l) Taking 1; c. C₁c₂...c_i...c_nSen or its corrected result, P (c)₁c₂...c_i...c_n) Calculating according to a 5-gram character language model, wherein alpha is a scale factor; is calculated to obtain c_bBy c_bReplacement of c in Cur_iAs output of modify (cur. sen, cur. loc).

4. The implementation method of the P300-based brain-controlled speech sound generator according to claim 1, wherein: the specific method for calculating the probability of the sentence in step S26 is as follows:

Sen in the sentence of the table L are read, the space is used as a separator to separate the words, and the words are sequentially stored in the w_iI 1.. m, then the probability of the sentence is calculated using a 3-gram word language model, the formula is as follows,

wherein C (w)_i-2w_i-1w_i) And C (w)_i-2w_i-1) Are respectively words w_i-2w_i-1w_iAnd w_i-2w_i-1Number of occurrences in the corpus.

5. the implementation method of the P300-based brain-controlled speech generator according to claim 3, wherein: selecting a plurality of characters with high possibility to be input actually according to the character of Cur.sen at the Cur.loc position and the probability model of the P300 spelling matrix to form a character setAnd P (c'_l|c_l) The specific method is taken from a P300 spelling matrix probability model and comprises the following steps:

The user carries out P300 spelling training before using the user, and a P300 spelling matrix probability model is obtained through calculation and is represented as a matrix A; element a in A_ij＝P(c_j|c_i)，c_iSpelling the resulting character for the user, c_jFor the character to be spelled actually, P (c)_j|c_i) The character obtained when spelling is c_iWhen the character actually intended to be spelled is c_jThe probability of (a) of (b) being,

c_i,c_j∈{'A','B',...,'Z','0',...,'9'}，i＝1,2,...,36,j＝1,2,..36；

For the character of Cur.sen at the Cur.loc position, querying the row corresponding to the matrix A to obtain the characters with high possibility of actually spelling;

P(c′_l|c_l) C in (1)_lAnd c'_lall are characters, which can correspond to the rows and columns of the matrix A respectively, and the corresponding probabilities are taken out from A.

6. The method for realizing the P300-based brain-controlled speech generator according to claim 2 or 3, wherein: the specific method for calculating by adopting the 5-gram character language model comprises the following steps:

5-gram character language model calculates any character sequence c₁c₂...c_nthe probability of (a) is determined by using,Wherein the content of the first and second substances,C(c₁...c_i-1c_i) And C (C)₁...c_i-1) Are respectively a character c₁...c_i-1c_iAnd c₁...c_i-1In the number of times the corpus is present,C(c_i-4...c_i-1c_i) And C (C)_i-4...c_i-1) Are respectively a character c_i-4...c_i-1c_iAnd c_i-4...c_i-1number of occurrences in the corpus.