CN113792588B

CN113792588B - Brain wave processing device, brain wave processing method, computer equipment and storage medium

Info

Publication number: CN113792588B
Application number: CN202110895163.0A
Authority: CN
Inventors: 陈海波; 邓钢清
Original assignee: Deep Blue Technology Shanghai Co Ltd
Current assignee: Deep Blue Technology Shanghai Co Ltd
Priority date: 2021-08-05
Filing date: 2021-08-05
Publication date: 2024-04-09
Anticipated expiration: 2041-08-05
Also published as: CN113792588A

Abstract

The embodiment of the application discloses an electroencephalogram processing device, an electroencephalogram processing method, computer equipment and a storage medium, wherein the device comprises: a first encoder, a second encoder, a third encoder, and a decoder, wherein: the first encoder is used for carrying out data processing on the received multipath electroencephalogram sample signals and respectively outputting electroencephalogram semantic signals, wherein the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, and the second encoder is used for carrying out data processing on the received plurality of electroencephalogram semantic sub-signals and outputting static electroencephalogram semantic signals, and the static electroencephalogram semantic signals of each electroencephalogram semantic sub-signal are the same; the third encoder is used for carrying out data processing on the received plurality of electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals and outputting dynamic electroencephalogram semantic signals, wherein the dynamic electroencephalogram semantic signals comprise time information; and the decoder is used for reconstructing an electroencephalogram sample signal according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal.

Description

Brain wave processing device, brain wave processing method, computer equipment and storage medium

Technical Field

The application relates to the technical field of deep learning. And more particularly, to an electroencephalogram processing apparatus, method, computer device, and storage medium.

Background

In electroencephalogram signals, the classification problem is generally difficult to solve, because the intensity of the electroencephalogram signals of different classes is very similar, and the difference between the electroencephalogram signals is very small, so that the electroencephalogram signals are difficult to completely distinguish. In most brain-computer interface operations, for example, classification problems are critical for the electroencephalogram operation of a mechanical arm. The prior art usually resorts to external brain stimulation protocols like SSVEP, otherwise it is difficult to classify using conventional classifiers because the EEG data of different evoked potentials are very similar.

Disclosure of Invention

In view of this, the present application provides an electroencephalogram processing apparatus, method, computer device, and storage medium.

In a first aspect, the present application provides an electroencephalogram processing apparatus, the apparatus comprising:

a first encoder, a second encoder, a third encoder, and a decoder, wherein:

the first encoder is used for carrying out data processing on the received multipath electroencephalogram sample signals and respectively outputting electroencephalogram semantic signals, the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, and each electroencephalogram semantic sub-signal corresponds to different time information;

the second encoder is used for carrying out data processing on the received plurality of brain electrical semantic sub-signals and outputting static brain electrical semantic signals, and the static brain electrical semantic signals of each brain electrical semantic sub-signal are the same;

The third encoder is used for carrying out data processing on the received plurality of electroencephalogram semantic sub-signals and static electroencephalogram semantic signals and outputting dynamic electroencephalogram semantic signals, and the dynamic electroencephalogram semantic signals comprise the time information;

the decoder is used for reconstructing the electroencephalogram sample signal according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal.

In a specific embodiment, the first encoder includes a first input layer, a first convolution layer, a first flattening layer, a first fully-connected layer, a second fully-connected layer, and a first output layer, where

The first input layer transmits the received multi-path electroencephalogram sample signals to the first convolution layer, the multi-path electroencephalogram sample signals are output to the first flattening layer after being subjected to convolution calculation by the first convolution layer, the multi-path electroencephalogram sample signals are output to the first full-connection layer after being flattened by the first flattening layer, and the electroencephalogram semantic signals corresponding to the multi-path electroencephalogram sample signals are respectively output through the first output layer after being subjected to classification operation by the first full-connection layer and the second full-connection layer.

In a specific embodiment, the kernel size of the first convolution layer is 3*3, the step size of the first convolution layer is 2, and the filling mode of the first convolution layer is a complement mode.

In a specific embodiment, the second encoder includes a second input layer, a first two-way long and short-term memory network, a first splice layer, a third full-connection layer, and a first sampling layer, wherein

The second input layer receives a plurality of electroencephalogram semantic signals and transmits the electroencephalogram semantic signals to the first two-way long-short-term memory network, the electroencephalogram semantic signals are transmitted to the first splicing layer after being subjected to time-cycle processing operation by the first two-way long-short-term memory network, the electroencephalogram semantic signals are transmitted to the third full-connection layer after being spliced by the first splicing layer, and the static electroencephalogram semantic signals are output by the first sampling layer after being subjected to classification operation by the third full-connection layer.

In a specific embodiment, the first two-way long-term memory network comprises a first forward long-term memory network and a first reverse long-term memory network, wherein:

the first forward long-term and short-term memory network is used for performing forward data processing on the received plurality of electroencephalogram semantic sub-signals according to the time information, outputting forward static electroencephalogram semantic sub-signals and outputting the forward static electroencephalogram semantic sub-signals to the first splicing layer;

the first reverse long-short-term memory network is used for performing reverse data processing on the received plurality of electroencephalogram semantic sub-signals according to the time information, outputting reverse static electroencephalogram semantic sub-signals and outputting the reverse static electroencephalogram semantic sub-signals to the first splicing layer;

The first splicing layer splices according to the forward static brain electricity semantic sub-signals and the reverse static brain electricity semantic sub-signals and outputs static brain electricity semantic splicing signals.

In a specific embodiment, the third fully-connected layer comprises a first fully-connected sub-layer and a second fully-connected sub-layer, wherein

The first full-connection sub-layer is used for carrying out mean value processing on the static electroencephalogram semantic splicing signals and outputting static electroencephalogram semantic mean value signals to the first sampling layer;

the second full-connection sub-layer is used for performing variance processing on the static electroencephalogram semantic splicing signals and outputting static electroencephalogram semantic variance signals to the first sampling layer;

the first sampling layer outputs the static electroencephalogram semantic signal according to the static electroencephalogram semantic mean signal and the static electroencephalogram semantic variance signal.

In a specific embodiment, the third encoder includes a third input layer, a second bidirectional long short-term memory network, a second splice layer, an RNN layer, a fourth full connection layer, and a second sampling layer, wherein

The third input layer receives the plurality of electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals, transmits the static electroencephalogram semantic signals to the second bidirectional long-short-term memory network, transmits the static electroencephalogram semantic signals to the second splicing layer after time-loop processing operation is performed on the static electroencephalogram semantic signals by the second bidirectional long-short-term memory network, transmits the static electroencephalogram semantic signals to the RNN layer after time-information processing operation is performed on the RNN layer after splicing of the static electroencephalogram semantic signals by the second splicing layer, and transmits the static electroencephalogram semantic signals to the fourth full-connection layer after classification operation is performed on the static electroencephalogram semantic signals with time information by the fourth full-connection layer through the second sampling layer.

In one embodiment, the second bidirectional long short term memory network includes a second forward long term memory network and a second reverse long term memory network, wherein,

the second forward long-short term memory network is used for performing forward data processing on the received multiple electroencephalogram semantic sub-signals and static electroencephalogram semantic signals according to the time information, respectively outputting multiple forward dynamic electroencephalogram semantic sub-signals corresponding to different time information, and outputting the multiple forward dynamic electroencephalogram semantic sub-signals to the second splicing layer;

the second reverse long-short-term memory network is used for performing reverse data processing on the received multiple brain electrical semantic sub-signals and static brain electrical semantic signals according to the time information, respectively outputting multiple reverse dynamic brain electrical semantic sub-signals corresponding to different time information, and outputting the multiple reverse dynamic brain electrical semantic sub-signals to the second splicing layer;

the second splicing layer respectively splices the forward dynamic electroencephalogram semantic sub-signals and the reverse dynamic electroencephalogram semantic sub-signals corresponding to the same time information according to the received forward dynamic electroencephalogram semantic sub-signals and the received reverse dynamic electroencephalogram semantic sub-signals, and outputs a plurality of dynamic electroencephalogram semantic splicing signals corresponding to different time information to the RNN layer;

The RNN layer is used for respectively carrying out data processing on the received plurality of dynamic electroencephalogram semantic splicing signals and outputting a plurality of dynamic electroencephalogram semantic time sequence signals comprising the time information.

In a specific embodiment, the fourth full-connection layer includes a third full-connection sub-layer and a fourth full-connection sub-layer, wherein

The third full-connection sub-layer is used for carrying out mean value processing on the plurality of dynamic electroencephalogram semantic time sequence signals and outputting dynamic electroencephalogram semantic mean value signals to the second sampling layer;

the fourth full-connection sub-layer is used for performing variance processing on the plurality of dynamic electroencephalogram semantic time sequence signals and outputting dynamic electroencephalogram semantic variance signals to the second sampling layer;

the second sampling layer outputs the dynamic electroencephalogram semantic signal according to the dynamic electroencephalogram semantic mean signal and the dynamic electroencephalogram semantic variance signal.

In a specific embodiment, the decoder reconstructs the electroencephalogram sample signal through deconvolution operation according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal.

In a second aspect, the present application provides a brain wave processing method, including:

carrying out data processing on the received multipath electroencephalogram sample signals and respectively outputting electroencephalogram semantic signals, wherein the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, and each electroencephalogram semantic sub-signal corresponds to different time information;

Data processing is carried out on the plurality of received brain electricity semantic sub-signals, and static brain electricity semantic signals are output, wherein the static brain electricity semantic signals of each brain electricity semantic sub-signal are the same;

performing data processing on the received plurality of electroencephalogram semantic sub-signals and static electroencephalogram semantic signals and outputting dynamic electroencephalogram semantic signals, wherein the dynamic electroencephalogram semantic signals comprise the time information;

reconstructing the electroencephalogram sample signal according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal.

In a specific embodiment, the first encoder includes a first input layer, a first convolution layer, a first flattening layer, a first fully-connected layer, a second fully-connected layer, and a first output layer, where the first convolution layer has a kernel size of 3*3, the step size of the first convolution layer is 2, and the filling mode of the first convolution layer is a complement mode.

In a specific embodiment, the second encoder includes a second input layer, a first bidirectional long-short-term memory network, a first splicing layer, a third full-connection layer and a first sampling layer, the first bidirectional long-short-term memory network includes a first forward long-short-term memory network and a first reverse long-short-term memory network, the third full-connection layer includes a first full-connection sub-layer and a second full-connection sub-layer, and the data processing is performed on the received plurality of brain electrical semantic sub-signals and the static brain electrical semantic signals are output further including:

The first forward long-short-term memory network performs forward data processing on the received plurality of electroencephalogram semantic sub-signals according to the time information, outputs forward static electroencephalogram semantic sub-signals, and outputs the forward static electroencephalogram semantic sub-signals to the first splicing layer;

the first reverse long-short-term memory network performs reverse data processing on the received plurality of electroencephalogram semantic sub-signals according to the time information, outputs reverse static electroencephalogram semantic sub-signals and outputs the reverse static electroencephalogram semantic sub-signals to the first splicing layer;

the first splicing layer splices according to the forward static electroencephalogram semantic sub-signal and the reverse static electroencephalogram semantic sub-signal and outputs a static electroencephalogram semantic splicing signal;

the first full-connection sub-layer carries out mean value processing on the static electroencephalogram semantic splicing signals and outputs static electroencephalogram semantic mean value signals to the first sampling layer;

the second full-connection sub-layer carries out variance processing on the static electroencephalogram semantic splicing signals and outputs static electroencephalogram semantic variance signals to the first sampling layer;

In a specific embodiment, the third encoder includes a third input layer, a second bidirectional long-short-term memory network, a second splicing layer, an RNN layer, a fourth full-connection layer, and a second sampling layer, the second bidirectional long-short-term memory network includes a second forward long-short-term memory network and a second reverse long-short-term memory network, the fourth full-connection layer includes a third full-connection sub-layer and a fourth full-connection sub-layer, and the data processing is performed on the received plurality of electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals, and outputting dynamic electroencephalogram semantic signals further includes:

The second forward long-short-term memory network performs forward data processing on the received multiple electroencephalogram semantic sub-signals and static electroencephalogram semantic signals according to the time information, outputs multiple forward dynamic electroencephalogram semantic sub-signals corresponding to different time information respectively, and outputs the multiple forward dynamic electroencephalogram semantic sub-signals to the second splicing layer;

the second reverse long-short-term memory network performs reverse data processing on the received multiple brain electricity semantic sub-signals and static brain electricity semantic signals according to the time information, respectively outputs multiple reverse dynamic brain electricity semantic sub-signals corresponding to different time information, and outputs the multiple reverse dynamic brain electricity semantic sub-signals to the second splicing layer;

the RNN layer respectively carries out data processing on the received plurality of dynamic electroencephalogram semantic splicing signals and outputs a plurality of dynamic electroencephalogram semantic time sequence signals comprising the time information;

The third full-connection sub-layer carries out mean value processing on the plurality of dynamic electroencephalogram semantic time sequence signals and outputs dynamic electroencephalogram semantic mean value signals to the second sampling layer;

the fourth full-connection sub-layer carries out variance processing on the plurality of dynamic electroencephalogram semantic time sequence signals and outputs dynamic electroencephalogram semantic variance signals to the second sampling layer;

In a third aspect, the present application provides a computer readable storage medium having stored thereon a computer program, characterized in that the program when executed by a processor implements the method according to the second aspect.

In a fourth aspect, the present application provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to the second aspect when executing the program.

The beneficial effects of this application are as follows:

aiming at the existing problems at present, the application provides an electroencephalogram processing device, an electroencephalogram processing method, computer equipment and a storage medium, input electroencephalogram sample signals are subjected to data processing through a first encoder, electroencephalogram semantic sub-signals corresponding to time information are output, the same static electroencephalogram semantic signals in all the electroencephalogram semantic sub-signals are extracted through a second encoder, and finally noise in the electroencephalogram signals is separated from effective dynamic electroencephalogram semantic sub-signals according to the electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals from different angles through a third encoder, so that classification precision of electroencephalogram signals of different categories is effectively improved, quality of the electroencephalogram signals is enhanced, and the electroencephalogram processing device has wide application prospect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 shows a schematic structure of an electroencephalogram processing apparatus according to an embodiment of the present application.

Fig. 2 shows a waveform schematic of an electroencephalogram sample signal according to one embodiment of the present application.

Fig. 3 shows a schematic structural diagram of a first encoder according to an embodiment of the present application.

Fig. 4 shows a schematic structural diagram of a second encoder according to an embodiment of the present application.

Fig. 5 illustrates an input-output schematic diagram of a first forward long-short-term memory network according to one embodiment of the present application.

Fig. 6 illustrates an input-output schematic diagram of a first reverse long-short-term memory network according to one embodiment of the present application.

Fig. 7 shows a schematic structural diagram of a third encoder according to an embodiment of the present application.

Fig. 8 illustrates an input-output schematic diagram of a second bidirectional long-short term memory network according to one embodiment of the present application.

FIG. 9 shows a schematic diagram of a comparison of brain wave sample signals before and after separation according to one embodiment of the present application

Fig. 10 shows a flow diagram of a brain wave processing method according to an embodiment of the present application.

Fig. 11 shows a schematic structural diagram of a computer device suitable for use in the present application.

Detailed Description

For a clearer description of the present application, the present application is further described below with reference to preferred embodiments and the accompanying drawings. Like parts in the drawings are denoted by the same reference numerals. It is to be understood by persons skilled in the art that the following detailed description is intended to be illustrative, and not restrictive, and that this invention is not to be limited to the specific embodiments shown.

The traditional method for improving the electroencephalogram signal is denoising, and the electroencephalogram signal is divided into a plurality of different signals through a filter, so that noise caused by external environment and heartbeat is screened out, and the data without noise is classified. However, although the above method removes noise other than the electroencephalogram signals, there is still a great similarity between the remaining clean electroencephalogram signals, and it is very difficult to distinguish their categories.

To this end, an embodiment of the present application discloses an electroencephalogram processing apparatus including a first encoder, a second encoder, a third encoder, and a decoder, wherein:

The first encoder is used for carrying out data processing on the received multipath electroencephalogram sample signals and respectively outputting electroencephalogram semantic signals, the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, and each electroencephalogram semantic sub-signal corresponds to different time information; the second encoder is used for carrying out data processing on the received plurality of brain electrical semantic sub-signals and outputting static brain electrical semantic signals, and the static brain electrical semantic signals of each brain electrical semantic sub-signal are the same; the third encoder is used for carrying out data processing on the received plurality of electroencephalogram semantic sub-signals and static electroencephalogram semantic signals and outputting dynamic electroencephalogram semantic signals, and the dynamic electroencephalogram semantic signals comprise the time information; the decoder is used for reconstructing the electroencephalogram sample signal according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal.

Aiming at the existing problems, the embodiment provides an electroencephalogram processing device, input electroencephalogram sample signals are subjected to data processing through a first encoder, electroencephalogram semantic sub-signals corresponding to time information are output, the same static electroencephalogram semantic signals in all the electroencephalogram semantic sub-signals are extracted through a second encoder, finally noise in the electroencephalogram signals is separated from effective dynamic electroencephalogram semantic sub-signals from different angles through a third encoder according to the electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals, so that classification accuracy of the electroencephalogram signals of different types is effectively improved, quality of the electroencephalogram signals is enhanced, and the electroencephalogram processing device has wide application prospect.

In one embodiment, as shown in fig. 1, the brain wave processing device includes a first encoder 10, a second encoder 20, a third encoder 30, and a decoder 40, wherein,

the first encoder 10, also called a self-encoder, is configured to perform data processing on the received multi-path electroencephalogram sample signal datax shown in fig. 2, and output an electroencephalogram semantic signal VEC respectively, where the electroencephalogram semantic signal VEC includes a plurality of electroencephalogram semantic sub-signals, and each electroencephalogram semantic sub-signal corresponds to different time information.

For example, in one specific scenario, an apparatus with multiple needles is used to detect a brain of a human body, the apparatus transmits multiple paths of electroencephalogram sample signals data x detected by the multiple needles for a period of time to the first encoder 10, and the first encoder 10 outputs an electroencephalogram semantic signal VEC corresponding to the electroencephalogram sample signals, where the electroencephalogram semantic signal includes N sub-sample signals ordered according to a time sequence.

In this example, as shown in fig. 3, the first encoder 10 specifically includes a first input layer 100, a first convolution layer 102, a first leveling layer 104, a first fully-connected layer 106, a second fully-connected layer 108, and a first output layer 110, where the first input layer 100 transmits a received multi-path electroencephalogram sample signal to the first convolution layer 102, performs convolution calculation by the first convolution layer 102, outputs the multi-path electroencephalogram sample signal to the first leveling layer 104, performs flattening by the first leveling layer 104, outputs the multi-path electroencephalogram sample signal to the first fully-connected layer 106, performs classification operation by the first fully-connected layer 106 and the second fully-connected layer 108, and then outputs the electroencephalogram semantic signal corresponding to the multi-path electroencephalogram sample signal by the first output layer 110.

Specifically, after convolution processing, flattening the data by using the flat, and further using two layers of full connection to finally obtain the brain electrical semantic signal VEC. The kernel size of the first convolution layer 102 is 3*3, the step size is 2, and the filling mode is a complement mode (same).

In this embodiment, taking the electroencephalogram sample signal data x in 8 time slots as an example for illustration, if the electroencephalogram semantic signal of one time slot is set as one sub-sample, the electroencephalogram semantic signal VEC includes 8 sub-sample signals, which are denoted as V1-V8.

The second encoder 20 is also called a static encoder, and is configured to perform data processing on the received plurality of electroencephalogram semantic sub-signals and output a static electroencephalogram semantic signal F, where the static electroencephalogram semantic signal F of each electroencephalogram semantic sub-signal is the same. Wherein,

the second encoder 20 includes a two-way long short-term memory network (LSTM), and in one particular example, as shown in fig. 4, the second encoder 20 includes a second input layer 200, a first two-way long-term memory network 202, a first splice layer 204, a third full-connection layer 206, and a first sampling layer 208, wherein

The second input layer 200 receives the plurality of electroencephalogram semantic signals and transmits the signals to the first bidirectional long-short-term memory network 202, the signals are transmitted to the first splicing layer 204 after being subjected to time-cycle processing operation by the first bidirectional long-short-term memory network 202, the signals are transmitted to the third full-connection layer 206 after being spliced by the first splicing layer 204, and the static electroencephalogram semantic signals are output by the first sampling layer 208 after being subjected to classification operation by the third full-connection layer 206.

In one specific example, the first two-way long-term memory network includes a first forward long-term memory network and a first reverse long-term memory network, wherein:

the first forward long-short-term memory network is configured to perform forward data processing on the received multiple electroencephalogram semantic sub-signals according to the time information, output forward static electroencephalogram semantic sub-signals, and output the forward static electroencephalogram semantic sub-signals to the first splicing layer 204. For example, as shown in fig. 5, the brain electrical semantic sub-signals V1 to V8 are sequentially input to the first forward long-short-term memory network in the order from V1 to V8, thereby outputting the forward static brain electrical semantic sub-signal FY.

The first reverse long-short-term memory network is configured to perform reverse data processing on the received multiple brain electrical semantic sub-signals according to the time information, output a reverse static brain electrical semantic sub-signal, and output the reverse static brain electrical semantic sub-signal to the first splicing layer 204, for example, as shown in fig. 6, sequentially input brain electrical semantic sub-signals V1-V8 to the first reverse long-short-term memory network according to the sequence from V8 to V1, so as to output a reverse static brain electrical semantic sub-signal BY.

As shown in fig. 4, the first splicing layer 204 splices according to the forward static electroencephalogram semantic sub-signal FY and the reverse static electroencephalogram semantic sub-signal BY and outputs a static electroencephalogram semantic splicing signal Y.

In one specific example, as shown in fig. 4, the third fully-connected layer 206 includes a first fully-connected sub-layer and a second fully-connected sub-layer,

the first full-connection sub-layer is configured to perform mean processing on the static electroencephalogram semantic stitching signal Y and output a static electroencephalogram semantic mean signal to the first sampling layer 208;

the second full-connection sub-layer is configured to perform variance processing on the static electroencephalogram semantic stitching signal Y and output a static electroencephalogram semantic variance signal to the first sampling layer 208;

the first sampling layer outputs the static electroencephalogram semantic signal F according to the static electroencephalogram semantic mean signal and the static electroencephalogram semantic variance signal.

Thus, the static brain electricity semantic signal F of each sub-sample signal V1-V8 is obtained, and the common hidden semantics of V1-V8 are obtained. The same parts in the electroencephalogram signals are extracted through the second arranged encoder, namely the static encoder, the same parts are the same parts of the sub-sample signals V1-V8, namely the static electroencephalogram semantic signals of V1-V8 are the same, and the noise removal processing is carried out on the electroencephalogram signals to a certain extent by extracting the same parts, so that the classification precision of the electroencephalogram signals among different categories is improved.

The third encoder 30 is also called a dynamic encoder, and is configured to perform data processing on the received multiple electroencephalogram semantic sub-signals V1-V8 and the static electroencephalogram semantic signal F, and output a dynamic electroencephalogram semantic signal Z, where the dynamic electroencephalogram semantic signal includes the time information.

The third encoder 30 includes a two-way long short term memory network (LSTM) and a Recurrent Neural Network (RNN). In a specific example, as shown in fig. 7, the third encoder 30 includes a third input layer 300, a second bidirectional long short term memory network 302, a second splicing layer 304, an RNN layer 306, a fourth full connection layer 308, and a second sampling layer 310, wherein

The third input layer 300 receives the plurality of electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals, transmits the signals to the second bidirectional long-short term memory network 302, performs a time-loop processing operation through the second bidirectional long-short term memory network 302, transmits the signals to the second splicing layer 304, performs a time information processing operation through the second splicing layer 304, transmits the signals to the RNN layer 306, performs a time information processing operation through the RNN layer 306, transmits the signals to the fourth full-connection layer 308, performs a classification operation through the fourth full-connection layer 308, and outputs the dynamic electroencephalogram semantic signals with time information through the second sampling layer 310.

Wherein the second bidirectional long short term memory network 302 comprises a second forward long short term memory network and a second reverse long term memory network, wherein,

the second forward long-short term memory network is configured to perform forward data processing on the received multiple electroencephalogram semantic sub-signals and static electroencephalogram semantic signals according to the time information, output multiple forward dynamic electroencephalogram semantic sub-signals corresponding to different time information respectively, and output the multiple forward dynamic electroencephalogram semantic sub-signals to the second splicing layer 304. For example, as shown in fig. 8, the electroencephalogram semantic sub-signals V1 to V8 and the static electroencephalogram semantic signal F (not shown in fig. 8) are sequentially input to the second forward long-short-term memory network in order from V1 to V8, thereby outputting forward dynamic electroencephalogram semantic sub-signals FY1 to FY8.

The second reverse long-short term memory network is configured to perform reverse data processing on the received multiple brain electrical semantic sub-signals and static brain electrical semantic signals according to the time information, output multiple reverse dynamic brain electrical semantic sub-signals corresponding to different time information respectively, and output the multiple reverse dynamic brain electrical semantic sub-signals to the second splicing layer 304. For example, as shown in fig. 8, the brain electrical semantic sub-signals V1 to V8 and the static brain electrical semantic signal F (not shown in fig. 8) are sequentially input to the second reverse long-short-term memory network in the order from V8 to V1, thereby outputting the reverse dynamic brain electrical semantic sub-signals BY1 to BY8

The second splicing layer 304 respectively splices the forward dynamic electroencephalogram semantic sub-signal and the reverse dynamic electroencephalogram semantic sub-signal corresponding to the same time information according to the received forward dynamic electroencephalogram semantic sub-signals and the received reverse dynamic electroencephalogram semantic sub-signals, for example, as shown in fig. 8, splices FY1 and BY1 to form Y1, splices FY2 and BY2 to form Y2, and so on, and outputs a plurality of dynamic electroencephalogram semantic splicing signals Y1-Y8 corresponding to different time information to the RNN layer.

The RNN layer, as shown in fig. 7, performs data processing on the received multiple dynamic electroencephalogram semantic splicing signals Y1-Y8, and outputs multiple dynamic electroencephalogram semantic timing signals H1-H8 including the time information.

In a specific example, the fourth full-connection layer includes a third full-connection sub-layer and a fourth full-connection sub-layer, wherein

The third full-connection sub-layer is configured to perform mean processing on the plurality of dynamic electroencephalogram semantic time sequence signals H1-H8 and output a dynamic electroencephalogram semantic mean signal to the second sampling layer 310;

the fourth full-connection sub-layer is configured to perform variance processing on the plurality of dynamic electroencephalogram semantic timing signals H1-H8 and output a dynamic electroencephalogram semantic variance signal to the second sampling layer 310;

The second sampling layer outputs the dynamic electroencephalogram semantic signal Z according to the dynamic electroencephalogram semantic mean signal and the dynamic electroencephalogram semantic variance signal.

According to the method, the third encoder, namely the dynamic encoder, is arranged, noise in the electroencephalogram sample signal and effective data are separated by utilizing the obtained same static electroencephalogram semantic signal from different angles, so that dynamic electroencephalogram semantic signals, namely information of different parts in the electroencephalogram semantic signals, are obtained, the quality of the electroencephalogram sample signals is improved, and the classification precision of electroencephalogram signals of different categories is improved.

The brain wave processing device in the present application further comprises a decoder 40 for reconstructing the brain wave sample signal from the static brain wave semantic signal (F) and the dynamic brain wave semantic signal (Z).

In the present embodiment, the decoder 40 reconstructs the electroencephalogram sample signal through a deconvolution operation according to the static electroencephalogram semantic signal (F) and the dynamic electroencephalogram semantic signal (Z), that is, performs reconstruction of the electroencephalogram sample signal through a reverse operation with the first encoder.

Specifically, in the training process of the brain wave processing device, when the difference between the reconstructed brain wave sample signal data x 'reconstructed by the decoder according to the static brain wave semantic signal (F) and the dynamic brain wave semantic signal (Z) and the brain wave sample signal data x is large, the relative entropy of the mean square error function plus F, Z distribution and the standard normal distribution is used as an objective function to further train until the difference between the reconstructed brain wave sample signal data x' reconstructed by the decoder according to the static brain wave semantic signal (F) and the dynamic brain wave semantic signal (Z) and the brain wave sample signal data x meets a preset difference threshold.

When the brain wave processing device finishes training and predicts, the decoder reconstructs brain wave sample signals data x 'reconstructed according to the dynamic brain wave semantic signals (Z) so as to conveniently identify brain wave signals, see fig. 9, wherein the upper graph is the brain wave sample signals data x, and the lower graph is the brain wave sample signals data x' reconstructed by the decoder according to the dynamic brain wave semantic signals (Z) through deconvolution operation. The reconstructed brain wave sample signal data x 'is the brain wave signal restored after the same part in the brain wave signal is removed, each section in the brain wave sample signal data x can be seen to be quite similar from fig. 9, after the same part is removed, namely, the static brain wave semantic signal F is obtained, obvious distinction between brain wave signals of different types can be seen from the graph waveforms of the reconstructed brain wave sample signal data x', and therefore, the accuracy of classification of brain wave signals of different types can be improved.

According to the method, the input electroencephalogram sample signals are subjected to data processing through the set first encoder (namely the self-encoder) and the electroencephalogram semantic sub-signals corresponding to time information are output, the same static electroencephalogram semantic signals in all the electroencephalogram semantic sub-signals are extracted through the set second encoder (namely the static encoder), and finally, the noise in the electroencephalogram signals and the effective dynamic electroencephalogram semantic signals are separated from each other according to the electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals through the third encoder (namely the dynamic encoder), so that the classification precision of the electroencephalogram signals of different categories is effectively improved, the quality of the electroencephalogram signals is enhanced, and the method has wide application prospects.

Yet another embodiment of the present application provides a brain wave processing method, as shown in fig. 10, including:

s10, carrying out data processing on the received multipath electroencephalogram sample signals and respectively outputting electroencephalogram semantic signals, wherein the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, and each electroencephalogram semantic sub-signal corresponds to different time information;

s20, carrying out data processing on the received plurality of brain electrical semantic sub-signals and outputting static brain electrical semantic signals, wherein the static brain electrical semantic signals of each brain electrical semantic sub-signal are the same;

s30, carrying out data processing on the received plurality of electroencephalogram semantic sub-signals and static electroencephalogram semantic signals and outputting dynamic electroencephalogram semantic signals, wherein the dynamic electroencephalogram semantic signals comprise the time information;

s40, reconstructing the electroencephalogram sample signal according to the static electroencephalogram semantic signal F and the dynamic electroencephalogram semantic signal Z.

Since the brain wave processing method provided in the embodiment of the present application corresponds to the brain wave processing device provided in the above embodiment, the previous embodiment is also applicable to the brain wave processing method provided in the embodiment, and will not be described in detail in the embodiment.

The embodiment provides a brain wave processing method, which comprises the steps of performing data processing on input brain wave sample signals through a first encoder (namely a self-encoder) and outputting brain wave semantic sub-signals corresponding to time information, extracting the same static brain wave semantic signals in all brain wave semantic sub-signals through a second encoder (namely a static encoder), and finally separating noise in brain wave signals from effective dynamic brain wave semantic signals according to brain wave semantic sub-signals and static brain wave semantic signals from different angles through a third encoder (namely a dynamic encoder), so that classification precision of brain wave signals of different types is effectively improved, quality of brain wave signals is enhanced, and the brain wave processing method has wide application prospect.

Fig. 11 shows a schematic structural diagram of a computer device according to another embodiment of the present application. The computer device 50 shown in fig. 11 is merely an example, and should not be construed as limiting the functionality and scope of use of the embodiments herein.

As shown in FIG. 11, computer device 50 is in the form of a general purpose computing device. Components of computer device 50 may include, but are not limited to: one or more processors or processing units 500, a system memory 516, and a bus 501 that connects the various system components, including the system memory 516 and the processing units 500.

Bus 501 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Computer device 50 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer device 50 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 516 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 504 and/or cache memory 506. The computer device 50 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 508 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 11, commonly referred to as a "hard disk drive"). Although not shown in fig. 11, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be coupled to bus 501 through one or more data medium interfaces. The memory 516 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the brain wave processing method provided by the above-described embodiments.

A program/utility 510 having a set (at least one) of program modules 512 may be stored, for example, in a memory 516, such program modules 512 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 512 generally perform the functions and/or methods in the embodiments described herein.

The computer device 50 may also communicate with one or more external devices 70 (e.g., keyboard, pointing device, display 60, etc.), one or more devices that enable a user to interact with the computer device 50, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 50 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 502. Moreover, computer device 50 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 514. As shown in fig. 9, the network adapter 514 communicates with other modules of the computer device 50 over the bus 501. It should be appreciated that although not shown in fig. 11, other hardware and/or software modules may be used in connection with computer device 50, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

The processor unit 500 executes various functional applications and data processing by running programs stored in the system memory 516, for example, implementing an electroencephalogram processing method provided in the embodiments of the present application.

Another embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a brain wave processing method provided by the above embodiments.

In practical applications, the computer-readable storage medium may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this embodiment, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

In the description of the present application, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

It should be apparent that the foregoing examples of the present application are merely illustrative of the present application and not limiting of the embodiments of the present application, and that various other changes and modifications may be made by one of ordinary skill in the art based on the foregoing description, and it is not intended to be exhaustive of all embodiments, and all obvious changes and modifications that come within the scope of the present application are intended to be embraced by the technical solution of the present application.

Claims

1. An electroencephalogram processing apparatus, comprising a first encoder, a second encoder, a third encoder, and a decoder, wherein:

the decoder is used for reconstructing the electroencephalogram sample signal according to the static electroencephalogram semantic signal and the dynamic electroencephalogram semantic signal;

the first encoder includes a first input layer, a first convolution layer, a first flattening layer, a first full-connection layer, a second full-connection layer, and a first output layer, where

The first input layer transmits the received multi-path electroencephalogram sample signals to the first convolution layer, carries out convolution calculation through the first convolution layer, outputs the multi-path electroencephalogram sample signals to the first flattening layer, and outputs the multi-path electroencephalogram sample signals to the first full-connection layer after flattening through the first flattening layer, and respectively outputs the electroencephalogram semantic signals corresponding to the multi-path electroencephalogram sample signals through the first output layer after classification operation through the first full-connection layer and the second full-connection layer;

the second encoder comprises a second input layer, a first two-way long-short-term memory network, a first splicing layer, a third full-connection layer and a first sampling layer, wherein

The second input layer receives a plurality of electroencephalogram semantic signals and transmits the electroencephalogram semantic signals to the first two-way long-short-term memory network, the electroencephalogram semantic signals are transmitted to the first splicing layer after being subjected to time-cycle processing operation by the first two-way long-short-term memory network, the electroencephalogram semantic signals are transmitted to the third full-connection layer after being spliced by the first splicing layer, and the static electroencephalogram semantic signals are output by the first sampling layer after being subjected to classification operation by the third full-connection layer;

the first two-way long-short-term memory network includes a first forward long-short-term memory network and a first reverse long-short-term memory network, wherein:

the third full-connection layer comprises a first full-connection sub-layer and a second full-connection sub-layer, wherein

the first sampling layer outputs the static electroencephalogram semantic signal according to the static electroencephalogram semantic mean signal and the static electroencephalogram semantic variance signal;

The third encoder comprises a third input layer, a second bidirectional long-short-term memory network, a second splicing layer, an RNN layer, a fourth full connection layer and a second sampling layer, wherein

2. The brain wave processing device according to claim 1, wherein a kernel size of the first convolution layer is 3*3, a step size of the first convolution layer is 2, and a filling mode of the first convolution layer is a complement mode.

3. The brain wave processing device according to claim 1, wherein said second bidirectional long-short term memory network comprises a second forward long-short term memory network and a second reverse long-short term memory network, wherein,

4. The brain wave processing device according to claim 3, wherein said fourth full-connection layer includes a third full-connection sub-layer and a fourth full-connection sub-layer, wherein

5. The brain wave processing device according to claim 1, wherein said decoder reconstructs said brain wave sample signal by deconvolution operation from static brain wave semantic signals and dynamic brain wave semantic signals.

6. A brain wave processing method, which is applied to the brain wave processing device according to claim 1, the brain wave processing device including a first encoder, a second encoder, a third encoder, and a decoder, the brain wave processing method comprising:

The first encoder comprises a first input layer, a first convolution layer, a first flattening layer, a first full-connection layer, a second full-connection layer and a first output layer, processes data of received multipath electroencephalogram sample signals and outputs electroencephalogram semantic signals respectively, the electroencephalogram semantic signals comprise a plurality of electroencephalogram semantic sub-signals, each electroencephalogram semantic sub-signal corresponds to different time information, and the first encoder further comprises: the first input layer transmits the received multi-path electroencephalogram sample signals to the first convolution layer, carries out convolution calculation through the first convolution layer, outputs the multi-path electroencephalogram sample signals to the first flattening layer, and outputs the multi-path electroencephalogram sample signals to the first full-connection layer after flattening through the first flattening layer, and respectively outputs the electroencephalogram semantic signals corresponding to the multi-path electroencephalogram sample signals through the first output layer after classification operation through the first full-connection layer and the second full-connection layer;

the second encoder comprises a second input layer, a first two-way long-short-term memory network, a first splicing layer, a third full-connection layer and a first sampling layer, wherein the first two-way long-short-term memory network comprises a first forward long-short-term memory network and a first reverse long-short-term memory network, the third full-connection layer comprises a first full-connection sub-layer and a second full-connection sub-layer, data processing is carried out on a plurality of received brain electrical semantic sub-signals, static brain electrical semantic signals are output, the static brain electrical semantic signals of each brain electrical semantic sub-signal are the same, and the second encoder further comprises:

the third encoder comprises a third input layer, a second bidirectional long-short-term memory network, a second splicing layer, an RNN layer, a fourth full-connection layer and a second sampling layer, performs data processing on the received plurality of electroencephalogram semantic sub-signals and static electroencephalogram semantic signals and outputs dynamic electroencephalogram semantic signals, wherein the dynamic electroencephalogram semantic signals comprise the time information and further comprise: the third input layer receives the plurality of electroencephalogram semantic sub-signals and the static electroencephalogram semantic signals, transmits the static electroencephalogram semantic signals to the second bidirectional long-short-term memory network, performs time-loop processing operation through the second bidirectional long-short-term memory network, transmits the static electroencephalogram semantic signals to the second splicing layer, splices the static electroencephalogram semantic sub-signals through the second splicing layer, transmits the static electroencephalogram semantic signals to the RNN layer, performs time information processing operation through the RNN layer, transmits the static electroencephalogram semantic signals to the fourth full-connection layer, and outputs the dynamic electroencephalogram semantic signals with time information through the second sampling layer after classification operation through the fourth full-connection layer;

7. The brain wave processing method according to claim 6, wherein a kernel size of the first convolution layer is 3*3, a step size of the first convolution layer is 2, and a filling mode of the first convolution layer is a complement mode.

8. The brain wave processing method according to claim 6, wherein the second bidirectional long-short term memory network includes a second forward long-short term memory network and a second reverse long-short term memory network, the fourth full-connection layer includes a third full-connection sub-layer and a fourth full-connection sub-layer, and the data processing of the received plurality of brain electrical semantic sub-signals and the static brain electrical semantic signals and outputting the dynamic brain electrical semantic signals further includes:

9. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 6-8.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 6-8 when the program is executed by the processor.