CN115169384A

CN115169384A - Electroencephalogram classification model training method, intention identification method, equipment and medium

Info

Publication number: CN115169384A
Application number: CN202210675356.XA
Authority: CN
Inventors: 宋永昊; 郑青青; 王琼
Original assignee: Shenzhen Institute of Advanced Technology of CAS
Current assignee: Shenzhen Institute of Advanced Technology of CAS
Priority date: 2022-06-15
Filing date: 2022-06-15
Publication date: 2022-10-11

Abstract

The application provides an electroencephalogram classification model training method, an intention identification method, equipment and a medium, and belongs to the technical field of machine learning. The method comprises the following steps: acquiring target domain characteristics of target domain data and source domain characteristics of source domain data; determining global time domain dependence of the target domain features based on an attention mechanism; migrating the characteristics of the global time domain dependence to the source domain characteristics to obtain the migrated source domain characteristics; and training the initial model based on the target domain characteristics and the migrated source domain characteristics to obtain a trained classification model. According to the method and the device, the initial model can be trained by the electroencephalogram data of the first object and the electroencephalogram data of the target object, so that the requirement on the electroencephalogram data of the target object is reduced, the training efficiency of the classification model is improved, the preparation time of the brain-computer interface system before use is shortened, and the universality of the classification model after training is better.

Description

Electroencephalogram classification model training method, intention identification method, equipment and medium

Technical Field

The application belongs to the technical field of machine learning, and particularly relates to an electroencephalogram classification model training method, an intention identification method, equipment and a medium.

Background

The brain-computer interface is a connection established between the brain of a human or animal and an external device, and realizes information exchange between the brain and the external device. Brain-computer interface based systems can help users control devices to perform tasks with brain intent. Brain-computer interface systems have been widely used in a number of areas. For example, when the brain-computer interface system is applied to the field of games, the virtual characters in the game picture can be controlled to make corresponding actions, such as walking, jumping, steering and the like, according to the electroencephalogram signals of the user. After the brain-computer interface system collects the electroencephalograms of the user, how to determine the categories of the electroencephalograms so as to accurately analyze the control intentions corresponding to the electroencephalograms is a very important problem.

At present, a large number of machine learning methods are applied to classification analysis of electroencephalogram signals. However, because the difference of the electroencephalograms of different users is very large, the classification model trained based on the data of a single user is difficult to be applied to classification of the electroencephalograms of other users. The classification model in practical application has poor universality, is difficult to serve as a classification model universal for multiple users, and has low training efficiency.

Disclosure of Invention

In view of this, embodiments of the present application provide an electroencephalogram classification model training method, an intention recognition method, a device, and a medium, which can solve the problem of low efficiency of classification model training.

The first aspect of the embodiments of the present application provides a method for training an electroencephalogram classification model, including:

acquiring target domain characteristics of target domain data and source domain characteristics of source domain data; the target domain data includes a brain electrical signal of a target object, and the source domain data includes a brain electrical signal of a first object including objects other than the target object.

And determining the global time-domain dependence of the target domain feature based on an attention mechanism, wherein the global time-domain dependence is correlation data between feature data of each position of the target domain feature in the time domain.

And migrating the characteristics of the global time domain dependence to the source domain characteristics to obtain the migrated source domain characteristics.

And training the initial model based on the target domain characteristics and the migrated source domain characteristics to obtain a trained classification model.

In a first possible implementation manner of the first aspect, determining the global temporal dependency of the target domain feature based on the attention mechanism includes:

and converting the characteristic channel of the target domain characteristic into one dimension to obtain first one-dimensional data.

And performing linear transformation on the first one-dimensional data to obtain a first query and a first key.

And calculating the dot product between the first query and the rotated first key to obtain a global correlation matrix of the target domain features.

And carrying out scale transformation and normalization on the global correlation matrix to obtain a correlation weight matrix of the target domain characteristics, wherein the correlation weight matrix is global time domain dependence.

Based on the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, migrating a global time-domain dependent characteristic to a source domain characteristic to obtain a migrated source domain characteristic, includes:

and converting the characteristic channel of the source domain characteristic into one dimension to obtain second one-dimensional data.

And performing linear transformation on the second one-dimensional data to obtain a first value.

And calculating the dot product of the first value and the correlation weight matrix to obtain the source domain characteristics after the migration.

Based on the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, converting a feature channel of a target domain feature into a one-dimensional feature to obtain first one-dimensional data includes:

the target domain feature is divided into a plurality of first slice features along a time dimension.

And respectively converting the characteristic channels of the first slice characteristics into one dimension to obtain a plurality of third one-dimensional data, wherein the first one-dimensional data comprises a plurality of third one-dimensional data.

Based on the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, converting a feature channel of a source domain feature into a one-dimensional feature to obtain second one-dimensional data includes:

the source domain features are divided into a plurality of second slice features along the time dimension.

And respectively converting the characteristic channels of the second slice characteristics into one dimension to obtain a plurality of fourth one-dimensional data, wherein the second one-dimensional data comprises a plurality of fourth one-dimensional data.

In a fifth possible implementation manner of the first aspect, the obtaining a target domain feature of the target domain data and a source domain feature of the source domain data includes:

respectively performing time domain convolution and space domain convolution on the first data to obtain a first time domain characteristic and a first space domain characteristic; the first data is target domain data or source domain data.

The first time domain feature and the first spatial domain feature are resized to be uniform.

Performing operation based on the first time domain feature and the first space domain feature with consistent sizes to obtain a first feature; the first feature is a target domain feature or a source domain feature.

In a sixth possible implementation manner of the first aspect, the training the initial model based on the target domain feature and the migrated source domain feature to obtain a trained classification model includes:

and performing edge distribution alignment on the migrated source domain features and the target domain features based on the first loss function and the second loss function.

And performing conditional distribution alignment on the migrated source domain features and the migrated target domain features based on a third loss function.

And classifying the target domain features and the source domain features after edge distribution alignment and condition distribution alignment based on a fourth loss function to obtain a first classification result.

And training the initial model based on the first classification result to obtain a trained classification model.

Based on the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, the second loss function, the third loss function, and the fourth loss function may form a collaborative optimization function.

A second aspect of an embodiment of the present application provides an intention identifying method, including:

acquiring an electroencephalogram signal of a target object.

Classifying the electroencephalogram signals of the target object based on the classification model to obtain a second classification result; the classification model is obtained by training according to the electroencephalogram signal classification model training method provided by the first aspect.

And identifying the intention corresponding to the electroencephalogram signal of the target object based on the second classification result.

A third aspect of the embodiments of the present application provides an electroencephalogram classification model training device, including:

the acquisition module is used for acquiring target domain characteristics of the target domain data and source domain characteristics of the source domain data; the target domain data includes brain electrical signals of a target object, and the source domain data includes brain electrical signals of a first object including objects other than the target object.

And the determining module is used for determining the global time domain dependence of the target domain feature based on the attention mechanism, wherein the global time domain dependence is correlation data between feature data of each position of the target domain feature in a time domain.

And the migration module is used for migrating the characteristics of the global time domain dependence to the source domain characteristics to obtain the migrated source domain characteristics.

And the training module is used for training the initial model based on the target domain characteristics and the migrated source domain characteristics to obtain a trained classification model.

A fourth aspect of the embodiments of the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program that is executable on the processor, and the processor implements the steps of the electroencephalogram classification model training method provided in the first aspect, or implements the steps of the intent recognition method provided in the second aspect, when executing the computer program.

A fifth aspect of embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored, where the computer program is implemented by a processor to implement the steps of the electroencephalogram classification model training method provided in the above first aspect, or to implement the steps of the intent recognition method provided in the above second aspect.

A sixth aspect of the embodiments of the present application provides a computer program product, which, when running on a terminal, causes the terminal to execute the steps of the electroencephalogram classification model training method provided in the above first aspect, or execute the steps of the intent recognition method provided in the above second aspect.

It is understood that the beneficial effects of the second to sixth aspects can be seen from the description of the first aspect, and are not described herein again.

Compared with the prior art, the embodiment of the application has the advantages that:

after the target domain features and the source domain features are obtained, the globally time-domain dependent characteristics of the target domain features are migrated to the source domain features. The alignment of the source domain features and the target domain features can be promoted, so that the migrated source domain features can be used for training an initial model to obtain a classification model adaptive to a target object. Based on an attention mechanism, the global time domain dependence of the target domain features is determined, so that the feature distribution of the source domain features is migrated and adjusted, and the effect of migration learning can be improved. According to the electroencephalogram signal classification model training method provided by the embodiment of the application, the electroencephalogram signal data of the first object and the electroencephalogram signal data of the target object can be used for training the initial model together, so that the requirement on the electroencephalogram signal data of the target object is reduced, the training efficiency of the classification model is improved, and the preparation time before the brain-computer interface system is used is shortened. When the classification model is applied to a new target object, the previous training data can still be used for training the classification model to obtain a classification model adapted to the new target object. Therefore, the trained classification model has better universality.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

FIG. 1 is a schematic flow chart of a method for training an electroencephalogram classification model according to an embodiment of the present application;

FIG. 2 is a schematic flowchart of a method for training an electroencephalogram classification model according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a parallel convolutional neural network provided in an embodiment of the present application;

FIG. 4 is a schematic flowchart of a method for training an electroencephalogram classification model according to an embodiment of the present application;

FIG. 5 is a flowchart illustrating a method for training a classification model of an electroencephalogram signal according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a global adaptor according to an embodiment of the present application;

FIG. 7 is a flowchart illustrating a method for training a classification model of an electroencephalogram signal according to an embodiment of the present application;

FIG. 8 is a flowchart illustrating a method for training a classification model of an electroencephalogram signal according to an embodiment of the present application;

FIG. 9 is a flowchart illustrating a method for training a classification model of an electroencephalogram signal according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a training process of a method for training a classification model of an electroencephalogram signal according to an embodiment of the present application;

FIG. 11 is a flowchart illustrating an intent recognition method according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of an electroencephalogram classification model training apparatus according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

The brain-computer interface is a connection established between the brain of a human or animal and an external device, and realizes information exchange between the brain and the external device.

The brain-computer interface may include both noninvasive and implantable types. The noninvasive brain-computer interface is a device worn on the periphery of the brain sack and does not need to go deep into the tissues below the skull. The implanted brain-computer interface needs to reach deep into the tissues below the skull, even into the cerebral cortex.

The advantages of a noninvasive brain-machine interface are lower cost, no trauma and no surgical risk. The implanted brain-computer interface has the advantages of higher time-space resolution and larger information content of recorded brain electrical signals. The type of the brain-computer interface is not specifically limited in the embodiments of the present application, and a technician may select the type as needed.

Brain-computer interface based systems have been widely used in a number of areas. For example, when the brain-computer interface system is applied to the field of games, the virtual characters in the game screen can be controlled to make corresponding actions such as walking, jumping, steering and the like according to the electroencephalogram signals of the user.

Brain-computer interface based systems can help users control devices to perform tasks with brain intent. The working principle of the method is that after the brain-computer interface collects the brain electrical signals, the brain electrical signals are classified, corresponding control intentions are identified according to classification results, and corresponding control instructions are sent to equipment.

Therefore, how to determine the category of the electroencephalogram signal so as to accurately analyze the control intention corresponding to the electroencephalogram signal is a very important problem.

For this purpose, a machine learning method can be adopted to train and obtain a classification model for determining the category of the electroencephalogram signal.

Machine learning studies how computers simulate human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to improve themselves. Therefore, in the embodiment of the present application, the classification model is to predict the category of new electroencephalogram data by learning and training the existing electroencephalogram data.

However, because the electroencephalogram signal itself has a lot of noise, a lot of target user data are usually needed to calibrate the initial model, and a classification model with a good classification effect can be obtained. And because of individual differences caused by the physiological state and the environment, the electroencephalogram signal data of other users cannot be directly applied to the calibration of the classification model of the target user. The effectiveness of the classification model may even be reduced if applied directly. This can lead to an excessively long preparation time before the brain-computer interface system is used, which greatly affects the practicability of the brain-computer interface system.

Based on the analysis, because the difference of the electroencephalograms of different users is very large, the classification model trained based on the data of a single user is difficult to be applied to classification of the electroencephalograms of other users. The classification model in practical application has poor universality, is difficult to serve as a classification model universal for multiple users, and has low training efficiency.

In view of this, the embodiment of the present application provides a method for training an electroencephalogram classification model, which helps a new model to train by migrating trained model parameters to the new model through migration learning. The electroencephalogram data of other users are migrated to a new model to help the new model training, so that a classification model suitable for a target user is obtained.

In the embodiment of the application, the global time domain dependence of the electroencephalogram signal data of the target user can be determined based on the attention mechanism neural network. The global time domain dependency refers to correlation data between feature data of each position of electroencephalogram data of a target user in a time domain. And transferring the characteristics of the global time domain dependence to electroencephalogram data of other users. Therefore, the electroencephalogram data of other migrated users can participate in training the initial model of the target user, and the classification model suitable for the target user is obtained.

It should be understood that the initial model, with respect to the target user, refers to a classification model that has not been classification trained, or has not been fully classification trained.

The distribution difference of the characteristics of the electroencephalogram signals at each position in the time domain is more obvious than that of the spatial domain, so that the characteristics of global time domain dependence are transferred in the embodiment of the application. The classification accuracy of the obtained classification model is higher.

According to the electroencephalogram signal classification model training method provided by the embodiment of the application, the electroencephalogram signal data of other users and the electroencephalogram signal data of the target user can be used for training the initial model together, so that the training efficiency of the classification model is improved. Because the initial model can be trained by using electroencephalogram data of other users, when the classification model is applied to a new target user, the prior training data can still be used for adjusting and training the classification model to obtain the classification model suitable for the new target user. Therefore, the trained classification model has better universality.

In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.

Fig. 1 shows a schematic flow chart of a method for training a classification model of an electroencephalogram signal provided in an embodiment of the present application, which is detailed as follows:

s101, acquiring target domain characteristics of target domain data and source domain characteristics of source domain data. The target domain data includes a brain electrical signal of the target object and the source domain data includes a brain electrical signal of the first object.

It should be noted that, because a human or an animal can emit an electroencephalogram signal, both the target object and the first object herein may refer to a human or an animal, and this is not particularly limited in this embodiment of the present application. For convenience of explanation, in the embodiments of the present application, the target object and the first object both refer to a human.

Due to the need of migrating the relevant characteristics of the target domain data to the source domain data by a migration learning method. The migrated source domain data can also participate in training the initial model, so that the training efficiency is improved. Therefore, the first object should include objects other than the target object.

It should be understood that in the field of machine learning technology, raw data generally needs to be subjected to feature extraction so as to be input as training data into an initial model for training. Therefore, before training, it is necessary to acquire feature data of the raw data. The feature data is used as training data. In the embodiment of the application: the target domain characteristics of the target domain data and the source domain characteristics of the source domain data need to be obtained.

And because some interference, such as noise and artifacts, may be present in the raw data. Therefore, before the feature data of the raw data is obtained, the raw data needs to be preprocessed, i.e., the interference in the target domain data and/or the source domain data is removed.

By way of example and not limitation, the process of preprocessing may include three parts: truncation, band pass filtering, and normalization.

The segmentation is to segment the electroencephalogram signal into a plurality of segments according to a time range set by an experimental paradigm, and simultaneously convert a data channel of each segment into a one-dimension. And correspondingly obtaining one-dimensional electroencephalogram data by each segment. Thus, the data form of the segmented brain electrical signal includes two dimensions, channel and time.

It should be noted that the experimental paradigm is a relatively fixed experimental procedure. Including the purpose, specific flow, means and the like of the experiment.

The band-pass filtering is to filter out high-frequency and low-frequency noise irrelevant to classification training and more obvious artifacts. By way of example, and not limitation, a band pass filter may be employed for this purpose. For example, the data is filtered to 4-40 Hz using a Chebyshev filter sixth-order bandpass filter, which contains the beta and mu waves of the brain rhythm.

Normalization is to eliminate non-stationary interference caused by some abnormal data samples or outliers. By way of example and not limitation, the data may be normalized using the following equation:

wherein x is _i And x _o Representing input and output, mu and sigma, respectively ² Mean and variance of the training data set are respectively represented.

After the above preprocessing, step S101 is performed again: and acquiring target domain characteristics of the target domain data and source domain characteristics of the source domain data.

In some embodiments, target domain features of the target domain data and source domain features of the source domain data may be extracted by a feature extractor.

It should be understood that, in the present embodiment, how to extract the target domain features of the target domain data and how to extract the source domain features of the source domain data are not particularly limited. Because, the purpose of step S01 is to obtain the target domain features and the source domain features without considering how to extract the features.

By way of example and not limitation, based on characteristics of the brain electrical data, a spatio-temporal depth characterization of the target domain data and the source domain data may be obtained along a time dimension and a spatial channel dimension. And taking the space-time depth characterization of the target domain data as the target domain characteristics. And characterizing the spatiotemporal depth of the source domain data as source domain features.

As shown in fig. 2, in some embodiments, step S101 may include steps S201 to S203.

S201, performing time domain convolution and space domain convolution on the first data respectively to obtain a first time domain feature and a first space domain feature. The first data is target domain data or source domain data.

By way of example and not limitation, the feature extractor includes one-dimensional convolutional layers along the temporal dimension and the spatial channel dimension.

It should be noted that the time domain convolution and the spatial domain convolution may be connected in series or in parallel.

For example, when the data are connected in series, the first data are subjected to time domain convolution, and then the first data subjected to time domain convolution are subjected to spatial domain convolution to obtain the space-time depth representation of the first data. However, the first data has a large amount of characteristic information lost during the serial convolution calculation.

As shown in fig. 3, to avoid the information loss caused by the serial convolution calculation process, better temporal and spatial characterization can be obtained. In some embodiments, the time-domain convolution and the spatial-domain convolution of the feature extractor employ a parallel connection for performing the time-domain convolution and the spatial-domain convolution, respectively, on the first data.

S202, adjusting the sizes of the first time domain feature and the first space domain feature to be consistent.

And after the time domain convolution and the space domain convolution are respectively carried out on the first data, a first time domain characteristic and a first space domain characteristic are obtained. Due to the non-uniform size of the first time-domain feature and the first spatial-domain feature. Therefore, the sizes of the first time domain feature and the first space domain feature need to be adjusted to be consistent, so that the first time domain feature and the first space domain feature are operated to obtain a space-time depth characterization of the first data, that is, the first feature is obtained.

S203, operating the first time domain feature and the first space domain feature with the same size to obtain a first feature. The first feature is a target domain feature or a source domain feature.

It should be noted that, in the embodiment of the present application, a specific algorithm of the operation is not specifically limited, and a skilled person may select the algorithm as needed.

By way of example, and not limitation, the operation includes, but is not limited to: addition, subtraction, multiplication, and division.

In this embodiment, the sum of the first time domain feature and the first spatial domain feature with the same size is calculated by addition to obtain the first feature.

By way of example and not limitation, the convolution kernel size of the time domain convolution is k × 1 × n. The convolution kernel size of the space-domain convolution is k × m × 1. The feature of the first time domain feature adjusts the convolution kernel size of the convolution to 1 × m × 1. The feature-adjusted convolution of the first spatial domain feature has a convolution kernel size of 1 × 1 × n. Wherein k, m and n are integers. n can be valued according to actual conditions. m is usually the number of channels of the input EEG signal data.

Among other things, the batch normalization and activation functions shown in FIG. 3 may be used to improve the fit ability. Normalization is a simplified calculation mode, i.e. a dimensional expression is transformed into a dimensionless expression to become a scalar. Batch normalization means batch normalization. Activation functions, which are functions that run on neurons of a neural network, are responsible for mapping the inputs of the neurons to outputs. The fitting ability is improved because the activation function can introduce non-linear factors.

The skilled person can select a specific batch normalization method and an activation function according to actual needs, which is not specifically limited in the embodiment of the present application.

In this embodiment, more accurate time or space representation can be obtained by using parallel connection of time domain convolution and space domain convolution. The space-time depth representation of the first data is more accurate, and the training quality of the classification model can be improved.

And S102, determining the global time domain dependence of the target domain features based on the attention mechanism. The global time domain dependency is correlation data between feature data of each position of the target domain feature in the time domain.

S103, migrating the characteristics of the global time domain dependence to the source domain characteristics to obtain the migrated source domain characteristics.

Attention Mechanism (Attention Mechanism) stems from the study of human vision. In cognitive science, humans selectively focus on a portion of all information while ignoring other visible information due to bottlenecks in information processing. The above mechanism is commonly referred to as an attention mechanism.

Based on the above description, in this embodiment, the global time domain dependency of the target domain feature is determined based on the attention mechanism, so that the effect of the transfer learning can be improved, and the alignment of the source domain feature and the target domain feature is promoted.

As shown in fig. 4, in some embodiments, step S102 includes steps S401 to S404.

S401, converting the characteristic channel of the target domain characteristic into one dimension to obtain first one-dimensional data.

According to the characteristics of the electroencephalogram data, the target domain characteristics can comprise a plurality of characteristic channel numbers, and each characteristic channel corresponds to a plurality of characteristic data. That is, in each feature channel, a plurality of feature data thereof respectively correspond to feature data of a plurality of positions in the time domain.

Therefore, in step S401, the feature channels of the target domain feature are converted into one dimension, which is equivalent to adjusting the number of the feature channels to one, that is, all the feature data are arranged in one dimension.

It should be noted that, in the embodiment of the present application, the arrangement order of the feature data in one dimension is not specifically limited, and a skilled person may perform adjustment as needed.

It should be noted that, in some embodiments, each element in the first one-dimensional data corresponds to the feature data of each position of the target domain feature in the time domain.

S402, carrying out linear transformation on the first one-dimensional data to obtain a first query and a first key.

By way of example and not limitation, a first linear transformation is performed on the first one-dimensional data resulting in a first query. And performing second linear transformation on the first one-dimensional data to obtain a first key.

Note that the attention mechanism includes three parameters, i.e., query (query), key (key), and value (value). In the embodiment of the application, the migrated source domain feature is obtained by calculating the values of the query, the key and the source domain feature corresponding to the target domain feature.

S403, calculating a dot product between the first query and the rotated first key to obtain a global correlation matrix of the target domain feature.

In the embodiment of the application, a global correlation matrix of the target domain feature can be obtained by calculating the dot product between the first query and the rotated first key.

S404, carrying out scale transformation and normalization on the global correlation matrix to obtain a correlation weight matrix of the target domain features, wherein the correlation weight matrix is global time domain dependent.

Before the global correlation matrix is migrated, the global correlation matrix is subjected to scale transformation and normalization so as to unify dimensions and simplify calculation. And obtaining a correlation weight matrix of the target domain features after carrying out scale transformation and normalization. The correlation weight matrix is a global time-domain dependency.

As shown in fig. 5, in some embodiments, step S103 includes steps S501 to S503.

S501, converting the characteristic channel of the source domain characteristic into one dimension to obtain second one-dimensional data.

In order to be able to transfer the global time-domain dependent characteristics to the source domain features, the feature channels of the source domain features need to be also converted into one dimension to obtain second one-dimensional data.

It should be noted that, in some embodiments, each element in the second one-dimensional data corresponds to the feature data of each position of the source domain feature in the time domain.

And S502, performing linear transformation on the second one-dimensional data to obtain a first value.

S503, calculating the dot product of the first value and the correlation weight matrix to obtain the source domain characteristics after the migration.

Based on the above description, the migrated source domain features can be obtained by calculating the dot product of the first value and the correlation weight matrix based on the attention mechanism.

By way of example and not limitation, as shown in FIG. 6, a process of migrating global time-domain dependent features of a target domain feature to the target domain feature is illustrated. By way of example, and not limitation, the process may be expressed by the following equation:

wherein Q is _t Representing a first query, K, corresponding to a target domain feature _t Representing a first key, V, corresponding to a feature of the target domain _s Representing a source domainThe characteristic corresponds to a first value. K _t ^T Represents K _t The transposing of (1). Softmax represents an exponential normalization function. d represents Q _t And K _t Length of (d).

As described above, in the embodiments shown in fig. 4 and 5, each element in the first one-dimensional data corresponds to the feature data of each position of the target domain feature in the time domain. And each element in the second one-dimensional data respectively corresponds to the characteristic data of each position of the source domain characteristic in the time domain. In order to improve the accuracy of the global time-domain dependence of the solution and reduce the complexity of the solution calculation process, the first one-dimensional data and the second one-dimensional data may be divided into a plurality of one-dimensional data, respectively.

For example, in some embodiments, as shown in fig. 7, step S401 includes the steps of:

s701, dividing the target domain features into a plurality of first slice features along the time dimension.

S702, respectively converting the characteristic channels of the first cutting features into one dimension to obtain a plurality of third one-dimensional data corresponding to the first cutting features. The first one-dimensional data includes the plurality of third one-dimensional data.

Therefore, steps S701 and S702 are substituted into the embodiment shown in fig. 4. Steps S402 to S404 include the steps of:

and respectively carrying out linear transformation on the plurality of third one-dimensional data to obtain a second query and a second key corresponding to each third one-dimensional data.

And respectively calculating dot products between the second query corresponding to each third one-dimensional data and the rotated second key to obtain a first correlation matrix corresponding to each first slice feature.

And respectively carrying out scale transformation and normalization on each first correlation matrix to obtain a first weight matrix corresponding to each first slice characteristic. Each first weight matrix constitutes a correlation weight matrix.

For another example, in some embodiments, as shown in fig. 8, step S501 includes the following steps:

and S801, dividing the source domain features into a plurality of second slice features along the time dimension.

And S802, respectively converting the characteristic channels of the second slice characteristics into one dimension to obtain a plurality of fourth one-dimensional data corresponding to the second slice characteristics. The second one-dimensional data includes the plurality of fourth one-dimensional data.

Therefore, steps S801 and S802 are substituted into the embodiment shown in fig. 5. Steps S502 to S503 include the steps of:

and respectively carrying out linear transformation on the plurality of fourth one-dimensional data to obtain a second value corresponding to each fourth one-dimensional data.

And respectively calculating dot products between the second values corresponding to the fourth one-dimensional data and the first weight matrixes to obtain a plurality of second slice characteristics after migration. The plurality of migrated second slice features constitute migrated source domain features.

In this embodiment, the target domain feature is divided into a plurality of first slice features along a time dimension, and then a first weight matrix corresponding to each first slice feature is solved. Each first weight matrix constitutes a correlation weight matrix. The process can improve the accuracy of the calculated correlation weight matrix and reduce the calculation complexity.

In some embodiments, in step S102, a global temporal dependency of the target domain feature may be determined based on a multi-head attention mechanism to improve the calculation accuracy of the global temporal dependency.

A multi-head attention mechanism: means to divide the input of the attention mechanism into a plurality of equal small parts called heads; and realizing attention mechanism on each head, and splicing the results of the attention mechanism of each head to obtain the final output.

This process can be expressed by the following equation:

MHA(Q _t ,K _t ,V _s )＝[head ₀ ；…；head _h-1 ] (3)

wherein h represents the number of divided multiple heads,W _l ^Q 、

respectively representing the linear transformation relations for obtaining the query, key, value, and l represents the index of the header. x is the number of _t Is the target domain data, x _s Is the source domain data.

And S104, training the initial model based on the target domain characteristics and the migrated source domain characteristics to obtain a trained classification model.

In this embodiment, the initial model may be trained jointly by using the brain electrical signal data of the first object and the brain electrical signal data of the target object. Therefore, the initial model can be trained together by using less electroencephalogram data of the target object and matching with more electroencephalogram data of the first object, and the training efficiency of the classification model is improved. When the classification model is applied to a new target object, the previous training data can still be used for training the classification model to obtain a classification model adapted to the new target object. Therefore, the trained classification model has better universality.

As shown in fig. 9, in some embodiments, step S104 includes the following steps S901 to S904.

S901, performing edge distribution alignment on the migrated source domain features and the migrated target domain features based on the first loss function and the second loss function.

By way of example and not limitation, to edge-distribute align the migrated source domain features with the target domain features, the first penalty function may be set in the manner of a WGAN (Wasserstein generated adaptive Networks). The first loss function is an objective function of the discriminator.

The first loss function is:

wherein E represents an expected value; x is the number of _s For source domain data, H _s Is a range of source domain data; x is the number of _t As target domain data, H _t Is a range of target domain data; GP represents a gradient penalty for increasing convergence speed; lambda [ alpha ] _gp Is a hyper-parameter; α is a random value between 0 and 1; F. a, D represents the processes of feature extraction, global time-domain dependent feature migration, and training of the initial model, respectively.

To form the antagonistic learning relationship, the constraints on the feature extractor and the global adaptor corresponding to the objective function of the arbiter are a second loss function.

The second loss function is:

after the above-mentioned feature alignment process, both the target domain features and the aligned source domain features can be used to train the classifier of the target object.

Under alternating training, the feature extractor and the global adaptor tend to find features in the source domain and target domain data that are uniformly distributed, i.e., gradually complete the alignment of the source domain and target domain distribution pairs.

And S902, performing conditional distribution alignment on the migrated source domain features and the migrated target domain features based on a third loss function.

By way of example, and not limitation, the third loss function is:

wherein, ct represents the characteristic center of different categories of electroencephalogram signals in the target domain data.

Thus, under the constraint of the third loss function, the source domain features of different classes are distributed close to the target domain features of the corresponding class. At the same time, intra-class differences become smaller and inter-class differences become larger.

And S903, classifying the target domain features and the source domain features after edge distribution alignment and condition distribution alignment based on a fourth loss function to obtain a first classification result.

By way of example and not limitation, the fourth loss function is:

wherein N is _s Number of source domain data, N _t Is the number of target domain data, M is the number of classes of the electroencephalogram signal, y and

respectively representing classified real labels and predicted labels, i and c are integers respectively.

By way of example and not limitation, the second, third and fourth loss functions described above may constitute a co-optimized loss function as follows.

L _joint ＝L _cls +ω _G L _G +ω _act L _act (11)

Wherein, ω is _G 、ω _act Weights respectively representing the second loss function and the third loss function are hyperparameters.

In this embodiment, based on the first loss function and the second loss function, under the constraint of counterlearning, the edge distribution alignment is performed on the migrated source domain feature and the target domain feature.

In this embodiment, the condition distribution alignment is performed on the migrated source domain feature and the target domain feature through the third loss function, so that the condition distribution difference between the source domain data and the target domain data can be further eliminated.

And S904, training the initial model based on the first classification result to obtain a trained classification model.

Based on the above description, training the initial model under the alternate confrontation learning, a trained classification model can be obtained.

In summary, fig. 10 shows a training process schematic diagram of the electroencephalogram signal classification model training method provided in an embodiment of the present application.

In this embodiment, after the target domain data is input to the feature extractor, the target domain features are output. After the source domain data is input into the feature extractor, the source domain features are output. When the target domain features are input into the global adaptor, the global time domain dependence of the target domain features is obtained based on an attention mechanism, and the global time domain dependence is transferred to the source domain features.

The global adaptor thus outputs the target domain features and the migrated source domain features. The target domain features and the migrated source domain features may be input to the classifier and the discriminator simultaneously. And finally obtaining the trained classification model after counterstudy. Wherein the adaptive center loss represents a third loss function.

Fig. 11 is a flowchart illustrating an intention identifying method according to an embodiment of the present application. The intention identification method includes:

s1101, acquiring an electroencephalogram signal of a target object.

And S1102, classifying the electroencephalogram signals of the target object based on the classification model to obtain a second classification result. The classification model is obtained by training the electroencephalogram signal classification model training method provided by any one of the embodiments shown in fig. 1 to fig. 10.

And S1103, identifying the intention corresponding to the electroencephalogram signal of the target object based on the second classification result.

In this embodiment, a classification model obtained by training in any one of the embodiments shown in fig. 1 to fig. 10 is applied to classify the electroencephalogram signal of the target object. An intent corresponding to the brain electrical signal of the target object may then be identified based on the second classification result.

The intention recognition method provided by this embodiment uses the classification model trained by any one of the embodiments shown in fig. 1 to fig. 10, so that the electroencephalogram signal of the target object can be accurately classified. And the training time before use is short, so that the practicability of the classification model is improved.

It should be understood that the embodiments of the above applications can be combined with each other to adapt to practical application requirements without logic conflict. These embodiments or embodiments obtained by combination are still within the scope of protection of the present application.

Corresponding to the electroencephalogram classification model training method described in the above embodiment, fig. 12 shows a schematic structural diagram of the electroencephalogram classification model training device provided in an embodiment of the present application, and for convenience of description, only the relevant parts of the electroencephalogram classification model training device are shown in the embodiment of the present application.

Referring to fig. 12, the electroencephalogram classification model training apparatus includes:

the obtaining module 121 is configured to obtain a target domain feature of the target domain data and a source domain feature of the source domain data. The target domain data includes brain electrical signals of a target object, and the source domain data includes brain electrical signals of a first object including objects other than the target object.

A determination module 122, configured to determine a global temporal dependency of the target domain feature based on the attention mechanism. The global time domain dependency is correlation data between feature data of each position of the target domain feature in the time domain.

The migration module 123 is configured to migrate the global time domain dependent characteristic to the source domain characteristic to obtain a migrated source domain characteristic.

And the training module 124 is configured to train the initial model based on the target domain features and the migrated source domain features to obtain a trained classification model.

In some embodiments, the determination module 122 includes:

and the first conversion unit is used for converting the characteristic channel of the target domain characteristic into one dimension to obtain first one-dimensional data.

And the first transformation unit is used for carrying out linear transformation on the first one-dimensional data to obtain a first query and a first key.

And the first calculation unit is used for calculating the dot product between the first query and the rotated first key to obtain a global correlation matrix of the target domain feature.

And the second calculation unit is used for carrying out scale transformation and normalization on the global correlation matrix to obtain a correlation weight matrix of the target domain characteristics, wherein the correlation weight matrix is global time domain dependence.

In some embodiments, the migration module 123 includes:

and the second conversion unit is used for converting the characteristic channel of the source domain characteristic into one dimension to obtain second one-dimensional data.

And the second transformation unit is used for carrying out linear transformation on the second one-dimensional data to obtain a first value.

And the third calculating unit is used for calculating the dot product of the first value and the correlation weight matrix to obtain the source domain characteristics after the migration.

In some embodiments, the first conversion unit comprises:

the first dividing unit is used for dividing the target domain feature into a plurality of first slice features along the time dimension.

And the third conversion unit is used for respectively converting the characteristic channels of the first slice characteristics into one dimension to obtain a plurality of third one-dimensional data, and the first one-dimensional data comprises the plurality of third one-dimensional data.

In some embodiments, the second conversion unit comprises:

and the second dividing unit is used for dividing the source domain characteristics into a plurality of second slice characteristics along the time dimension.

And the fourth conversion unit is used for respectively converting the characteristic channels of the second slice characteristics into one dimension to obtain a plurality of fourth one-dimensional data, and the second one-dimensional data comprises the plurality of fourth one-dimensional data.

In some embodiments, the obtaining module 121 includes:

and the first convolution unit is used for respectively performing time domain convolution and space domain convolution on the first data to obtain a first time domain characteristic and a first space domain characteristic. The first data is target domain data or source domain data.

And the second convolution unit is used for adjusting the sizes of the first time domain feature and the first space domain feature to be consistent.

And the fourth calculating unit is used for performing operation based on the first time domain feature and the first space domain feature with consistent sizes to obtain the first feature. The first feature is a target domain feature or a source domain feature.

The process of implementing each function by each module in the electroencephalogram classification model training device provided in the embodiment of the present application may specifically refer to the description of the embodiment shown in fig. 1 and other related method embodiments, and is not described herein again.

It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

It will be understood that the terms "comprises," "comprising," "includes" and/or the like, when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing a relative importance or importance. It will also be understood that, although the terms first, second, etc. may be used in some embodiments of the present application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first computing unit may be named a second computing unit, and similarly, a second computing unit may be named a first computing unit, without departing from the scope of the various described embodiments. The first and second calculation units are both calculation units, but they are not the same calculation unit.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

The electroencephalogram classification model training method provided by the embodiment of the application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented Reality (AR)/Virtual Reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal Digital Assistants (PDAs), and the like, and the embodiment of the application does not have any limitation on the specific types of the electronic devices.

Fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 13, the electronic device 13 of this embodiment includes: at least one processor 130 (only one shown in fig. 13), a memory 131, the memory 131 having stored therein a computer program 132 executable on the processor 130. When the processor 130 executes the computer program 132, the steps in the above-mentioned various electroencephalogram classification model training method embodiments are implemented, for example, steps S101 to S104 shown in fig. 1; or steps S1101 to S1103 shown in fig. 11. Alternatively, the processor 130 implements the functions of the modules/units in the above-described device embodiments, such as the functions of the modules 121 to 124 shown in fig. 12, when executing the computer program 132.

The electronic device 13 may be a desktop computer, a notebook, a palm computer, a cloud server, or the like. The electronic device 13 may include, but is not limited to: a processor 130, and a memory 131. Those skilled in the art will appreciate that fig. 13 is merely an example of the electronic device 13, and does not constitute a limitation of the electronic device 13, and may include more or less components than those shown, or combine certain components, or different components, for example, the electronic device 13 may also include an input transmitting device, a network access device, a bus, etc.

The Processor 130 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The storage 131 may be an internal storage unit of the electronic device 13 in some embodiments, such as a hard disk or a memory of the electronic device 13. The memory 131 may also be an external storage device of the electronic device 13, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the electronic device 13. The memory 131 may also include both an internal storage unit of the electronic device 13 and an external storage device. The memory 131 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program 132. The memory 131 may also be used to temporarily store data that has been transmitted or is to be transmitted.

In addition, it is obvious to those skilled in the art that, for convenience and simplicity of description, the foregoing functional units and modules are merely illustrated in terms of division, and in practical applications, the foregoing functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the apparatus may be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The embodiment of the present application further provides an electronic device, where the electronic device includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor, and when the processor executes the computer program, the electronic device is enabled to implement the steps in the above method embodiments.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments may be implemented.

Embodiments of the present application provide a computer program product, which when executed on an electronic device, enables the electronic device to implement the steps in the above method embodiments.

The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like.

In the above embodiments, the description of each embodiment has its own emphasis, and reference may be made to the related description of other embodiments for parts that are not described or recited in any embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application, and are intended to be included within the scope of the present application.

Claims

1. The electroencephalogram signal classification model training method is characterized by comprising the following steps:

acquiring target domain characteristics of target domain data and source domain characteristics of source domain data; the target domain data comprises an electroencephalogram signal of a target object, the source domain data comprises an electroencephalogram signal of a first object, and the first object comprises objects except the target object;

determining a global time-domain dependence of the target domain feature based on an attention mechanism, wherein the global time-domain dependence is correlation data between feature data of each position of the target domain feature in a time domain;

migrating the characteristics of the global time domain dependence to the source domain characteristics to obtain the migrated source domain characteristics;

and training an initial model based on the target domain features and the source domain features after the migration to obtain the trained classification model.

2. The training method of claim 1, wherein the determining the global temporal dependence of the target domain feature based on the attention mechanism comprises:

converting the characteristic channel of the target domain characteristic into one dimension to obtain first one-dimensional data;

performing linear transformation on the first one-dimensional data to obtain a first query and a first key;

calculating a dot product between the first query and the transposed first key to obtain a global correlation matrix of the target domain feature;

and carrying out scale transformation and normalization on the global correlation matrix to obtain a correlation weight matrix of the target domain characteristics, wherein the correlation weight matrix is the global time domain dependence.

3. The training method according to claim 2, wherein the migrating the global time-domain dependent characteristic to the source domain feature to obtain the migrated source domain feature comprises:

converting the characteristic channel of the source domain characteristic into one dimension to obtain second one-dimensional data;

performing linear transformation on the second one-dimensional data to obtain a first value;

4. The training method of claim 2, wherein the converting the feature channel of the target domain feature into one dimension to obtain a first one-dimensional data comprises:

dividing the target domain features into a plurality of first slice features along a time dimension;

5. The training method of claim 3, wherein converting the eigen-channel of the source domain feature into one-dimensional data to obtain second one-dimensional data comprises:

dividing the source domain features into a plurality of second slice features along a time dimension;

6. The training method of claim 1, wherein the obtaining the target domain features of the target domain data and the source domain features of the source domain data comprises:

respectively performing time domain convolution and space domain convolution on the first data to obtain a first time domain characteristic and a first space domain characteristic; the first data is the target domain data or the source domain data;

resizing the first time-domain feature and the first spatial-domain feature to be consistent;

performing operation based on the first time domain feature and the first space domain feature with consistent sizes to obtain a first feature; the first feature is the target domain feature or the source domain feature.

7. The training method according to claim 1, wherein the training an initial model based on the target domain features and the migrated source domain features to obtain the trained classification model comprises:

performing edge distribution alignment on the migrated source domain features and the migrated target domain features based on a first loss function and a second loss function;

performing conditional distribution alignment on the migrated source domain features and the migrated target domain features based on a third loss function;

classifying the target domain features, the edge distribution alignment and the source domain features after the condition distribution alignment based on a fourth loss function to obtain a first classification result;

training the initial model based on the first classification result to obtain the trained classification model;

wherein the third loss function is:

e denotes the expected value, x _s For the source domain data, H _s Is the range of the source domain data, x _t For the target domain data, H _t For the range of the target domain data, F, A represents the feature extraction, the migration process of the global time-domain dependent feature, respectively, c _t Representing the feature centers of different classes of electroencephalogram signals in the target domain data.

8. An intention recognition method, comprising:

acquiring an electroencephalogram signal of a target object;

classifying the electroencephalogram signal of the target object based on a classification model to obtain a second classification result; the classification model is obtained by training according to the electroencephalogram signal classification model training method of any one of claims 1 to 7;

9. Electronic device, characterized in that it comprises a memory, a processor, said memory having stored thereon a computer program operable on said processor, said processor implementing the steps of the electroencephalogram classification model training method according to any one of claims 1 to 7, or implementing the steps of the intent recognition method according to claim 8, when executing said computer program.

10. Computer readable storage medium, storing a computer program, characterized in that the computer program, when being executed by a processor, carries out the steps of the brain electrical signal classification model training method of any one of claims 1 to 7, or the steps of the intention recognition method of claim 8.