CN113791688B - Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface - Google Patents
Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface Download PDFInfo
- Publication number
- CN113791688B CN113791688B CN202111082146.1A CN202111082146A CN113791688B CN 113791688 B CN113791688 B CN 113791688B CN 202111082146 A CN202111082146 A CN 202111082146A CN 113791688 B CN113791688 B CN 113791688B
- Authority
- CN
- China
- Prior art keywords
- domain data
- matrix
- adaptive
- target domain
- source domain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 59
- 238000012360 testing method Methods 0.000 claims abstract description 41
- 239000013598 vector Substances 0.000 claims abstract description 32
- 230000009467 reduction Effects 0.000 claims abstract description 21
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 13
- 239000011159 matrix material Substances 0.000 claims description 94
- 230000003044 adaptive effect Effects 0.000 claims description 24
- 230000015654 memory Effects 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 15
- 238000012880 independent component analysis Methods 0.000 claims description 12
- 210000004556 brain Anatomy 0.000 claims description 11
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 230000005611 electricity Effects 0.000 claims description 4
- 230000008929 regeneration Effects 0.000 claims description 4
- 238000011069 regeneration method Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 claims description 2
- 210000001061 forehead Anatomy 0.000 claims description 2
- 210000003128 head Anatomy 0.000 claims description 2
- 238000002156 mixing Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 description 10
- 238000000513 principal component analysis Methods 0.000 description 4
- 238000013526 transfer learning Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000012706 support-vector machine Methods 0.000 description 3
- 206010008190 Cerebrovascular accident Diseases 0.000 description 2
- 208000006011 Stroke Diseases 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 201000007201 aphasia Diseases 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000002490 cerebral effect Effects 0.000 description 2
- 238000007635 classification algorithm Methods 0.000 description 2
- 230000006735 deficit Effects 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000005036 nerve Anatomy 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 210000005013 brain tissue Anatomy 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000003920 cognitive function Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 201000005264 laryngeal carcinoma Diseases 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000638 stimulation Effects 0.000 description 1
- 208000011580 syndromic disease Diseases 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2134—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on separation criteria, e.g. independent component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/027—Concept to speech synthesisers; Generation of natural phrases from machine-based concepts
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Neurosurgery (AREA)
- Multimedia (AREA)
- Acoustics & Sound (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- Neurology (AREA)
- General Health & Medical Sciences (AREA)
- Dermatology (AREA)
- Biomedical Technology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention discloses a method and a system for generating multisource domain self-adaptive imagination voice based on a brain-computer interface, which acquire source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the test set are electroencephalogram signals of known classification labels; target domain data, comprising: presetting an electroencephalogram signal of a pseudo tag; extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features; obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag. By reducing the edge distribution difference and the conditional distribution difference between the source domain and the target domain, the use has higher recognition accuracy when the number of source domains increases.
Description
Technical Field
The invention relates to the technical field of voice generation, in particular to a multi-source domain self-adaptive imagination voice generation method and system based on a brain-computer interface.
Background
The statements in this section merely relate to the background of the present disclosure and may not necessarily constitute prior art.
Aphasia refers to a disorder caused by pathological changes of brain tissues related to language functions of people, such as cerebral apoplexy, laryngeal carcinoma, locking syndrome and the like. Such diseases can cause impairment of the patient's ability to understand and express the human interaction symbology, especially speech, vocabulary, grammar, etc., language structure and language content and meaning, as well as impairment of the language-based cognitive processes and functions. Taking cerebral apoplexy as an example, although the condition of aphasia occurs in a patient, the language nerves of the brain of the patient are not diseased, in other words, the patient can generate voice nerve signals in the brain, and the possibility is provided for the language expression of the patient by combining the brain-computer interface technology which is currently used.
In recent years, imagined speech has become a new neuroparadigm based on internal stimulation of brain activity to achieve communication behavior without resorting to auditory language. Imagined speech includes the mental imagined pronunciation of phonemes, words or words, without producing any sound or pronunciation actions.
From previous experiments and demonstrations, electroencephalogram (EEG) is a technique suitable for imagining speech classification. Currently there is a rapid development in techniques for extracting features from EEG signals, the purpose of which is to construct a classifier for imagined speech. Conventional feature extraction and feature classification algorithms include co-space mode (CSP), support Vector Machine (SVM), and the like. The development of these techniques has been fairly well established at present, called classical algorithms in the field of brain-computer interfaces. However, one limitation of the prior art is that a large number of labeled samples cannot be collected. Traditional machine learning requires that training data and test data must be subject to independent co-distributed constraints, but the individual variability of the brain electrical signals is very strong, which limits the applicability of current classifiers to new subjects.
Disclosure of Invention
In order to solve the defects in the prior art, the invention provides a multisource domain self-adaptive imagination voice generation method and system based on a brain-computer interface; using a cross-discipline approach, new users are classified using a set of different discipline EEG data. On this basis, the transfer learning shows utility in solving the interdisciplinary approach of imagined speech. In the context of interdisciplinary, the goal of the transfer learning is to extract the same parts of interdisciplinary and apply this knowledge to new disciplines. However, for the transfer learning of Shan Yuanyu, if the similarity between the source domain and the target domain is poor, the transfer learning may generate a negative transfer phenomenon. So that multi-source domain migration learning can be employed to remedy this deficiency. The method provides an imagination voice generating algorithm, so that data of a plurality of patients can be used for new patients, the applicability of the algorithm is improved, and further development of imagination voice equipment is facilitated.
In a first aspect, the present invention provides a multi-source domain adaptive imagination speech generation method based on a brain-computer interface;
a multisource domain self-adaptive imagination voice generation method based on a brain-computer interface comprises the following steps:
acquiring source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag.
In a second aspect, the present invention provides a multi-source domain adaptive imagination speech generation system based on a brain-computer interface;
a brain-computer interface based multi-source domain adaptive imagination speech generation system comprising:
a preprocessing module configured to: acquiring source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
a feature extraction module configured to: extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
a speech generation module configured to: obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag.
In a third aspect, the present invention also provides an electronic device, including:
a memory for non-transitory storage of computer readable instructions; and
a processor for executing the computer-readable instructions,
wherein the computer readable instructions, when executed by the processor, perform the method of the first aspect described above.
In a fourth aspect, the invention also provides a storage medium storing non-transitory computer readable instructions, wherein the instructions of the method of the first aspect are executed when the non-transitory computer readable instructions are executed by a computer.
Compared with the prior art, the invention has the beneficial effects that:
(1) The applicability is stronger; the traditional brain-computer interface classification algorithm needs to measure and analyze the data of each patient independently, and the patients often need to acquire multiple groups of test data, and the data of each patient exists independently. The acquired data of other patients can be directly applied to a new patient for use according to the algorithm, so that the universality of the algorithm is greatly improved.
(2) The response speed is faster: in the multi-source domain self-adaptation field, the algorithm can rapidly obtain the classification result through fewer iteration times, thereby meeting the requirement of brain-computer interface imagination voice on algorithm instantaneity and providing better user experience for users.
(3) The accuracy is higher: the algorithm has higher recognition accuracy when the number of the source domains is increased by reducing the edge distribution difference and the condition distribution difference between the source domains and the target domains.
Additional aspects of the invention will be set forth in part in the description which follows, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
FIG. 1 is a flow chart of a method of a first embodiment;
fig. 2 is an adaptive matrix optimization method according to the first embodiment.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
Embodiments of the invention and features of the embodiments may be combined with each other without conflict.
All data acquisition in the embodiment is legal application of the data on the basis of meeting laws and regulations and agreements of users.
Example 1
The embodiment provides a multisource domain self-adaptive imagination voice generation method based on a brain-computer interface;
as shown in fig. 1 and 2, the multi-source domain adaptive imagination speech generating method based on brain-computer interface includes:
s101: acquiring source domain data and target domain data; preprocessing source domain data and target domain data;
the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
s102: extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
s103: obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag.
Further, the step S101: acquiring source domain data and target domain data; the method specifically comprises the following steps:
and acquiring EEG signals which the user imagines to sound by using an EEG electrode cap worn on the head of the user.
The user wears the EEG electrode cap to imagine in the brain, such as imagining pronunciation up, down, left, right, etc., and corresponding EEG signal is collected as X n (t) and store it in a database.
Further, the step S101: preprocessing source domain data and target domain data; the method specifically comprises the following steps:
eliminating artifacts of the brain electrical signal to be processed by adopting independent component analysis (Independent Component Analysis, ICA) and a hurst index;
the common average reference value CAR (common average reference) is adopted to calculate the average value of all the electrodes of the brain electric signal for eliminating the artifact, and the average value of all the electrodes is removed to improve the signal to noise ratio.
Further, the artifact of the brain electrical signal to be processed is eliminated by adopting independent component analysis (Independent Component Analysis, ICA) and a hurst index; the method specifically comprises the following steps:
first, the acquired EEG signal is decomposed by independent component analysis into a sum of temporally independent and spatially fixed components, the eye activity corresponding to the forehead She Buwei of the brain;
then, calculating the Hurst index of the space where the eye electricity is located, namely, the components with the Hurst index in the range of 0.58-0.69, evaluating all independent components of each channel by using the Hurst index, and mixing the rest components into the total amount of the whole channel after removing the eye electricity components to finish the artifact processing.
Because of the invalid information in the acquired electroencephalogram data that affects experimental accuracy, independent Component Analysis (ICA) and hurst index are employed to eliminate artifacts on the EEG signal, and then Common Average Reference (CAR) data is employed for preprocessing. Preprocessing the acquired electroencephalogram data, and improving the signal-to-noise ratio of the data so as to better filter the influence of noise on subsequent work.
Further, the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; among these, classification labels are known as "up", "down", "left" or "right" that the user imagines.
Further, the target domain data includes: presetting an electroencephalogram signal of a pseudo tag; wherein, preset pseudo tag, manual setting, for example: setting the pseudo tag to be left or right; the pseudo tag set at this time may be wrong, but the value of the pseudo tag is continuously corrected and gets closer to the true tag value through continuous optimization of the classifier and the training set.
Further, the step S102: performing dimension reduction treatment on the extracted features; the main component analysis (PrincipalComponents Analysis, PCA) is adopted for dimension reduction treatment.
Further, the step S103: obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; wherein the classifier is a support vector machine.
Further, the specific process includes the steps of:
s1031: aiming at the feature vectors of the training set obtained by the dimension reduction processing, the feature vectors are arranged into a matrix according to the sequence from the large to the small of the corresponding feature values, and the first k feature vectors are selected as a self-adaptive matrix A; performing iterative optimization on the adaptive matrix A to obtain an optimized adaptive matrix A;
s1032: based on the optimized self-adaptive matrix A, all the extracted features of the training set electroencephalogram signals and the corresponding labels of the training set electroencephalogram signals, all the extracted features of the test set electroencephalogram signals and the corresponding labels of the test set electroencephalogram signals, training the classifier to obtain corrected pseudo labels corresponding to the target domain data.
Further, the S1031: performing iterative optimization on the adaptive matrix A to obtain an optimized adaptive matrix A; the method specifically comprises the following steps:
s10311: performing feature decomposition on the data of the training set, sequencing all feature values obtained by the feature decomposition in a sequence from large to small, and selecting the first k feature vectors as a self-adaptive matrix A;
s10312: projecting all extracted features of the optimized self-adaptive matrix A and the training set electroencephalogram signals and corresponding labels of the training set electroencephalogram signals to a regeneration kernel Hilbert space, and classifying the projected data by using a classifier to update pseudo labels in a target domain;
s10313: calculating a conditional distribution coefficient matrix M c ；M c Is a coefficient matrix of the maximum mean difference MMD, which contains each X u ∈X s X is as follows t Category labels of (c); category labels refer to several categories in one data, e.g., one source data in a source domain contains imagined up, down, left, right four categories;
s10314: calculating a diagonal secondary gradient matrix G; the samples in the source domain are re-weighted to reduce the influence of uncorrelated samples on the prediction accuracy, and the weights of the various samples are included in the diagonal sub-gradient matrix G.
S10315: judging whether the iteration times reach the set times (10 times), if so, enabling the conditional distribution coefficient matrix and the diagonal sub-gradient matrix to reach a convergence state, and obtaining an optimized self-adaptive matrix; if not, return to S10311.
Further, S1032: training a classifier based on the optimized self-adaptive matrix A, all extracted features of the training set electroencephalogram signals and corresponding labels of the training set electroencephalogram signals, all extracted features of the test set electroencephalogram signals and corresponding labels of the test set electroencephalogram signals to obtain corrected pseudo labels corresponding to target domain data; the method specifically comprises the following steps:
s10321: projecting the data input by the maximum mean difference MMD to a regeneration kernel Hilbert space (Reproducing kernel Hilbert space, RKHS), and selecting a linear kernel function according to the mapping relation; obtaining a kernel matrix through linear kernel function calculation;
s10322: combining the edge distribution matrix and the conditional distribution coefficient matrix, and calculating the weight of the overall distribution of the data; obtaining a normalized Laplace matrix according to the weight of the edge distribution;
s10323: constructing a diagonal matrix; constructing parameters of a classifier according to the kernel matrix, the diagonal matrix and the normalized Laplace matrix;
s10324: setting a classifier according to parameters and kernel functions of the classifier;
s10325: training the classifier based on the optimized self-adaptive matrix A, all the extracted features of the training set electroencephalogram signals and the corresponding labels of the training set electroencephalogram signals;
based on the optimized self-adaptive matrix A, all the extracted features of the test set electroencephalogram signals and the corresponding labels of the test set electroencephalogram signals, testing the classifier;
and inputting the extracted characteristics of the target domain into a tested classifier to obtain the corrected pseudo tag corresponding to the target domain data.
Illustratively, the parameter representation is performed: dividing electroencephalogram data into source domain dataTarget domain data X t The method comprises the steps of carrying out a first treatment on the surface of the And set the source domain tag +.>Regularization parameters are expressed as lambda, sigma and gamma, and subspace is k;
source domain data X s The method comprises a training set and a testing set; target domain data X t Comprises a group of electroencephalogram signals to be processed. Will y t Setting the source domain sample use model as a target domain pseudo tag, and training the source domain sample use model; setting pseudo tag y using pseudo tag learning in semi-supervised domain t Training the source domain in the source domain sample, and defining the decision boundary of the source domain and the target domain.
Measuring the edge distribution difference between a target domain and a source domain by adopting the maximum mean value difference (maximum mean discrepancy, MMD) on the electroencephalogram, reducing the edge distribution difference between the target domain and the source domain, and improving the phase between the two domains; setting an intra-class scattering matrix S, and measuring a sample distance between the intra-class scattering matrix S and source domain data; performing dimensionality reduction on the data by adopting a Principal Component Analysis (PCA) technology, arranging eigenvectors into a matrix according to the corresponding eigenvalue from top to bottom in rows, and selecting the first k eigenvectors as an adaptive matrix A; projection-based source domain data { A } T X s ，y s Training a standard classifier f, updating a target label y as a target domain pseudo label t The method comprises the steps of carrying out a first treatment on the surface of the Updating a maximum average deviation matrixUpdating the secondary gradient matrix G; circularly running until the maximum average deviation matrix and the secondary gradient matrix are converged;
selecting a kernel function k (x i ，x j ) And calculating a kernel matrix K on the projection data; the method comprises the steps of learning a prediction function with good target domain performance through edge distribution, and determining the weight of an edge by using a weight function; constructing a diagonal matrix, determining a parameter alpha of the classifier, and further determining an adaptive classifier f; predicting the data of the target number through a constructed classifier to obtain a label of the target domain;
using MMD algorithm to measure the distribution difference between the source domain and the target domain, each X is calculated according to formula (1) u ∈X s And X t Is a matrix of MMD coefficients of (c).
Wherein, the liquid crystal display device comprises a liquid crystal display device,representing the number of samples of the ith source domain data, n t Representing the number of samples of the target domain data. />A nth dataset representing a source domain, X t Representing a target domain dataset, (M) 0 ) i，j Representing an edge distribution matrix.
To ensure differentiation between the different classes of each source domain in the training set, the intra-class variance is employed to measure a minimization measure between each predicted data and its mean. For all X u ∈X s Constructing compressed domain invariant clusters to preserve input data and at each X u Generates a clear decision domain between different classes as shown in equation (2), in whichIs of the X type u ∈X s S is set as shown in formula (3). A is that s Mapping matrix representing all source domain samples consisting of a plurality of source domains, A t Representing a mapping matrix of target domain samples. C represents the set of category numbers for one dataset. S denotes an intra-class scatter matrix for each c-class.
According to the formula (4), carrying out feature decomposition work on the data of the source domain, and carrying out all features after feature decomposition:
(XM c X T +S+λG)A＝XHX T Aφ (4)
the values are sorted in descending order, and the first k values are selected as the minimum eigenvectors and used as the adaptive matrix A. M is M c Representing a conditional distribution coefficient matrix, S representing an intra-class scatter matrix, G representing a diagonal sub-gradient matrix, lambda representing regularization parameters, A representing an adaptive matrix, XHX T Is the covariance matrix, phi is the mapping function of X→phi (X).
Source data { A after being reduced to k dimension T X s ，y s And (3) projecting, and then classifying the projected source data by using a standard classifier f to finish the updating of the pseudo tag.
Updating coefficient matrix M of conditional distribution MMD according to equation (5) c The coefficient matrix includes all X' s u ∈X s X is as follows t 。Is the source domain->Sample set of class c, +.>Is the target domain X t Class c sample sets. />Is corresponding to->Base of->Is->Is a base of (c).
Updating the diagonal sub-gradient matrix, G, in equation (4) ii The updated formula of (2) is as follows:
wherein a is i Is the element of the ith row and column of the original diagonal gradient matrix G.
Repeating the above steps until the coefficient matrix M is the maximum average c Diagonal sub-gradient matrix G ii Until convergence. Although the PCA dimension reduction technique has high efficiency, it cannot minimize the distribution difference between the source domain and the target domain, but the distribution difference across the domains is large. The above modules are employed to minimize the distribution differences between each source domain in the training set and the target data sink.
Selecting radial basis (Radial Basis function, RBF) kernel function k (x) according to mapping relation i ，x j ) Obtaining a kernel matrix K through calculation, wherein the formula (7) is as follows:
k(x i ，x j )＝exp(-γ||x i -x j || 2 ) (7)
wherein x is i ，x j ∈X[X s ，X t ]Gamma is the regularization parameter used for the calculation.
Then, the weight of the edge is calculated according to the formula (8) and the formula (9), and then the normalized Laplace matrix is obtained through the formula (10).
The diagonal matrix R and the classifier parameters α are constructed according to equation (11) and equation (12).
Wherein σ and γ are regularization parameters for computation, R is R even if the main diagonal is R ii Is a diagonal matrix of the (a),is a normalized Laplace matrix, K is a kernel matrix, Y is a kernel matrix containing [ Y ] s ，y t ]Is a matrix of (a) in the matrix.
After the parameters are determined, training a classifier according to a formula (13), and performing classification prediction on the target domain by using an adaptive classifier f to determine a target domain label so as to finish identification.
Wherein alpha is i ∈α，k(x i X) represents a radial basis function.
Example two
The embodiment provides a multisource domain self-adaptive imagination voice generation system based on a brain-computer interface;
a brain-computer interface based multi-source domain adaptive imagination speech generation system comprising:
a preprocessing module configured to: acquiring source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
a feature extraction module configured to: extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
a speech generation module configured to: obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag.
It should be noted that the preprocessing module, the feature extraction module, and the speech generation module correspond to steps S101 to S103 in the first embodiment, and the modules are the same as examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the first embodiment. It should be noted that the modules described above may be implemented as part of a system in a computer system, such as a set of computer-executable instructions.
The foregoing embodiments are directed to various embodiments, and details of one embodiment may be found in the related description of another embodiment.
The proposed system may be implemented in other ways. For example, the system embodiments described above are merely illustrative, such as the division of the modules described above, are merely a logical function division, and may be implemented in other manners, such as multiple modules may be combined or integrated into another system, or some features may be omitted, or not performed.
Example III
The embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is coupled to the memory, the one or more computer programs being stored in the memory, the processor executing the one or more computer programs stored in the memory when the electronic device is running, to cause the electronic device to perform the method of the first embodiment.
It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate array FPGA or other programmable logic device, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may include read only memory and random access memory and provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store information of the device type.
In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software.
The method in the first embodiment may be directly implemented as a hardware processor executing or implemented by a combination of hardware and software modules in the processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method. To avoid repetition, a detailed description is not provided herein.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Example IV
The present embodiment also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, perform the method of embodiment one.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. The multi-source domain self-adaptive imagination voice generation method based on the brain-computer interface is characterized by comprising the following steps:
acquiring source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; according to the corrected pseudo tag, generating a corresponding voice signal, wherein the specific process comprises the following steps:
aiming at the feature vectors of the training set obtained by the dimension reduction processing, the feature vectors are arranged into a matrix according to the sequence from the large to the small of the corresponding feature values, and the first k feature vectors are selected as a self-adaptive matrix A; performing iterative optimization on the adaptive matrix A to obtain an optimized adaptive matrix A; training a classifier based on the optimized self-adaptive matrix A, all extracted features of the training set electroencephalogram signals and corresponding labels of the training set electroencephalogram signals, all extracted features of the test set electroencephalogram signals and corresponding labels of the test set electroencephalogram signals to obtain corrected pseudo labels corresponding to target domain data;
performing iterative optimization on the adaptive matrix A to obtain an optimized adaptive matrix A; the method specifically comprises the following steps:
(1) Performing feature decomposition on the data of the training set, sequencing all feature values obtained by the feature decomposition in a sequence from large to small, and selecting the first k feature vectors as a self-adaptive matrix A;
(2) Projecting all extracted features of the self-adaptive matrix A and the training set electroencephalogram signals and corresponding labels of the training set electroencephalogram signals to a regeneration kernel Hilbert space, and classifying the projected data by using a classifier to update pseudo labels in a target domain;
(3) Calculating a conditional distribution coefficient matrix；
(4) Calculating a diagonal secondary gradient matrix G;
(5) Judging whether the iteration times reach the set times, if so, enabling the conditional distribution coefficient matrix and the diagonal sub-gradient matrix to reach a convergence state, and obtaining an optimized self-adaptive matrix; if not, returning to (1);
training a classifier based on the optimized self-adaptive matrix A, all extracted features of the training set electroencephalogram signals and corresponding labels of the training set electroencephalogram signals, all extracted features of the test set electroencephalogram signals and corresponding labels of the test set electroencephalogram signals to obtain corrected pseudo labels corresponding to target domain data; the method specifically comprises the following steps:
projecting the data input by the maximum mean difference MMD to a regeneration kernel Hilbert space, and selecting a linear kernel function according to the mapping relation;
obtaining a kernel matrix through linear kernel function calculation;
combining the edge distribution matrix and the conditional distribution coefficient matrix, and calculating the weight of the overall distribution of the data;
obtaining a normalized Laplace matrix according to the weight of the edge distribution; constructing a diagonal matrix;
constructing parameters of the classifier according to the kernel matrix, the diagonal matrix and the normalized Laplace matrix：
；
；
Classifier parameters areAnd->Is a regularization parameter for calculation, R is the main diagonal is +.>Diagonal matrix of>Is a normalized Laplace matrix, K is a kernel matrix, Y is a kernel matrix containing +.>Matrix of->Target tag being a pseudo tag of a target domain +.>Is a source domain label; />Is a source domain dataset;
setting a classifier according to parameters and kernel functions of the classifier; training the classifier based on the optimized self-adaptive matrix A, all the extracted features of the training set electroencephalogram signals and the corresponding labels of the training set electroencephalogram signals;
based on the optimized self-adaptive matrix A, all the extracted features of the test set electroencephalogram signals and the corresponding labels of the test set electroencephalogram signals, testing the classifier;
and inputting the extracted characteristics of the target domain into a tested classifier to obtain the corrected pseudo tag corresponding to the target domain data.
2. The brain-computer interface-based multi-source domain adaptive imagination speech generation method according to claim 1, wherein source domain data and target domain data are acquired; the method specifically comprises the following steps:
and acquiring EEG signals which the user imagines to sound by using an EEG electrode cap worn on the head of the user.
3. The brain-computer interface-based multi-source domain adaptive imagination speech generation method according to claim 1, wherein source domain data and target domain data are preprocessed; the method specifically comprises the following steps:
adopting independent component analysis and a Hurst index to eliminate artifacts of the brain electrical signals to be processed;
and calculating the average value of all the electrodes of the brain electrical signal without the artifact by adopting a common average reference value, and improving the signal to noise ratio by removing the average value of all the electrodes.
4. The brain-computer interface-based multi-source domain adaptive imagination speech generation method according to claim 3, wherein artifacts of the brain-electrical signal to be processed are eliminated by adopting independent component analysis and a hurst index; the method specifically comprises the following steps:
first, the acquired EEG signal is decomposed by independent component analysis into a sum of temporally independent and spatially fixed components, the eye activity corresponding to the forehead She Buwei of the brain;
then, calculating the Hurst index of the space where the eye electricity is located, namely, the components with the Hurst index in the range of 0.58-0.69, evaluating all independent components of each channel by using the Hurst index, and mixing the rest components into the total amount of the whole channel after removing the eye electricity components to finish the artifact processing.
5. A brain-computer interface based multi-source domain adaptive imagination speech generation system for implementing the brain-computer interface based multi-source domain adaptive imagination speech generation method according to any one of claims 1 to 4, characterized by comprising:
a preprocessing module configured to: acquiring source domain data and target domain data; preprocessing source domain data and target domain data; the source domain data includes: a training set and a testing set; the training set and the testing set are electroencephalogram signals of known classification labels; the target domain data includes: presetting an electroencephalogram signal of a pseudo tag;
a feature extraction module configured to: extracting features of the preprocessed source domain data and the preprocessed target domain data; performing dimension reduction treatment on the extracted features;
a speech generation module configured to: obtaining corrected pseudo labels corresponding to the target domain data according to the training set feature vectors, the test set feature vectors, the feature vectors of the electroencephalogram signals corresponding to the known pseudo labels and the classifier which are obtained through the dimension reduction processing; and generating a corresponding voice signal according to the corrected pseudo tag.
6. An electronic device, comprising:
a memory for non-transitory storage of computer readable instructions; and
a processor for executing the computer-readable instructions,
wherein the computer readable instructions, when executed by the processor, perform the method of any of the preceding claims 1-4.
7. A storage medium, characterized by non-transitory storing computer-readable instructions, wherein the instructions of the method of any one of claims 1-4 are performed when the non-transitory computer-readable instructions are executed by a computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111082146.1A CN113791688B (en) | 2021-09-15 | 2021-09-15 | Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111082146.1A CN113791688B (en) | 2021-09-15 | 2021-09-15 | Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113791688A CN113791688A (en) | 2021-12-14 |
| CN113791688B true CN113791688B (en) | 2023-09-26 |
Family
ID=78878553
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111082146.1A Active CN113791688B (en) | 2021-09-15 | 2021-09-15 | Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113791688B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348579A (en) * | 2019-05-28 | 2019-10-18 | 北京理工大学 | A kind of domain-adaptive migration feature method and system |
| CN111695456A (en) * | 2020-05-28 | 2020-09-22 | 西安工程大学 | Low-resolution face recognition method based on active discriminability cross-domain alignment |
| CN112488081A (en) * | 2020-12-23 | 2021-03-12 | 杭州电子科技大学 | Electroencephalogram mental state detection method based on DDADSM (distributed denial of service) cross-test transfer learning |
| CN112684891A (en) * | 2020-12-30 | 2021-04-20 | 杭州电子科技大学 | Electroencephalogram signal classification method based on multi-source manifold embedding migration |
-
2021
- 2021-09-15 CN CN202111082146.1A patent/CN113791688B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348579A (en) * | 2019-05-28 | 2019-10-18 | 北京理工大学 | A kind of domain-adaptive migration feature method and system |
| CN111695456A (en) * | 2020-05-28 | 2020-09-22 | 西安工程大学 | Low-resolution face recognition method based on active discriminability cross-domain alignment |
| CN112488081A (en) * | 2020-12-23 | 2021-03-12 | 杭州电子科技大学 | Electroencephalogram mental state detection method based on DDADSM (distributed denial of service) cross-test transfer learning |
| CN112684891A (en) * | 2020-12-30 | 2021-04-20 | 杭州电子科技大学 | Electroencephalogram signal classification method based on multi-source manifold embedding migration |
Non-Patent Citations (2)
| Title |
|---|
| A Novel Domain Adaptation-Based Intelligent Fault Diagnosis Model to Handle Sample Class Imbalanced Problem;Zhongwei Zhang 等;《sensors》;正文第3节 * |
| Standardization-refinement domain adaptation method for cross-subject EEG-based classification in imagined speech recognition;Magdiel Jiménez-Guarneros 等;《Pattern Recognition Letters》;正文第3-4节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113791688A (en) | 2021-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ghazal et al. | Alzheimer disease detection empowered with transfer learning | |
| Özseven | Investigation of the effect of spectrogram images and different texture analysis methods on speech emotion recognition | |
| Cui et al. | Automatic sleep stage classification based on convolutional neural network and fine-grained segments | |
| Jiang et al. | Recognition of epileptic EEG signals using a novel multiview TSK fuzzy system | |
| Rieg et al. | High-performance detection of alcoholism by unfolding the amalgamated EEG spectra using the Random Forests method | |
| Valizadeh et al. | Decrypting the electrophysiological individuality of the human brain: Identification of individuals based on resting-state EEG activity | |
| Jayarathne et al. | Person identification from EEG using various machine learning techniques with inter-hemispheric amplitude ratio | |
| CN115662576B (en) | Method and system for generating neurofeedback training paradigm of associated cognitive disorder conditions | |
| Soni et al. | Graphical representation learning-based approach for automatic classification of electroencephalogram signals in depression | |
| Agarwal et al. | Classification of alcoholic and non-alcoholic EEG signals based on sliding-SSA and independent component analysis | |
| Kumar et al. | Comparison of Machine learning models for Parkinson’s Disease prediction | |
| Latha et al. | Brain tumour detection using neural network classifier and kmeans clustering algorithm for classification and segmentation | |
| CN113791688B (en) | Multi-source domain self-adaptive imagination voice generation method and system based on brain-computer interface | |
| Özbeyaz et al. | Familiar/unfamiliar face classification from EEG signals by utilizing pairwise distant channels and distinctive time interval | |
| Mahanand et al. | An enhanced effect-size thresholding method for the diagnosis of Autism spectrum disorder using resting state functional MRI | |
| Mahmud et al. | Decoding categorical speech perception from evoked brain responses | |
| Oliva et al. | The use of one-class classifiers for differentiating healthy from epileptic EEG segments | |
| CN114626408A (en) | Electroencephalogram signal classification method and device, electronic equipment, medium and product | |
| Shekokar et al. | Epileptic seizure classification using LSTM | |
| Gorur et al. | The single-channel dry electrode SSVEP-based biometric approach: data augmentation techniques against overfitting for RNN-based deep models | |
| Sinha et al. | Validation of stimulus for EEG signal based cognitive load analysis | |
| Gupta et al. | On-line EEG denoising and cleaning using correlated sparse signal recovery and active learning | |
| Huang et al. | A two-phase model for sleep staging using single channel eeg | |
| Mussabayeva et al. | Event-Related Spectrogram Representation of EEG for CNN-Based P300 Speller | |
| Talbot et al. | Estimating a brain network predictive of stress and genotype with supervised autoencoders |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |