CN115565540B - Invasive brain-computer interface Chinese pronunciation decoding method - Google Patents
Invasive brain-computer interface Chinese pronunciation decoding method Download PDFInfo
- Publication number
- CN115565540B CN115565540B CN202211545924.0A CN202211545924A CN115565540B CN 115565540 B CN115565540 B CN 115565540B CN 202211545924 A CN202211545924 A CN 202211545924A CN 115565540 B CN115565540 B CN 115565540B
- Authority
- CN
- China
- Prior art keywords
- hyperbolic
- data
- chinese pronunciation
- space
- computer interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013528 artificial neural network Methods 0.000 claims abstract description 33
- 210000002569 neuron Anatomy 0.000 claims abstract description 32
- 238000007477 logistic regression Methods 0.000 claims abstract description 26
- 238000012549 training Methods 0.000 claims abstract description 20
- 230000001360 synchronised effect Effects 0.000 claims abstract description 9
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000012216 screening Methods 0.000 claims abstract description 7
- 238000004364 calculation method Methods 0.000 claims description 11
- 210000004556 brain Anatomy 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 7
- 238000010304 firing Methods 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 230000017105 transposition Effects 0.000 claims description 2
- 238000002372 labelling Methods 0.000 abstract 1
- 230000006870 function Effects 0.000 description 15
- 230000001537 neural effect Effects 0.000 description 9
- 239000013598 vector Substances 0.000 description 9
- 238000005457 optimization Methods 0.000 description 5
- 238000007792 addition Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 210000000337 motor cortex Anatomy 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000001755 vocal effect Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 201000007201 aphasia Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000035484 reaction time Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
- G10L19/16—Vocoder architecture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
- G10L15/063—Training
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/16—Speech classification or search using artificial neural networks
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
- G10L2015/025—Phonemes, fenemes or fenones being the recognition units
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Theoretical Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Signal Processing (AREA)
- Machine Translation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an invasive brain-computer interface Chinese pronunciation decoding method, which comprises the following steps: screening effective neurons from the electroencephalogram data, removing the neurons with high similarity, and labeling the electroencephalogram data by using synchronous audio data after standardization; according to the characteristics of the Chinese pronunciation electroencephalogram data, the electroencephalogram data are projected into a hyperbolic space; constructing an effective hyperbolic neural network and a hyperbolic multiple logistic regression classifier to classify Chinese phonemes in the electroencephalogram data; in the training process, a certain number of triples are extracted from training data, the losses of hierarchical clustering are calculated for the triples based on the output characteristics of a network, and the triples are added into a total loss function to be optimized according to a certain weight; decoding by using the trained hyperbolic neural network and the hyperbolic multiple logistic regression classifier. The method better utilizes the structural characteristics of the Chinese pronunciation electroencephalogram data by introducing the hyperbolic space and the hyperbolic decoding method, and effectively improves the classification decoding performance of the Chinese pronunciation electroencephalogram data.
Description
Technical Field
The invention relates to the field of electroencephalogram data decoding, in particular to an invasive brain-computer interface Chinese pronunciation decoding method.
Background
The invasive brain-computer interface utilizes the high resolution intra-cortical brain electrical signals recorded by the invasive electrodes to identify the state and intent of the brain, thereby assisting the clinical patient in performing a variety of different tasks. In recent years, the application and research of an invasive brain-computer interface on voice are rapidly developed. Advanced speech brain-computer interfaces have enabled direct speech synthesis or decoding of speech phonemes, words and sentences from brain electrical signals, which means that invasive speech brain-computer interfaces have great potential for restoring the communication ability of aphasia patients.
In general, the speech brain-computer interface regards pronunciation as a motion process, and decodes neural signals into speech by decoding oral pronunciation kinematics as an intermediate link. One way is to convert the electroencephalogram signals recorded from the motor cortex into oral vocal actions during speaking, and then convert the corresponding oral vocal actions into voice. With the help of machine learning methods such as deep networks, some speech brain-computer interfaces tend to learn decoders in an end-to-end manner, generating speech waveforms directly from brain electrical signals.
For example, chinese patent publication No. CN111681636A discloses a method for generating a technical term sound based on a brain-computer interface, which includes collecting electroencephalogram signals reflecting brain activity information, external audio signals and video image signals, extracting features, performing nonlinear computation and learning through a plurality of neural networks, adding external context information and feedback input, directly decoding intentions and language contents expressed by the brain from the brain signals, and finally completing speech generation through a neural network, thereby realizing speech generation of the brain-computer interface technology.
However, decoding speech directly from neural signals faces the problem of vocabulary limitation. This is time consuming because the test needs to repeatedly speak words in the vocabulary for decoder training before the speech brain-computer interface is constructed. On the other hand, phonemes are basic sound units in pronunciation. Typically, the number of phonemes is much smaller than the number of words. Through accurate recognition of phonemes, free decoding of words after combination is expected. But it is difficult to accurately decode the speech phonemes from the neural signal. From a kinematic process perspective, the kinematics associated with speech is a combination of orofacial movements, including lips, tongue, chin, and other joints. Therefore, the phonemes with similar kinematics are often mixed up and difficult to distinguish, and the overall classification performance of the phonemes is reduced. How to accurately decode speech phonemes from neural signals remains a challenging problem.
More importantly, the application and research of the brain-computer interface aiming at Chinese pronunciation do not exist, and how to design an algorithm aiming at the Chinese pronunciation characteristics is realized, so that the good classification and decoding performance is realized, and then the high-efficiency voice brain-computer interface is constructed and is still in a blank state at present.
Disclosure of Invention
The invention provides an invasive brain-computer interface Chinese pronunciation decoding method, which can effectively improve the classification decoding performance of Chinese pronunciation electroencephalogram data.
An invasive brain-computer interface Chinese pronunciation decoding method comprises the following steps:
(1) Collecting electroencephalogram data of Chinese pronunciation and synchronous audio data, screening effective neurons from the electroencephalogram data, removing the neurons with high similarity, and standardizing the electroencephalogram data; marking the time node of the sound production on the electroencephalogram data by using the synchronous audio data, and intercepting data segments with fixed window length, wherein each data segment corresponds to a Chinese phoneme;
(2) Projecting the electroencephalogram data processed in the step (1) into a hyperbolic space, and forming training data by the electroencephalogram data and corresponding Chinese phonemes in the hyperbolic space;
(3) Constructing a hyperbolic neural network and a hyperbolic multiple logistic regression classifier; the hyperbolic neural network is used for extracting the features of the electroencephalogram data in the hyperbolic space, and the hyperbolic multiple logistic regression classifier is used for classifying Chinese phonemes for the features of the electroencephalogram data;
(4) Training a hyperbolic neural network and a hyperbolic multiple logistic regression classifier;
in the training process, a certain number of triples are extracted from training data, the loss of hierarchical clustering is calculated for the triples on the basis of the output characteristics of the hyperbolic neural network, and the triples are added into a total loss function to be optimized according to a certain weight;
(5) And projecting the electroencephalogram data to be decoded to a hyperbolic space, and then sequentially inputting the data to the trained hyperbolic neural network and hyperbolic multiple logistic regression classifier to obtain the decoded Chinese phoneme classification.
According to the hierarchical classification structure of the phonemes in the Chinese pronunciation and the hierarchy of the pronunciation position and the pronunciation mode in the Chinese pronunciation electroencephalogram, the hyperbolic neural network is constructed to better learn the characteristics of the Chinese pronunciation electroencephalogram, and the logit vector is obtained through the hyperbolic multiple logistic regression classifier. Meanwhile, hierarchical clustering constraint is performed on the logit vectors, and the model is encouraged to better mine the hierarchical structure of the data, so that better representation is learned, and the classification decoding performance of the Chinese pronunciation electroencephalogram data is effectively improved.
Preferably, in step (1), the neurons are screened by off-line screening. The method for screening effective neurons from electroencephalogram data and removing the neurons with high similarity specifically comprises the following steps:
carrying out spike potential classification, extracting the issuance of all neurons in an electroencephalogram signal, and drawing a waveform; visually inspecting the firing waveform of each neuron, and reserving neurons with obvious waveforms and a total firing frequency of more than 100; cosine similarity is calculated for the issuance of different neurons, and when the similarity degree of a plurality of neurons is greater than 0.7, only one neuron is reserved so as to reduce the influence of crosstalk on data quality.
When the data are normalized, the original value is subtracted by the mean value and then divided by the standard deviation, so that the obtained data meet the normal distribution that the mean value is 0 and the standard deviation is 1.
Preferably, the sound-emitting time node is marked by synchronous audio data, and the data segment with the window of [ -500ms, +1500ms ] is intercepted and used for subsequent training and verification by taking the sound-emitting time node as the center.
In the step (2), a Poincare disc model is adoptedTo project the brain electrical data into the hyperbolic space:
wherein,representing a hyperbolic space with curvature c and dimension d; />Represents a data point, <' > is>Represents a Euclidean real number space of dimension d, <' > H>Represents->Is greater than or equal to>And &>Represents a Euclidean metric and a hyperbolic metric, respectively>Representing the conformities of both metrics.
In the step (3), the hyperbolic neural network is expressed as:
wherein,and &>Respectively representing a hyperbolic neural network function and an Euclidean neural network function, and>andrespectively representing an exponential transformation and a logarithmic transformation at the origin,crepresents the curvature of a hyperbolic space, and>represents a data point, <' > is>Represents->Absolute value of (a).
In step (3), when the hyperbolic multiple logistic regression classifier classifies Chinese phonemes, z classes are given, and probability calculation formulas of different classes are as follows:
wherein,and &>Is a parameter of hyperbolic multiple logistic regression>A conformal factor representing the classification boundary of the class z,represents an inverse hyperbolic sine function, -is>Denotes an exponential function based on a natural constant e>Represents->The absolute value of (a);representing a Mobius addition operation; c represents the curvature of a hyperbolic space; />Representing an inner product operation.
In the step (4), the formula of the total loss function is as follows:
wherein,represents a classification loss, is>Representing hierarchical clustering loss; />And &>Are coefficients that balance the two parts of the loss function.
The classification loss is calculated as follows:
wherein,is/>Is selected based on the category label, <' > is selected>Is after softmax->Is greater than the log probability of->The data amount of mini-batch is shown.
The hierarchical clustering loss is calculated as follows:
wherein,represents a normalized softmax function; />Representing triples extracted from training data; />Indicates pickin a triple>Is connected to the minimum common ancestor node, </or>Indicates pickin a triple>The smallest common ancestor node of (a),indicates pickin a triple>The smallest common ancestor node of (c); />Representing a hyperbolic distance to a center of a hyperbolic space; />Representing £ in a triplet>Hyperbolic similarity between->Indicates pickin a triple>Hyperbolic similarity between->Representing £ in a triplet>Hyperbolic similarity between them; />Representing a matrix transposition.
When hyperbolic similarity calculation is carried out, a certain number of triples are sampled by using a random sampling methodCalculating a hyperbolic distance->Divided by the sum of three>Is normalized to obtain->The degree of similarity is expressed as->。
And when the hierarchical clustering loss is calculated, selecting a logic layer of the hyperbolic multiple logistic regression classifier to perform triple sampling and hierarchical clustering.
Compared with the prior art, the invention has the following beneficial effects:
the hyperbolic neural network is applied to the classification and decoding of the Chinese pronunciation electroencephalogram signals, the neural representation of the Chinese pronunciation is classified in a hyperbolic space, the hierarchical characteristics of the Chinese pronunciation and the signal representation are considered, and the hierarchical structure of the neural representation of the phoneme is restrained by the hierarchical clustering loss. The result proves that the model learns the interpretable hierarchical phoneme embedding from the electroencephalogram signal, and the phoneme decoding performance is obviously improved.
Drawings
FIG. 1 is a timing diagram illustrating an experimental paradigm of a data set in accordance with an embodiment of the present invention.
FIG. 2 is a spike issuance visualization diagram after different Chinese initials of the data set are grouped according to the utterance position in the embodiment of the present invention.
FIG. 3 is a graph comparing the classification accuracy of treatments with and without the inventive method.
FIG. 4 is a comparison graph of the distribution obtained after two-dimensional multiple logistic regression classification boundary visualization learned using the method of the present invention and not using the method of the present invention.
Detailed Description
The invention will be described in further detail below with reference to the drawings and examples, which are intended to facilitate the understanding of the invention without limiting it in any way.
In the data collection phase, this example was used to collect neural signals from the left major motor cortex of a paralyzed patient by implanting two 96-channel Utah intracortical microelectrode arrays (Blackrock Microsystems, salt Lake City, UT, USA) to record the neural signals. Neural signals were sampled at 30kHz using the Neuroport system (NSP, blackrock Microsystems) with two 96-channel Utah intracortical microelectrode arrays. During the experiment, the audio signal was recorded simultaneously with a microphone placed in front of the patient. The audio signal is digitized by the NeuroPort system through an analog input port at 30 khz. The embodiment designs three tasks aiming at Chinese pronunciation: 21 different initial pronunciation tasks, 24 different final pronunciation tasks and 20 different Chinese character pronunciation tasks. An experimental paradigm for data acquisition is shown in fig. 1. Specifically, in each trial, the subject was asked to view a red phoneme prompt on a computer screen one meter in front of him and hear an audible prompt for that phoneme. After one second the phoneme on the screen turns green indicating the start of the "start" phase and is tried to be spoken subsequently with the prompt phoneme. To ensure that the test has had sufficient reaction time to complete the test, the "start" phase lasts 3 seconds. After the "start" phase is ended, the recording of one of the dials is complete, and the recording of the next dial then begins.
The invention provides an invasive brain-computer interface Chinese pronunciation decoding method, which specifically comprises the following steps:
step 1, preprocessing electroencephalogram data.
Designing an experimental paradigm of Chinese pronunciation, and acquiring electroencephalogram data and synchronous audio data of the Chinese pronunciation; screening effective neurons from the electroencephalogram data, removing the neurons with high similarity, standardizing the data, marking the electroencephalogram data by using synchronous audio data, and then intercepting a data segment with a proper window length to obtain the preprocessed electroencephalogram data.
Specifically, spike classification (spike classification) is performed first, all neurons in the electroencephalogram signal are extracted, and a waveform is drawn. The firing waveform for each neuron was visually examined, and neurons with significant waveforms present with a total firing count greater than 100 were retained. And (4) issuing different neurons to calculate cosine similarity, and only keeping one neuron when the similarity of a plurality of neurons is greater than 0.7.
When the data are normalized, the original value is subtracted by the mean value and then divided by the standard deviation, so that the obtained data meet the normal distribution that the mean value is 0 and the standard deviation is 1.
And then marking the sounding time node by using the synchronous audio data, and intercepting a data segment taking the [ -500ms, +1500ms ] as a window for subsequent training and verification by taking the sounding time node as a center.
And 2, projecting the electroencephalogram data into a hyperbolic space.
Hyperbolic space is a non-european space of everywhere negative curvature. In a hyperbolic space, the farther from the center of the space, the greater the curvature and the greater the degree of spatial curvature. This means that hyperbolic spaces are well suited for modeling data having a tree structure or hierarchy: the number of nodes of the tree grows exponentially with the depth of the tree. After the Chinese pronunciation electroencephalogram signals are subjected to visual analysis, as shown in fig. 2, the Chinese pronunciation electroencephalogram signals can be found to have a certain hierarchical structure, and the hierarchical structure is related to a sound production mode and a sound production position. This means that hyperbolic space can be used to model chinese phonation brain electrical signals.
The example uses the presently most common and best-performing hyperbolic space model: poincare disc modelTo project the brain electrical data into the hyperbolic space:
wherein c represents the curvature of a hyperbolic space,dwhat is represented is the dimension of a hyperbolic space,expressed are Euclidean measures and hyperbolic measures, respectively>Conformal factors for both metrics are shown.
And 3, constructing a hyperbolic neural network to extract features, and classifying the Chinese pronunciation by using a hyperbolic multiple logistic regression classifier.
Constructing a reasonable network structure according to the characteristics of small data volume and large dimensionality of Chinese pronunciation electroencephalogram; the hyperbolic neural network is a version of an Euclidean space neural network vector and matrix computing operation executed in a hyperbolic space. Since the vector and matrix calculation operations are too complex to be performed in the non-european space, the tangent space of the hyperbolic space is required to be used for the approximation operation. The tangent space of the hyperbolic space has the property of an Euclidean space, so that data only needs to be projected onto the tangent space, and after the calculation operation of vectors and matrixes is executed in the tangent space, the data is projected back to the Euclidean space. Here, the conversion between the tangent space and the original space needs to be completed by using exponential transformation and logarithmic transformation in the gyro vector space. In this way, a representation of the hyperbolic neural network can be obtained:
wherein,and &>Respectively representing a hyperbolic neural network function and an Euclidean neural network function, and>andrespectively, an exponential transformation and a logarithmic transformation at the origin, and c represents the curvature of a hyperbolic space.
Considering that the data volume of the Chinese pronunciation electroencephalogram signal is less, in the practical construction, a hyperbolic neural network structure with 2 layers is selected, and the number of neurons is respectively as follows: 256, 128.
Similar to the hyperbolic neural network, the hyperbolic multiple logistic regression is also a version of the european multiple logistic regression that performs operations in the hyperbolic space.
In particular, givenThe method comprises the following steps that (1) the logit probabilities of different types of samples are obtained by a hyperbolic multiple logistic regression method, and the specific calculation mode is as follows:
wherein,and &>Is a parameter of hyperbolic multiple logistic regression>A conformal factor representing the classification boundary of class z,represents an inverse hyperbolic sine function, -is>Denotes an exponential function based on a natural constant e>Represents->Absolute value of (d);representing a Mobius addition operation; c represents the curvature of a hyperbolic space; />Representing an inner product operation.
The Mobius addition operation is an operation method of a gyro vector space, and is obtained by derivation through exponential transformation and logarithmic transformation, and the specific calculation method is as follows:
and 4, training the hyperbolic neural network and the hyperbolic multiple logistic regression classifier.
And in the optimization process, a hyperbolic RSGD method is used for parameter optimization and updating. And considering the small data volume of the electroencephalogram signals, training and testing the model by using a leave-one-out method. Only one data is used as a test set at a time, and the rest is all used as a training set. In the training process, a hierarchical clustering constraint characteristic representation learning method based on triple similarity is added into the hyperbolic neural network.
(4-1) selecting a proper similarity calculation method: considering that the hyperbolic model is used to extract features, the hyperbolic distance can be directly adopted to calculate the similarity.
Given two points on a poincare discThe hyperbolic distance between two points is calculated as follows:
Sampling triples with the number of 20-50 by using a random sampling methodCalculating hyperbolic distances between each otherDivided by the sum of the three>Is normalized to obtain>The similarity can be expressed as。
(4-2) selecting a proper clustering position: the hierarchical clustering loss is directly calculated for the logit vector, the hierarchical clustering loss is added into the overall loss needing to be optimized according to a certain weight, and meanwhile, the classification and clustering targets are optimized, so that the following overall loss function can be obtained
Wherein,represents a classification loss, is>Representing hierarchical clustering penalties. />And &>Are coefficients that balance the two parts of the loss function.
For multi-class classification tasks, givenNumber of samples->Belongs to>A category, and a corresponding labelWherein->. Classification loss pick-up or-and-place>Can be represented by the following formula
For hierarchical clustering loss, specifically, a certain number of triples are randomly sampled from data, and the hierarchical clustering loss is calculated based on the triples, and the goal of the loss is to enable nodes with higher similarity in a hierarchical clustering tree to be merged earlier, and the specific calculation is as follows:
whereinRepresents a normalized softmax function, <' > based on the value of the sum>Represents triples extracted from data, representing data in a particular grouping field>Indicates pickin a triple>In the cluster, is based on the smallest common ancestor node in the cluster>Represents a hyperbolic distance to the center of the hyperbolic space, is>Representing £ in a triplet>Hyperbolic similarity between them.
And selecting to perform triple sampling and hierarchical clustering at a locality layer for simultaneously optimizing clustering and classification.
And 5, testing and applying the hyperbolic neural network and the hyperbolic multiple logistic regression classifier.
After the training is finished, whether the classification result of the data is correct is tested. And after all tests are finished, dividing the total test accuracy number by the total data volume to obtain a classification accuracy rate.
For comparison, the feature learning framework provided by the invention has the best effect in the hyperbolic space, the same network structure is used for carrying out experiments by using three different spatial metrics on the same data set, and the obtained comparison result is shown in fig. 3, wherein three sub-graphs respectively represent classification results of 21 Chinese initial pronunciations, 24 Chinese final pronunciations and 20 Chinese character pronunciations, and the performance of the framework in the hyperbolic space is obviously superior to that in the Euclidean space and the spherical space.
In order to illustrate that the learning framework can mine the potential hierarchy of data and learn characteristics with more voice characteristics, visual analysis is carried out on the multivariate logistic regression classification boundaries learned by the network, for example, as shown in fig. 4, the left sub-graph is the classification boundary added with hierarchical clustering optimization, the right sub-graph represents the classification boundary without hierarchical clustering optimization, different colors represent Chinese initials of different classes, and it can be seen that the learned classification boundaries are more dispersed after the hierarchical clustering optimization is added, and the classification boundaries of the initials at the same sounding position show aggregation.
The embodiments described above are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only specific embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions and equivalents made within the scope of the principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. An invasive brain-computer interface Chinese pronunciation decoding method is characterized by comprising the following steps:
(1) Collecting electroencephalogram data of Chinese pronunciation and synchronous audio data, screening effective neurons from the electroencephalogram data, removing the neurons with high similarity, and standardizing the electroencephalogram data; marking the time node of the sound production on the electroencephalogram data by using the synchronous audio data, and intercepting data segments with fixed window length, wherein each data segment corresponds to a Chinese phoneme;
(2) Projecting the electroencephalogram data processed in the step (1) into a hyperbolic space, and forming training data by the electroencephalogram data and corresponding Chinese phonemes in the hyperbolic space;
(3) Constructing a hyperbolic neural network and a hyperbolic multiple logistic regression classifier; the hyperbolic neural network is used for extracting the features of the electroencephalogram data in the hyperbolic space, and the hyperbolic multiple logistic regression classifier is used for classifying Chinese phonemes for the features of the electroencephalogram data;
(4) Training a hyperbolic neural network and a hyperbolic multiple logistic regression classifier;
in the training process, a certain number of triples are extracted from training data, the loss of hierarchical clustering is calculated for the triples on the basis of the output characteristics of the hyperbolic neural network, and the triples are added into a total loss function to be optimized according to a certain weight;
(5) And projecting the electroencephalogram data to be decoded to a hyperbolic space, and then sequentially inputting the data to the trained hyperbolic neural network and hyperbolic multiple logistic regression classifier to obtain the decoded Chinese phoneme classification.
2. The invasive brain-computer interface chinese pronunciation decoding method of claim 1, wherein in step (1), the screening of the effective neurons from the electroencephalogram data and the removal of the highly similar neurons specifically are:
spike potential classification is carried out firstly, issuing of all neurons in electroencephalogram signals is extracted, and waveforms are drawn; checking the firing waveform of each neuron, and keeping the neurons with obvious waveforms and the total firing frequency more than 100;
and (3) cosine similarity is calculated for the issuance of different neurons, and when the similarity of a plurality of neurons is greater than 0.7, only one neuron is reserved so as to reduce the influence of crosstalk on data quality.
3. The invasive brain-computer interface chinese pronunciation decoding method of claim 1, wherein in step (2), a poincare disk model is usedTo project the brain electrical data into a hyperbolic space:
wherein,a hyperbolic space with curvature c and dimension d is represented; />Represents a data point, <' > based on>The dimension of expression isdIn the Euclidean real number space, is greater than or equal to>RepresentxIs greater than or equal to>And &>Represents a Euclidean metric and a hyperbolic metric, respectively>Representing the conformities of both metrics.
4. The invasive brain-computer interface chinese pronunciation decoding method according to claim 1, wherein in step (3), the hyperbolic neural network is represented as:
wherein,and &>Respectively representing a hyperbolic neural network function and an Euclidean neural network function, and>andrespectively representing an exponential transformation and a logarithmic transformation at the origin,crepresents the curvature of a hyperbolic space, and>represents a data point, <' > based on>Represents->Absolute value of (a).
5. The invasive brain-computer interface chinese pronunciation decoding method of claim 1, wherein in step (3), when the hyperbolic multiple logistic regression classifier performs chinese phoneme classification, z classes are given, and the probability calculation formulas of the different classes are:
wherein,and &>Parameters which are hyperbolic multivariate logistic regression>A conformal factor representing the classification boundary of class z,represents an inverse hyperbolic sine function, -is>Represents an exponential function based on a natural constant e>Represents->Absolute value of (d);representing a Mobius addition operation; c represents the curvature of a hyperbolic space; />Representing an inner product operation.
6. The invasive brain-computer interface chinese pronunciation decoding method of claim 1, wherein in step (4), the formula of the total loss function is:
7. The invasive brain-computer interface chinese pronunciation decoding method of claim 6, wherein said classification penalty is calculated as follows:
8. The invasive brain-computer interface chinese pronunciation decoding method of claim 6, wherein said hierarchical clustering penalty is calculated as follows:
wherein,represents a normalized softmax function; />Representing triples extracted from training data; />Indicates pickin a triple>Is connected to the minimum common ancestor node, </or>Representing £ in a triplet>Is connected to the minimum common ancestor node, </or>Indicates pickin a triple>The smallest common ancestor node of (c); />Representing a hyperbolic distance to a center of a hyperbolic space; />Indicates pickin a triple>Hyperbolic similarity between->Indicates pickin a triple>Hyperbolic similarity between->Indicates pickin a triple>Hyperbolic similarity between them; />Representing a matrix transposition.
9. The invasive brain-computer interface chinese pronunciation decoding method of claim 8, wherein a random sampling method is used to sample a certain number of triplets in performing the hyperbolic similarity calculationCalculating a hyperbolic distance->Divided by the sum of the three>Is normalized to obtain>The degree of similarity is expressed as->。
10. The invasive brain-computer interface chinese pronunciation decoding method of claim 8, wherein during hierarchical clustering loss calculation, triple sampling and hierarchical clustering are performed on a logit layer of a hyperbolic multiple logistic regression classifier.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211545924.0A CN115565540B (en) | 2022-12-05 | 2022-12-05 | Invasive brain-computer interface Chinese pronunciation decoding method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211545924.0A CN115565540B (en) | 2022-12-05 | 2022-12-05 | Invasive brain-computer interface Chinese pronunciation decoding method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115565540A CN115565540A (en) | 2023-01-03 |
CN115565540B true CN115565540B (en) | 2023-04-07 |
Family
ID=84770115
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211545924.0A Active CN115565540B (en) | 2022-12-05 | 2022-12-05 | Invasive brain-computer interface Chinese pronunciation decoding method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115565540B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117851769B (en) * | 2023-11-30 | 2024-06-21 | 浙江大学 | Chinese character writing decoding method for invasive brain-computer interface |
CN117958765B (en) * | 2024-04-01 | 2024-06-21 | 华南理工大学 | Multi-mode voice viscera organ recognition method based on hyperbolic space alignment |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113031766A (en) * | 2021-03-15 | 2021-06-25 | 哈尔滨工业大学 | Method for decoding Chinese pronunciation through electroencephalogram |
CN113589937A (en) * | 2021-08-04 | 2021-11-02 | 浙江大学 | Invasive brain-computer interface decoding method based on twin network kernel regression |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0993135A (en) * | 1995-09-26 | 1997-04-04 | Victor Co Of Japan Ltd | Coder and decoder for sound data |
CN102789594B (en) * | 2012-06-28 | 2014-08-13 | 南京邮电大学 | Voice generation method based on DIVA neural network model |
CN111681636B (en) * | 2020-06-16 | 2022-02-18 | 深圳市华创技术有限公司 | Technical term sound generation method based on brain-computer interface, medical system and terminal |
-
2022
- 2022-12-05 CN CN202211545924.0A patent/CN115565540B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113031766A (en) * | 2021-03-15 | 2021-06-25 | 哈尔滨工业大学 | Method for decoding Chinese pronunciation through electroencephalogram |
CN113589937A (en) * | 2021-08-04 | 2021-11-02 | 浙江大学 | Invasive brain-computer interface decoding method based on twin network kernel regression |
Also Published As
Publication number | Publication date |
---|---|
CN115565540A (en) | 2023-01-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Jahangir et al. | Deep learning approaches for speech emotion recognition: State of the art and research challenges | |
CN110516696B (en) | Self-adaptive weight bimodal fusion emotion recognition method based on voice and expression | |
CN115565540B (en) | Invasive brain-computer interface Chinese pronunciation decoding method | |
US20170358306A1 (en) | Neural network-based voiceprint information extraction method and apparatus | |
CN109559736B (en) | Automatic dubbing method for movie actors based on confrontation network | |
CN103996155A (en) | Intelligent interaction and psychological comfort robot service system | |
CN115762536A (en) | Small sample optimization bird sound recognition method based on bridge transform | |
Ocquaye et al. | Dual exclusive attentive transfer for unsupervised deep convolutional domain adaptation in speech emotion recognition | |
CN109979436A (en) | A kind of BP neural network speech recognition system and method based on frequency spectrum adaptive method | |
Sahu et al. | Modeling feature representations for affective speech using generative adversarial networks | |
Ling | An acoustic model for English speech recognition based on deep learning | |
Rybicka et al. | End-to-End Neural Speaker Diarization with an Iterative Refinement of Non-Autoregressive Attention-based Attractors. | |
CN112466284B (en) | Mask voice identification method | |
Anjos et al. | Detection of voicing and place of articulation of fricatives with deep learning in a virtual speech and language therapy tutor | |
Wu et al. | Speech synthesis with face embeddings | |
CN110348482A (en) | A kind of speech emotion recognition system based on depth model integrated architecture | |
CN111462762B (en) | Speaker vector regularization method and device, electronic equipment and storage medium | |
Yang et al. | Speech emotion analysis of netizens based on bidirectional lstm and pgcdbn | |
CN113282718B (en) | Language identification method and system based on self-adaptive center anchor | |
Shome et al. | Speaker Recognition through Deep Learning Techniques: A Comprehensive Review and Research Challenges | |
CN114882888A (en) | Voiceprint recognition method and system based on variational self-coding and countermeasure generation network | |
CN115145402A (en) | Intelligent toy system with network interaction function and control method | |
CN115472182A (en) | Attention feature fusion-based voice emotion recognition method and device of multi-channel self-encoder | |
CN112951270B (en) | Voice fluency detection method and device and electronic equipment | |
Singh | Speaker emotion Recognition System using Artificial neural network classification method for brain-inspired application |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |