CN109247917A - A kind of spatial hearing induces P300 EEG signal identification method and device - Google Patents
A kind of spatial hearing induces P300 EEG signal identification method and device Download PDFInfo
- Publication number
- CN109247917A CN109247917A CN201811394789.8A CN201811394789A CN109247917A CN 109247917 A CN109247917 A CN 109247917A CN 201811394789 A CN201811394789 A CN 201811394789A CN 109247917 A CN109247917 A CN 109247917A
- Authority
- CN
- China
- Prior art keywords
- layer
- eeg signals
- convolutional
- convolutional layer
- neural networks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/377—Electroencephalography [EEG] using evoked responses
- A61B5/38—Acoustic or auditory stimuli
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Surgery (AREA)
- Psychiatry (AREA)
- Veterinary Medicine (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Signal Processing (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Physiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Acoustics & Sound (AREA)
- Psychology (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention discloses a kind of spatial hearings to induce P300 EEG signal identification method and device, is related to algorithm research field.To solve existing convolutional neural networks in the presence of the limitation by data volume, it is easy to appear serious over-fitting, recognition result and bad problem.This method comprises: pre-processing to collected P300 EEG signals, original signal is obtained;The original signal obtains multiple IMF components by EMD algorithm, by calculating IMF component normalized autocorrelation functions variance, obtains denoising EEG signals;By the convolutional layer of the denoising EEG signals input convolutional neural networks, the space characteristics of the denoising EEG signals are obtained;The space characteristics that the convolutional layer is obtained are assembled to form the full articulamentum that global characteristics are input to the convolutional neural networks, by the output layer output category accuracy rate of the convolutional neural networks.
Description
Technical field
The present invention relates to algorithm research fields, more particularly relate to a kind of spatial hearing induction EEG's Recognition side P300
Method and device.
Background technique
Brain is human body highest nerve center, controls every vital movement of human body.With in brain paralysis in recent years, brain stem
The increase of the diseases such as wind, spinal cord injury, many patient parts or the ability for having completely lost autonomous control muscle.Brain-computer interface
It aims at and does not depend on nervus peripheralis and musculature, directly carry out the platform contacted between brain and computer, in this way may be used
To help this kind of patient to carry out rehabilitation, its quality of life is improved.For healthy population, brain machine interface system is as a kind of completely new
Interactive mode will largely enrich the amusement exchange way of people.In medical system, automatic control and computer science etc.
Aspect also has important application.With the development of the fast development of the machine learning algorithms such as pattern-recognition, especially deep learning,
Brain machine interface system has become one of most important research field in the whole world.
Domestic and international vision induced brain-computer interface normal form makes great progress at present.In contrast, it is pierced based on the sense of hearing
Sharp brain-computer interface development is lagged.Sense of hearing brain-computer interface technology for the pathways for vision crowd of being obstructed provide it is a kind of new with it is outer
The logical mode of sulcus terminalis, the brain-computer interface of auditory stimulation are mainly directed towards advanced amyotrophic lateral sclerosis patients and pathways for vision
Be obstructed crowd.The research difficulty of Auditory Evoked Potential is big compared with for Visual Evoked Potential Signal, auditory evoked potential letter
Number very faint, intensity is about in several microvolts, and ambient noise interference and spontaneous electrical activity of the brain interference are believed more than auditory evoked potential
Number big, the state of mind of subject on the attention level of target and at that time can all influence.Based on auditory stimulation EEG signals type compared with
It is more, four classes are classified as according to the difference of auditory stimulation mode: based on sense of hearing P300, based on Steady-state evoked potential, be based on
Selective attention and be based on sterically defined sense of hearing normal form.Wherein P300 is a kind of event related potential, is lured by small probability event
Hair generates, and incubation period often and hence obtains one's name about 300 milliseconds after inducing stimulation.P300 has clear advantage: property is good when lock,
It can relatively easily be extracted from source signal;Induction needs not move through special training;Belong to noninvasive signals, is easy to
Subject receives;Currently has the theoretical research of enough maturations.
The experimental paradigm design of current brainstem auditory evoked EEG signals develops towards the polytypic direction of realization, it is intended to possess
Shorter stimulation time and the higher rate of information throughput.Especially spatial hearing evoked brain potential signal will be focused on providing comprehensive
Spatial information.It is merged with other brain-computer interface normal forms, realizes that multi-mode, diversified brain machine interface system and the sense of hearing lure
Send out one of the development trend of EEG signals.
In terms of the processing of EEG signals, traditional feature extraction include extract frequency-domain analysis, when-frequency analysis and AR mould
The features such as type coefficient, then using support vector machines, gradually linear discriminant analysis and the classification of Fisher linear discriminant scheduling algorithm.Closely
Nian Lai, deep learning also achieve significant achievement in terms of EEG Processing, such as convolutional neural networks, Recognition with Recurrent Neural Network
Etc..Convolutional neural networks are used in the intention assessment of upper extremity exercise EEG signals by Wang Wei magnitude, and double-handed exercise classification is put down
Equal accuracy rate has reached 84.68%, and the Average Accuracy of single-handed exercise classification is 65.51%, is above traditional recognizer;
Xie etc. is decoded about the finger motion locus of cortex current potential, has used deep learning related algorithm, result of study shows depth
Learning model is more preferable than traditional linear model classifying quality;
The one kind of P300 as event related potential, common detection method are that multiple stacking is average in time domain, great Liang Shi
It tests and shows that P300 signal can observe more apparent temporal signatures at superposed average 15 times or so.Traditional feature about P300 mentions
It takes and sorting algorithm is generally divided using its temporal signatures of extraction, then using support vector machines or gradually linear discriminant analysis
Class.Superposed average needs big data quantity, and average time-consuming, is unfavorable for the realization of online brain-computer interface.
For common convolutional neural networks, according to the method for superposed average, the data of training network will be reduced
Amount, is limited by data volume, is easy to appear serious over-fitting, recognition result is simultaneously bad.
Summary of the invention
The embodiment of the present invention provides a kind of spatial hearing and induces P300 EEG signal identification method and device, existing to solve
With the presence of convolutional neural networks limited by data volume, be easy to appear serious over-fitting, recognition result is simultaneously bad
Problem.
The embodiment of the present invention provides a kind of spatial hearing induction P300 EEG signal identification method, comprising:
Collected P300 EEG signals are pre-processed, original signal is obtained;The original signal passes through EMD algorithm
Multiple IMF components are obtained, by calculating IMF component normalized autocorrelation functions variance, obtain denoising EEG signals;
By the convolutional layer of the denoising EEG signals input convolutional neural networks, the space of the denoising EEG signals is obtained
Feature;The space characteristics that the convolutional layer is obtained, which are assembled, to be formed global characteristics and is input to the complete of the convolutional neural networks
Articulamentum, by the output layer output category accuracy rate of the convolutional neural networks.
Preferably, the EMD algorithm is determined by following equation:
Wherein, S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
The IMF component normalized autocorrelation functions function is determined by following equation:
Ri(τ)=E [ci(t)ci(t+τ)]
ρi(τ)=Ri(τ)/Ri(0)
Wherein, ci(t) i-th of IMF component is indicated.
Preferably, the convolutional layer by the denoising EEG signals input convolutional neural networks, obtains the denoising brain
The space characteristics of electric signal, specifically include:
The size of the convolution kernel of first convolutional layer is 3*11, and characteristic pattern number is 8;
The convolution kernel size of second convolutional layer is 3*11, and the number of characteristic pattern is 16;
The convolution kernel size of third convolutional layer is 3*11, and the number of characteristic pattern is 32;
The convolution kernel size of 4th convolutional layer is 2*11, and characteristic pattern number is 64;
The convolutional layer is indicated by following equation:
The activation primitive of each convolutional layer is ReLU;
Wherein,Indicate the activation value of the characteristic pattern m of l layers of output,Indicate that convolution function, * indicate convolution,
It is amount of bias.
Preferably, the full articulamentum is indicated by following equation:
The activation primitive of the full articulamentum is sigmoid;
The output layer output category accuracy rate by the convolutional neural networks, comprising:
Using cross entropy as loss function, the loss function is indicated by following equation:
Wherein, n is Current neural member number, and l is current layer number,For this layer of neuron j's and preceding layer neuron i
Bonding strength, b(I)For the biasing of this layer of neuron j, f is activation primitive, m group data, (x(i),y(i)) indicate i-th group of data and
Its corresponding category label, input datay(i)Indicate the output mark of i-th of sample corresponding network
Label, model parameter θ=(θ0,θ1,θ2,...θP)T。
Preferably, it is described collected P300 EEG signals are pre-processed before, further includes:
It is obtained by the EEG signals that three sound oddball normal forms acquire Fz, Cz, Pz, Oz, C3, C4, P3 and P4 lead described
P300 EEG signals.
The embodiment of the invention also provides a kind of spatial hearings to induce P300 EEG's Recognition device, comprising:
First obtains unit, for pre-processing to collected P300 EEG signals, obtains original signal;The original
Beginning signal obtains multiple IMF components by EMD algorithm, by calculating IMF component normalized autocorrelation functions variance, is denoised
EEG signals;
Second obtains unit, for obtaining the convolutional layer of the denoising EEG signals input convolutional neural networks described
Denoise the space characteristics of EEG signals;The space characteristics that the convolutional layer is obtained, which are assembled, to be formed global characteristics and is input to institute
The full articulamentum for stating convolutional neural networks, by the output layer output category accuracy rate of the convolutional neural networks.
Preferably, the EMD algorithm is determined by following equation:
Wherein, S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
The IMF component normalized autocorrelation functions function is determined by following equation:
Ri(τ)=E [ci(t)ci(t+τ)]
ρi(τ)=Ri(τ)/Ri(0)
Wherein, ci(t) i-th of IMF component is indicated.
Preferably, it described second obtains unit and is specifically used for:
The size of the convolution kernel of first convolutional layer is 3*11, and characteristic pattern number is 8;
The convolution kernel size of second convolutional layer is 3*11, and the number of characteristic pattern is 16;
The convolution kernel size of third convolutional layer is 3*11, and the number of characteristic pattern is 32;
The convolution kernel size of 4th convolutional layer is 2*11, and characteristic pattern number is 64;
The convolutional layer is indicated by following equation:
The activation primitive of each convolutional layer is ReLU;
Wherein,Indicate the activation value of the characteristic pattern m of l layers of output,Indicate that convolution function, * indicate convolution,
It is amount of bias.
Preferably, the full articulamentum is indicated by following equation:
The activation primitive of the full articulamentum is sigmoid;
The output layer output category accuracy rate by the convolutional neural networks, comprising:
Using cross entropy as loss function, the loss function is indicated by following equation:
Wherein, n is Current neural member number, and l is current layer number,For this layer of neuron j's and preceding layer neuron i
Bonding strength, b(I)For the biasing of this layer of neuron j, f is activation primitive, m group data, (x(i),y(i)) indicate i-th group of data and
Its corresponding category label, input datay(i)Indicate the output mark of i-th of sample corresponding network
Label, model parameter θ=(θ0,θ1,θ2,...θP)T。
Preferably, it described first obtains unit and is also used to:
It is obtained by the EEG signals that three sound oddball normal forms acquire Fz, Cz, Pz, Oz, C3, C4, P3 and P4 lead described
P300 EEG signals.
The embodiment of the invention provides a kind of spatial hearings to induce P300 EEG signal identification method and device, this method packet
It includes: collected P300 EEG signals being pre-processed, original signal is obtained;The original signal is obtained by EMD algorithm
Multiple IMF components obtain denoising EEG signals by calculating IMF component normalized autocorrelation functions variance;By the denoising brain
Electric signal inputs the convolutional layer of convolutional neural networks, obtains the space characteristics of the denoising EEG signals;The convolutional layer is obtained
To the space characteristics assemble the full articulamentum to form global characteristics and be input to the convolutional neural networks, by convolution mind
Output layer output category accuracy rate through network.This method network model training before using filtering, Wavelet Interpolation reconstruct and
The methods of empirical mode decomposition auto-correlation denoising based on noise statistics, improves the signal-to-noise ratio of EEG signals, furthermore, it constructs
Convolutional neural networks have rationally designed the size of the convolution number of plies and convolution kernel, reduce over-fitting by secondary superposed average less, mention
High recognition accuracy.Exist this method solve existing convolutional neural networks and limited by data volume, is easy to appear serious
Over-fitting, recognition result and bad problem.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is that a kind of spatial hearing provided in an embodiment of the present invention induces the signal of P300 EEG signal identification method process
Figure;
Fig. 2 is electrode position schematic diagram provided in an embodiment of the present invention;
Fig. 3 is that spatial hearing provided in an embodiment of the present invention stimulates schematic diagram;
Fig. 4 is P300 waveform diagram at eight after data prediction electrode provided in an embodiment of the present invention;
Fig. 5 is the model schematic of convolutional neural networks provided in an embodiment of the present invention;
Fig. 6 is that a kind of spatial hearing provided in an embodiment of the present invention induces the signal of P300 EEG's Recognition apparatus structure
Figure.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Fig. 1 illustratively shows a kind of spatial hearing provided in an embodiment of the present invention and induces the EEG's Recognition side P300
Method flow diagram, this method mainly comprise the steps that
Step 101, collected P300 EEG signals are pre-processed, obtains original signal;The original signal is logical
It crosses EMD algorithm and obtains multiple IMF components, by calculating IMF component normalized autocorrelation functions variance, obtain denoising brain telecommunications
Number;
Step 102, by the convolutional layer of the denoising EEG signals input convolutional neural networks, the denoising brain telecommunications is obtained
Number space characteristics;The space characteristics that the convolutional layer is obtained, which are assembled, to be formed global characteristics and is input to the convolutional Neural
The full articulamentum of network, by the output layer output category accuracy rate of the convolutional neural networks.
Before executing above step 101, one kind is first provided in the embodiment of the present invention probe into spatial hearing and pay attention to inducing
The experimental paradigm of P300 signal.In practical applications, using three sound oddball normal forms, which includes two
Kind target stimulation and a kind of non-target stimulation, specifically, each testee is required to pay attention to some target stimulation sound, ignores non-target thorn
Swashing sound, testee needs to keep comfortable sitting posture, close one's eyes in an experiment, so as to eliminate an electrical interference, further,
Testee also needs to wear electrode cap, and electrode cap presses international standard (10/20) mode connection electrode.
In collection process, using the EEG signals of this 8 leads of Fz, Cz, Pz, Oz, C3, C4, P3, P4.Fig. 2 is the present invention
The electrode position schematic diagram that embodiment provides, reference electrode as shown in Figure 2 are A1, the A2 for being located at left and right ear-lobe, and impedance is surveyed
It tries indicator light and requires display green.Further, in embodiments of the present invention, experimental design stimulation sound sequence belongs to space
Property, therefore with head related transfer function and sound sequence convolution, obtain the sequence of auditory stimuli of Virtual Space distribution.Testee is logical
The sound from different direction can be heard by crossing Sennheiser In-Ear Headphones, not need arrangement space loudspeaker, thus simple
Experimental system is changed.Fig. 3 is that spatial hearing provided in an embodiment of the present invention stimulates schematic diagram, using such as Fig. 3 in specific experiment
Shown in spatial hearing orientation.
Specifically, in the inventive embodiments, it is limited that eeg signal acquisition system newly opens up instrument using BeiJing ZhongKe
The NT9200 series electroencephalograph of responsible company, the amplifier have 32 to lead, maximum sample rate: 1000 times/second, input impedance: >=
10M Ω, common-mode rejection ratio: >=110dB, resolution ratio: 0.5 microvolt;Power supply: USB power supply.In actual experiment sample rate be 200 times/
Second.
It is designed using VC++, the program frame and class libraries provided by Microsoft Foundation class libraries (MFC).For what is used herein
Experimental method is specially devised and is adopted with graphical interface of user, parameter setting module, sonic stimulation module, timing module, data
Collect the software systems of display module, data storage module.
In a step 101, P300 EEG signals are induced for the spatial hearing of acquisition, it can be to above-mentioned P300 EEG signals
Using following processing:
In embodiments of the present invention, collected P300 EEG signals are generally 8 × 188 × 180, wherein 8 refer to channel
Number, 188 refer to sampling number, and 180 refer to that everyone acquires 180 data.Each signal refers to 1 × 188 data.
Since collected P300 EEG signals generally lie in the low-frequency range of 0-7.5Hz in the method, therefore 20Hz first is carried out to data
Low-pass filtering.1 × 188 data pass through low-pass filter, operate 8 × 180 times altogether.
In practical applications, since the common method of P300 EEG signals is that multiple stacking is average in time domain, but is superimposed flat
The data volume being required to is big, and short time consumption is long, is unfavorable for the realization of on-line system.In the present invention is implemented, after to low-pass filtering
Obtained original signal carries out Wavelet Interpolation reconstruct and removes linear drift, specifically, the frequency domain of original signal is divided into 3 layers,
It selects the bottom to be inserted into spline function, then utilizes wavelet function reconstruction signal.
In practical applications, empirical mode decomposition (Empirical Mode Decomposition, EMD) algorithm energy foundation
Temporal scale feature make sophisticated signal be decomposed into limited intrinsic mode function (Intrinsic Mode Function,
IMF) and a residue signal, in embodiments of the present invention, EMD algorithm can be as shown in formula (1):
In formula (1), S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
In practical applications, the thought based on noise statistics denoising is to find out noisy bigger IMF component, to it
- cumulative-averaging is carried out randomly ordered, to weaken the energy of random noise, then by the component and other IMF components one after being averaged
Play reconstruction signal.By repeatedly treated signal S'(t) signal-to-noise ratio is gradually improved, finally to S'(t) carry out from phase
Denoising is closed, final noise cancellation signal is obtained.Steps are as follows for specific algorithm:
1) EMD decomposition, is carried out to signals and associated noises S (t), obtains p IMF component;
2) it, enablesTo first IMF component c1(t) it carries out randomly ordered several times-cumulative-average
Obtain c1' (t), c1' (t) and S1(t) reconstruct obtains new signal S togethernew(t);
It should be noted that eliminating residue signal after EMD has been decomposed, the data point in first imf is arranged at random
Sequence-adds up-average (1000 times), and obtained signal and remaining imf forms new signal S togethernew(t);
3) it, repeats above step M times, and by each Snew(t) it adds up to be averaging and obtains new signal;
4) EMD decomposition, is carried out to new signal, calculates separately IMF component normalized autocorrelation functions variance, given threshold,
The noise contribution that the IMF component that variance is greater than the threshold value thinks that it contains is bigger;It is on the contrary, then it is assumed that the IMF component is noisy
It is lower;
In embodiments of the present invention, shown in IMF component normalized autocorrelation functions formula following (2) and (3):
Ri(τ)=E [ci(t)ci(t+τ)] (2)
ρi(τ)=Ri(τ)/Ri(0) (3)
In formula (2) and formula (3), C is the IMF component, ci(t) i-th of IMF component, E [c (t) c (t+ τ)] are indicated
For the expression formula of auto-correlation function, R (0) indicates autocorrelation value when independent variable is 0.
Threshold value selects wavelet soft-threshold, after wavelet decomposition, each layer wavelet coefficient is calculated using following formula, is conformed to
The reservation asked, it is incongruent to be set to 0.
Improvement threshold function table based on parameter alpha is indicated by following equation (4):
α is variable element, and α ∈ [0,1], λ in formula (4)iFor the threshold value of i-th of IMF component, formula can be with are as follows:Wherein, media is the absolute intermediate value of i-th of IMF component, and N is signal x
(t) length.
5), the component high to noisy ingredient uses wavelet soft threshold de-noising, and reconstructs letter together with noisy low IMF component
Number, the EMD auto-correlation denoising EEG signals based on noise statistics are obtained to get the denoising brain electricity of P300 EEG signals has been arrived
Signal.
In embodiments of the present invention, the P300 EEG signals induced for spatial hearing, using wavelet filtering interpolation, are based on
The methods of empirical mode decomposition auto-correlation denoising of noise statistics improves the signal-to-noise ratio of P300 EEG signals.Fig. 4 is the present invention
P300 waveform diagram at eight after the data prediction electrode that embodiment provides, as shown in figure 4, can be mentioned by this method
The signal-to-noise ratio of high P300 EEG signals.
In a step 102, in embodiments of the present invention, in terms of deep learning, convolutional neural networks mould is mainly used
Type.Building includes total 7 layers of convolutional neural networks such as input, convolution, output, and denoising EEG signals are imported into convolution mind
Through network, and the classification performance of the model with test set test training.
Fig. 5 is the model schematic of convolutional neural networks provided in an embodiment of the present invention, in embodiments of the present invention, denoising
After EEG signals are input in the model, it is substantially carried out following operation:
Input data (8 × 188 × 180) passes through BatchNormalization layers, i.e., to each layer single in network
Neuron input calculates its mean value and variance, then is standardized.It is two-dimensional convolution used in the embodiment of the present invention, with tradition
Square dimensions convolution kernel in image procossing is different, it is contemplated that EEG signals port number and sampling number (8 × 188) are mapped to
It is in rectangular matrix in two dimensional image, the size of convolution kernel should be adjusted in right amount.
In embodiments of the present invention, the size of the convolution kernel of first convolutional layer is 3 × 11, and characteristic pattern number is 8;Second
The convolution kernel size of a convolutional layer is 3 × 11, and the number of characteristic pattern is 16;The convolution kernel size of third convolutional layer is 3 × 11,
The number of characteristic pattern is 32;The convolution kernel size of 4th convolutional layer is 2 × 11, and characteristic pattern number is 64.Further, each
The activation primitive of convolutional layer is ReLU.It is found during parameter adjustment, with the increase of the convolution number of plies, characteristic pattern quantity
When increasing in pyramid, the effect of identification can be more preferable.
Specifically, convolutional layer is indicated by following equation (5):
ReLU is index linear unit, it is restrained fastly, asks gradient simple.In embodiments of the present invention, each convolutional layer
ReLU is indicated by formula (6):
In formula (5) and formula (6),Indicate the activation value of the characteristic pattern m of l layers of output,Indicate convolution letter
Number, * indicate convolution,It is amount of bias, f () indicates activation primitive.
In a step 102, after the convolutional layer of denoising EEG signals input convolutional neural networks, denoising brain telecommunications has been obtained
Number space characteristics.
In traditional convolutional neural networks, pond layer is used for compressed data and parameter after convolutional layer, carries out feature drop
Dimension.In embodiments of the present invention, give up pond layer, this is because the data volume of EEG signals is little, use pond layer rapid wear
Lose useful information.To prevent over-fitting, convolutional layer is also unsuitable too deep, also suitably adds in the output par, c of convolutional layer
Regularization term is added.
Further, full articulamentum is usually in the tail portion of convolutional neural networks, and all neurons are all had the right between front layer
It reconnects, the neuron between same layer is mutually not connected to." the distributed nature expression " acquired is mapped to sample mark by full articulamentum
Remember space, realizes classification.
Specifically, after the space characteristics for denoising EEG signals being input to full articulamentum, available classification results, tool
Body:
Full articulamentum can be indicated by following equation (7):
Two full articulamentums are used in the embodiment of the present invention, the neuron of full articulamentum 1 is 32, and activation primitive selects then
Elu, the neuron of full articulamentum 2 are 2, and activation primitive selects sigmoid.
Specifically, sigmoid is indicated by following equation (8):
In formula (7) and formula (8), n is Current neural member number, and l is current layer number,It is this layer of neuron j with before
The bonding strength of one layer of neuron i, b(I)For the biasing of this layer of neuron j, f is activation primitive.
Further, in the embodiment of the present invention, the output result of output layer is the output of full articulamentum.Convolutional Neural net
In network, network training largely be update that backpropagation (BP) carries out weight parameter, find and minimize loss function
Weight.
Specifically, network training measures the difference between predicted value and actual result using cross entropy as loss function
Away from, wherein loss function is indicated by following equation (9):
Wherein, m group data, (x(i),y(i)) indicate i-th group of data and its corresponding category label, input datay(i)Indicate the output label of i-th of sample corresponding network, model parameter θ=(θ0,θ1,
θ2,...θP)T。
In conclusion the embodiment of the invention provides a kind of spatial hearings to induce P300 EEG signal identification method, the party
Method includes: to pre-process to collected P300 EEG signals, obtains original signal;The original signal passes through EMD algorithm
Multiple IMF components are obtained, by calculating IMF component normalized autocorrelation functions variance, obtain denoising EEG signals;It is gone described
EEG signals of making an uproar input the convolutional layer of convolutional neural networks, obtain the space characteristics of the denoising EEG signals;By the denoising
The space characteristics of EEG signals are input to the full articulamentum of convolutional neural networks, obtain classification results;It is determined by loss function
The accuracy of the classification results.This method using filtering, Wavelet Interpolation reconstruct and is based on noise before network model training
The methods of empirical mode decomposition auto-correlation denoising of statistics, improves the signal-to-noise ratio of EEG signals, furthermore, construct convolutional Neural
Network has rationally designed the size of the convolution number of plies and convolution kernel, reduces over-fitting by secondary superposed average less, it is quasi- to improve identification
True rate.Exist this method solve existing convolutional neural networks and limited by data volume, it is existing to be easy to appear serious over-fitting
As recognition result and bad problem.
Based on the same inventive concept, the embodiment of the invention provides a kind of spatial hearings to induce P300 EEG's Recognition dress
It sets, since the principle and a kind of spatial hearing induction P300 EEG signal identification method of device solution technical problem are similar, because
The implementation of this device may refer to the implementation of method, and overlaps will not be repeated.
Fig. 6 is that a kind of spatial hearing provided in an embodiment of the present invention induces the signal of P300 EEG's Recognition apparatus structure
Figure.As shown in fig. 6, obtaining unit 61 and second the device mainly includes first obtains unit 62.
The embodiment of the invention also provides a kind of spatial hearings to induce P300 EEG's Recognition device, comprising:
First obtains unit 61, for pre-processing to collected P300 EEG signals, obtains original signal;It is described
Original signal obtains multiple IMF components by EMD algorithm, by calculating IMF component normalized autocorrelation functions variance, is gone
It makes an uproar EEG signals;
Second obtains unit 62, for obtaining institute for the convolutional layer of the denoising EEG signals input convolutional neural networks
State the space characteristics of denoising EEG signals;The space characteristics that the convolutional layer is obtained, which are assembled, to be formed global characteristics and is input to
The full articulamentum of the convolutional neural networks, by the output layer output category accuracy rate of the convolutional neural networks.
Preferably, the EMD algorithm is determined by following equation:
Wherein, S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
The IMF component normalized autocorrelation functions function is determined by following equation:
Ri(τ)=E [ci(t)ci(t+τ)]
ρi(τ)=Ri(τ)/Ri(0)
Wherein, C is the IMF component, and E [c (t) c (t+ τ)] is the expression formula of auto-correlation function, and R (0) indicates independent variable
Autocorrelation value when being 0.
Preferably, it described second obtains unit 62 and is specifically used for:
The size of the convolution kernel of first convolutional layer is 3*11, and characteristic pattern number is 8;
The convolution kernel size of second convolutional layer is 3*11, and the number of characteristic pattern is 16;
The convolution kernel size of third convolutional layer is 3*11, and the number of characteristic pattern is 32;
The convolution kernel size of 4th convolutional layer is 2*11, and characteristic pattern number is 64;
The convolutional layer is indicated by following equation:
The activation primitive of each convolutional layer is ReLU;
Wherein,Indicate the activation value of the characteristic pattern m of l layers of output,Indicate that convolution function, * indicate convolution,
It is amount of bias.
Preferably, the full articulamentum is indicated by following equation:
The activation primitive of the full articulamentum is sigmoid;
The output layer output category accuracy rate by the convolutional neural networks, comprising:
Using cross entropy as loss function, the loss function is indicated by following equation:
Wherein, n is Current neural member number, and l is current layer number,For this layer of neuron j's and preceding layer neuron i
Bonding strength, b(I)For the biasing of this layer of neuron j, f is activation primitive, m group data, (x(i),y(i)) indicate i-th group of data and
Its corresponding category label, input datay(i)Indicate the output mark of i-th of sample corresponding network
Label, model parameter θ=(θ0,θ1,θ2,...θP)T。
Preferably, it described first obtains unit 61 and is also used to:
It is obtained by the EEG signals that three sound oddball normal forms acquire Fz, Cz, Pz, Oz, C3, C4, P3 and P4 lead described
P300 EEG signals.
It should be appreciated that one of the above spatial hearing induce P300 EEG's Recognition device include unit only according to should
The logical partitioning that the function that apparatus is realized carries out in practical application, can carry out the superposition or fractionation of said units.And
A kind of spatial hearing that the embodiment provides induces the function that P300 EEG's Recognition device is realized and mentions with above-described embodiment
A kind of spatial hearing supplied induces P300 EEG signal identification method and corresponds, which is realized more detailed
Process flow has been described in detail in above method embodiment one, has been not described in detail herein.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (10)
1. a kind of spatial hearing induces P300 EEG signal identification method characterized by comprising
Collected P300 EEG signals are pre-processed, original signal is obtained;The original signal is obtained by EMD algorithm
Multiple IMF components obtain denoising EEG signals by calculating IMF component normalized autocorrelation functions variance;
By the convolutional layer of the denoising EEG signals input convolutional neural networks, the space for obtaining the denoising EEG signals is special
Sign;The space characteristics that the convolutional layer is obtained assemble to be formed global characteristics be input to the convolutional neural networks entirely connect
Layer is connect, by the output layer output category accuracy rate of the convolutional neural networks.
2. the method as described in claim 1, which is characterized in that the EMD algorithm is determined by following equation:
Wherein, S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
The IMF component normalized autocorrelation functions function is determined by following equation:
Ri(τ)=E [ci(t)ci(t+τ)]
ρi(τ)=Ri(τ)/Ri(0)
Wherein, ci(t) i-th of IMF component is indicated.
3. the method as described in claim 1, which is characterized in that described that the denoising EEG signals are inputted convolutional neural networks
Convolutional layer, obtain it is described denoising EEG signals space characteristics, specifically include:
The size of the convolution kernel of first convolutional layer is 3*11, and characteristic pattern number is 8;
The convolution kernel size of second convolutional layer is 3*11, and the number of characteristic pattern is 16;
The convolution kernel size of third convolutional layer is 3*11, and the number of characteristic pattern is 32;
The convolution kernel size of 4th convolutional layer is 2*11, and characteristic pattern number is 64;
The convolutional layer is indicated by following equation:
The activation primitive of each convolutional layer is ReLU;
Wherein,Indicate the activation value of the characteristic pattern m of l layers of output,Indicate that convolution function, * indicate convolution,It is biasing
Amount.
4. the method as described in claim 1, which is characterized in that the full articulamentum is indicated by following equation:
The activation primitive of the full articulamentum is sigmoid;
The output layer output category accuracy rate by the convolutional neural networks, comprising:
Using cross entropy as loss function, the loss function is indicated by following equation:
Wherein, n is Current neural member number, and l is current layer number,For the connection of this layer of neuron j and preceding layer neuron i
Intensity, b(I)For the biasing of this layer of neuron j, f is activation primitive, m group data, (x(i),y(i)) indicate i-th group of data and its right
The category label answered, input datay(i)Indicate the output label of i-th of sample corresponding network,
Model parameter θ=(θ0,θ1,θ2,...θP)T。
5. the method as described in claim 1, which is characterized in that described to carry out pre-processing it to collected P300 EEG signals
Before, further includes:
The P300 is obtained by the EEG signals that three sound oddball normal forms acquire Fz, Cz, Pz, Oz, C3, C4, P3 and P4 lead
EEG signals.
6. a kind of spatial hearing induces P300 EEG's Recognition device characterized by comprising
First obtains unit, for pre-processing to collected P300 EEG signals, obtains original signal;The original letter
Number multiple IMF components are obtained by EMD algorithm, by calculating IMF component normalized autocorrelation functions variance, obtains denoising brain electricity
Signal;
Second obtains unit, for obtaining the denoising for the convolutional layer of the denoising EEG signals input convolutional neural networks
The space characteristics of EEG signals;The space characteristics that the convolutional layer is obtained, which are assembled, to be formed global characteristics and is input to the volume
The full articulamentum of product neural network, by the output layer output category accuracy rate of the convolutional neural networks.
7. device as claimed in claim 6, which is characterized in that the EMD algorithm is determined by following equation:
Wherein, S (t) is original signal, rn(t) residue signal, c are indicatedi(t) IMF component is indicated, p is positive integer;
The IMF component normalized autocorrelation functions function is determined by following equation:
Ri(τ)=E [ci(t)ci(t+τ)]
ρi(τ)=Ri(τ)/Ri(0)
Wherein, ci(t) i-th of IMF component is indicated.
8. device as claimed in claim 6, which is characterized in that described second, which obtains unit, is specifically used for:
The size of the convolution kernel of first convolutional layer is 3*11, and characteristic pattern number is 8;
The convolution kernel size of second convolutional layer is 3*11, and the number of characteristic pattern is 16;
The convolution kernel size of third convolutional layer is 3*11, and the number of characteristic pattern is 32;
The convolution kernel size of 4th convolutional layer is 2*11, and characteristic pattern number is 64;
The convolutional layer is indicated by following equation:
The activation primitive of each convolutional layer is ReLU;
Wherein,Indicate the activation value of the characteristic pattern m of l layers of output,Indicate that convolution function, * indicate convolution,It is biasing
Amount.
9. device as claimed in claim 6, which is characterized in that the full articulamentum is indicated by following equation:
The activation primitive of the full articulamentum is sigmoid;
The output layer output category accuracy rate by the convolutional neural networks, comprising:
Using cross entropy as loss function, the loss function is indicated by following equation:
Wherein, n is Current neural member number, and l is current layer number,For the connection of this layer of neuron j and preceding layer neuron i
Intensity, b(I)For the biasing of this layer of neuron j, f is activation primitive, m group data, (x(i),y(i)) indicate i-th group of data and its right
The category label answered, input datay(i)Indicate the output label of i-th of sample corresponding network,
Model parameter θ=(θ0,θ1,θ2,...θP)T。
10. device as claimed in claim 6, which is characterized in that described first, which obtains unit, is also used to:
The P300 is obtained by the EEG signals that three sound oddball normal forms acquire Fz, Cz, Pz, Oz, C3, C4, P3 and P4 lead
EEG signals.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811394789.8A CN109247917A (en) | 2018-11-21 | 2018-11-21 | A kind of spatial hearing induces P300 EEG signal identification method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811394789.8A CN109247917A (en) | 2018-11-21 | 2018-11-21 | A kind of spatial hearing induces P300 EEG signal identification method and device |
Publications (1)
Publication Number | Publication Date |
---|---|
CN109247917A true CN109247917A (en) | 2019-01-22 |
Family
ID=65042024
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201811394789.8A Pending CN109247917A (en) | 2018-11-21 | 2018-11-21 | A kind of spatial hearing induces P300 EEG signal identification method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN109247917A (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110693493A (en) * | 2019-10-12 | 2020-01-17 | 北京工业大学 | Epilepsy electroencephalogram prediction method based on convolution and recurrent neural network combined time multiscale |
CN111428601A (en) * | 2020-03-18 | 2020-07-17 | 五邑大学 | Method, device and storage medium for identifying P300 signal based on MS-CNN |
CN111887845A (en) * | 2020-07-31 | 2020-11-06 | 昆明理工大学 | Attention regulation system based on EEG nerve feedback |
CN112450885A (en) * | 2020-11-27 | 2021-03-09 | 湖南师范大学 | Epileptic electroencephalogram-oriented identification method |
CN112528820A (en) * | 2020-12-03 | 2021-03-19 | 重庆邮智机器人研究院有限公司 | Motor imagery electroencephalogram signal identification method based on EMD and DCNN |
CN112528819A (en) * | 2020-12-05 | 2021-03-19 | 西安电子科技大学 | P300 electroencephalogram signal classification method based on convolutional neural network |
CN113791691A (en) * | 2021-09-18 | 2021-12-14 | 中国科学院自动化研究所 | Electroencephalogram signal band positioning method and device |
CN115474899A (en) * | 2022-08-17 | 2022-12-16 | 浙江大学 | Basic taste perception identification method based on multi-scale convolution neural network |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104635934A (en) * | 2015-02-28 | 2015-05-20 | 东南大学 | Brain-machine interface method based on logic thinking and imaginal thinking |
CN106510702A (en) * | 2016-12-09 | 2017-03-22 | 广州大学 | Auditory sense attention characteristic extraction and recognition system and method based on middle latency auditory evoked potential |
CN108236464A (en) * | 2017-12-29 | 2018-07-03 | 重庆邮电大学 | Feature extracting method and its Detection and Extraction system based on EEG signals |
-
2018
- 2018-11-21 CN CN201811394789.8A patent/CN109247917A/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104635934A (en) * | 2015-02-28 | 2015-05-20 | 东南大学 | Brain-machine interface method based on logic thinking and imaginal thinking |
CN106510702A (en) * | 2016-12-09 | 2017-03-22 | 广州大学 | Auditory sense attention characteristic extraction and recognition system and method based on middle latency auditory evoked potential |
CN108236464A (en) * | 2017-12-29 | 2018-07-03 | 重庆邮电大学 | Feature extracting method and its Detection and Extraction system based on EEG signals |
Non-Patent Citations (2)
Title |
---|
席旭刚: "基于EMD自相关的表面肌电信号消噪方法", 《仪器仪表学报》 * |
董贤光: "基于卷积神经网络的脑电信号检测与脑机接口实现", 《中国优秀硕士学位论文全文数据库 医药卫生科技辑》 * |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110693493A (en) * | 2019-10-12 | 2020-01-17 | 北京工业大学 | Epilepsy electroencephalogram prediction method based on convolution and recurrent neural network combined time multiscale |
CN110693493B (en) * | 2019-10-12 | 2022-05-31 | 北京工业大学 | Epilepsia electroencephalogram prediction feature extraction method based on convolution and recurrent neural network combined time multiscale |
WO2021184599A1 (en) * | 2020-03-18 | 2021-09-23 | 五邑大学 | Ms-cnn-based p300 signal identification method and apparatus, and storage medium |
CN111428601A (en) * | 2020-03-18 | 2020-07-17 | 五邑大学 | Method, device and storage medium for identifying P300 signal based on MS-CNN |
CN111428601B (en) * | 2020-03-18 | 2023-08-18 | 五邑大学 | P300 signal identification method, device and storage medium based on MS-CNN |
CN111887845A (en) * | 2020-07-31 | 2020-11-06 | 昆明理工大学 | Attention regulation system based on EEG nerve feedback |
CN112450885A (en) * | 2020-11-27 | 2021-03-09 | 湖南师范大学 | Epileptic electroencephalogram-oriented identification method |
CN112450885B (en) * | 2020-11-27 | 2022-04-22 | 湖南师范大学 | Epileptic electroencephalogram-oriented identification method |
CN112528820A (en) * | 2020-12-03 | 2021-03-19 | 重庆邮智机器人研究院有限公司 | Motor imagery electroencephalogram signal identification method based on EMD and DCNN |
CN112528819A (en) * | 2020-12-05 | 2021-03-19 | 西安电子科技大学 | P300 electroencephalogram signal classification method based on convolutional neural network |
CN112528819B (en) * | 2020-12-05 | 2023-01-20 | 西安电子科技大学 | P300 electroencephalogram signal classification method based on convolutional neural network |
CN113791691A (en) * | 2021-09-18 | 2021-12-14 | 中国科学院自动化研究所 | Electroencephalogram signal band positioning method and device |
CN113791691B (en) * | 2021-09-18 | 2022-05-20 | 中国科学院自动化研究所 | Electroencephalogram signal band positioning method and device |
CN115474899A (en) * | 2022-08-17 | 2022-12-16 | 浙江大学 | Basic taste perception identification method based on multi-scale convolution neural network |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109247917A (en) | A kind of spatial hearing induces P300 EEG signal identification method and device | |
Petrantonakis et al. | Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis | |
Ebrahimi et al. | Brain-computer interface in multimedia communication | |
Tucker et al. | Spatiotemporal analysis of brain electrical fields | |
CN110353702A (en) | A kind of emotion identification method and system based on shallow-layer convolutional neural networks | |
CN106569604B (en) | Audiovisual bimodal semantic matches and semantic mismatch collaboration stimulation brain-machine interface method | |
CN110765920A (en) | Motor imagery classification method based on convolutional neural network | |
CN109171770B (en) | Brain-computer interface system for attention training | |
CN107402635B (en) | Mental health adjusting method and system combining brain waves and virtual reality | |
Nykopp | Statistical modelling issues for the adaptive brain interface | |
Pun et al. | Brain-computer interaction research at the Computer Vision and Multimedia Laboratory, University of Geneva | |
CN107562191A (en) | The online brain-machine interface method of fine Imaginary Movement based on composite character | |
WO2024114480A1 (en) | Visual stimulation method, brain-computer training method, and brain-computer training system | |
Mzurikwao et al. | A channel selection approach based on convolutional neural network for multi-channel EEG motor imagery decoding | |
Bastos-Filho | Introduction to non-invasive EEG-Based brain-computer interfaces for assistive technologies | |
Feng et al. | Feature extraction algorithm based on csp and wavelet packet for motor imagery eeg signals | |
Kawala-Janik | Efficiency evaluation of external environments control using bio-signals | |
KR20080107961A (en) | User adaptative pattern clinical diagnosis/medical system and method using brain waves and the sense infomation treatment techniques | |
CN113713333A (en) | Dynamic virtual induction method and system for lower limb rehabilitation full training process | |
CN113057652A (en) | Brain load detection method based on electroencephalogram and deep learning | |
Munavalli et al. | Introduction to Brain–Computer Interface: Applications and Challenges | |
Ahmed et al. | A non Invasive Brain-Computer-Interface for Service Robotics | |
Petoku et al. | Object movement motor imagery for EEG based BCI system using convolutional neural networks | |
Plechawska-Wojcik et al. | Construction of neural nets in brain-computer interface for robot arm steering | |
Kanaga et al. | A Pilot Investigation on the Performance of Auditory Stimuli based on EEG Signals Classification for BCI Applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190122 |
|
RJ01 | Rejection of invention patent application after publication |