CN117257312B - Method for augmenting magnetoencephalography data in machine learning - Google Patents
Method for augmenting magnetoencephalography data in machine learning Download PDFInfo
- Publication number
- CN117257312B CN117257312B CN202311540863.3A CN202311540863A CN117257312B CN 117257312 B CN117257312 B CN 117257312B CN 202311540863 A CN202311540863 A CN 202311540863A CN 117257312 B CN117257312 B CN 117257312B
- Authority
- CN
- China
- Prior art keywords
- magnetoencephalography
- signal
- equivalent current
- current signal
- magnetic field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000002582 magnetoencephalography Methods 0.000 title claims abstract description 97
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 23
- 238000010801 machine learning Methods 0.000 title claims abstract description 13
- 210000004556 brain Anatomy 0.000 claims abstract description 56
- 210000004761 scalp Anatomy 0.000 claims abstract description 42
- 210000003710 cerebral cortex Anatomy 0.000 claims abstract description 15
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims description 40
- 239000013598 vector Substances 0.000 claims description 27
- 238000006243 chemical reaction Methods 0.000 claims description 18
- 238000013519 translation Methods 0.000 claims description 18
- 230000003416 augmentation Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 5
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000001629 suppression Effects 0.000 claims description 3
- 230000005672 electromagnetic field Effects 0.000 abstract description 6
- 238000013434 data augmentation Methods 0.000 abstract description 4
- 210000002569 neuron Anatomy 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 5
- 230000001054 cortical effect Effects 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 230000002490 cerebral effect Effects 0.000 description 3
- 238000005481 NMR spectroscopy Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 208000014644 Brain disease Diseases 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 206010015037 epilepsy Diseases 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 238000002599 functional magnetic resonance imaging Methods 0.000 description 1
- 229910052734 helium Inorganic materials 0.000 description 1
- 239000001307 helium Substances 0.000 description 1
- SWQJXJOGLNCZEY-UHFFFAOYSA-N helium atom Chemical compound [He] SWQJXJOGLNCZEY-UHFFFAOYSA-N 0.000 description 1
- 238000012880 independent component analysis Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002610 neuroimaging Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000035699 permeability Effects 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/242—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents
- A61B5/245—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents specially adapted for magnetoencephalographic [MEG] signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
- A61B5/004—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room adapted for image acquisition of a particular organ or body part
- A61B5/0042—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room adapted for image acquisition of a particular organ or body part for the brain
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Neurology (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physiology (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention relates to the field of magnetoencephalography, in particular to a method for augmenting magnetoencephalography data in machine learning. The method comprises the following steps: constructing a cerebral cortex grid model based on a human brain template, and registering and aligning the scalp surface of the human brain template with the position of a magnetoencephalography helmet sensor; collecting all original signals collected by the magnetoencephalography helmet sensors in a signal collection time period, and preprocessing the original signals to obtain magnetic field signals on the surface of the scalp; constructing a forward model to describe the relationship between the equivalent current signal and the magnetic field signal of the scalp surface; spatial position perturbation is performed on the forward model or numerical perturbation is to be performed on the equivalent current signal to augment the magnetoencephalography data. The data augmentation method of the magnetoencephalography accords with the physical rule of electromagnetic field propagation in the magnetoencephalography and the physiological rule of neuron discharge in the brain, can cover more actual use scenes, and increases the diversity of magnetoencephalography data.
Description
Technical Field
The invention relates to the field of magnetoencephalography, in particular to a method for augmenting magnetoencephalography data in machine learning.
Background
The magnetoencephalography is a non-invasive functional neuroimaging technology, and can realize indirect measurement of electrophysiological activity in brain. The magnetoencephalography measures the magnetic field intensity on the scalp surface, and indirectly reflects the electrophysiological signals in the brain. Compared with electroencephalogram, magnetoencephalography has higher positioning accuracy because the conductivity difference of different tissues in the brain affects the signal propagation of electroencephalogram, while the permeability of different tissues in the brain is approximately equal. Therefore, the magnetoencephalography has higher value clinically.
At present, commercial magnetoencephalography equipment records a magnetic field generated by weak current in the brain by using a superconducting quantum interferometer with extremely high sensitivity. To maintain the superconductivity of a superconducting quantum interferometer sensor, the sensor needs to be immersed in a large liquid helium cooling device at about-269 ℃. Therefore, the current commercial magnetoencephalography equipment adopts helmets with larger sizes and fixed positions to collect brain magnetic field signals of patients.
The production and operation costs of the magnetoencephalography equipment are high, and in addition, the magnetoencephalography data of the patient are marked, so that a great amount of time and effort of experienced doctors are needed. Therefore, only a few to hundreds of labeled patient magnetoencephalography data are generally collected for a specific brain disease (e.g., epilepsy). However, in actual clinical practice, the brains of different patients have similar shapes and functional areas, but the positions of the lesion areas and the shapes of the functional areas are different. Therefore, the usable brain map data set often does not cover well what may occur in actual clinics. And the correlation algorithm of machine learning, especially the correlation algorithm of deep learning in machine learning, has a large degree of dependence on data. Only the existing magnetoencephalography data set is utilized, a machine learning algorithm is used for modeling analysis, and as the number of patients in a training set is small, the types of signals which can be learned by a model are small, the problem of overfitting on the training set is easy to occur, so that the generalization capability of the model is low, and the performance of the model is obviously reduced in clinical application.
In order to reduce the above problems, reasonable data augmentation of existing magnetoencephalography data is required. Considering that the magnetoencephalography equipment mainly observes the magnetic field generated by the current in the human brain, the constraint of the actual physical process needs to be met to obtain reasonable and amplified magnetoencephalography data.
For time-series signals, some common augmentation methods exist in the industry, such as introducing some disturbance in the time domain space, the frequency domain space, the component decomposition space or the model feature space of the single-channel time-series signals to realize data augmentation; in the magnetoencephalography data, the data amplification is carried out by using the method, and the obtained multichannel time sequence signal does not accord with the physical electromagnetic field distribution of the human brain, so that the amplified signal does not appear in clinical practice, and has no obvious effect on improving the generalization capability of a machine learning model of the magnetoencephalography.
For the magnetoencephalography data, modeling analysis work of some electromagnetic fields, such as a magnetoencephalography strength positioning method, exists; a time-varying constraint electroencephalogram or magnetoencephalography tracing method based on functional magnetic resonance imaging; the methods are used for carrying out physical electromagnetic field modeling on the brain of a patient and realizing tracing positioning of a current source in the brain based on the observed magnetic field signals; the method is characterized in that the number of sources is far larger than that of the brain magnetic map helmet sensors, constraint conditions are added to underdetermined linear equations, the underdetermined linear equations are converted into function optimization problems, and partial current source signals are approximately solved. Because the approximate solution is carried out under the underdetermined condition, only partial strong current sources are reserved in the result, the original magnetoencephalography signals cannot be reconstructed from the current sources, and the method is not suitable for data augmentation of magnetoencephalography.
Disclosure of Invention
In order to solve the problems, the invention provides a method for augmenting magnetoencephalography data in machine learning.
The method comprises the following steps:
step one, constructing grid vertexes based on human brain templates, wherein the number of the grid vertexes isRegistering and aligning the scalp surface of the human brain template with the position of a magnetoencephalography helmet sensor, and registering the number of grid vertices +.>The method comprises the following steps:
；
wherein,to divide the symbol wholly->The number of helmet sensors for the magnetoencephalography;
step two, receiving in the signal acquisition time periodCollecting all original signals acquired by the magnetoencephalography helmet sensors, wherein the number of sampling points in a signal acquisition time period isPreprocessing the original signal to obtain magnetic field signal +.>；
Step three, constructing a forward model to describe equivalent current signalsMagnetic field signal to scalp surface->The relationship between the forward model is expressed as:
；
wherein,representing a steering matrix>Noise received for a magnetoencephalography helmet sensor, < ->Representing a matrix dot product operation;
step four, the magnetic field signal on the surface of the scalpConversion to equivalent current signal->Performing spatial position disturbance on the forward model to obtain a spatial position disturbed forward model, and based on the spatial position disturbed forward model, carrying out equivalent current signalConverting the magnetic field signals into magnetic field signals of the scalp surface to be used as a part of samples of the amplified magnetoencephalography data;
or the magnetic field signal of the scalp surfaceConversion to equivalent current signal->Equivalent current signal +.>Performing numerical disturbance to obtain equivalent current signal +.>Equivalent current signal after disturbance is +.>Is converted into a magnetic field signal of the scalp surface as a part of samples of the brain magnetic map data after the augmentation.
Further, when the forward model is constructed in the third step, noise received by the magnetoencephalography helmet sensor is ignored。
Further, a guide matrix is definedIs>Line, th->The element at column is +.>，/>，Element->The calculation method of (2) is as follows:
the number of sampling points in a fixed signal acquisition time period1, equivalent current signal +.>The length converted into one dimension is +.>Is>Current intensity vector +.>Is>The component is set to 1 and the other components are set to 0, resulting in the +.>A unit current intensity vector;
will be the firstThe unit current intensity vector is split into +.>Three-dimensional current intensity vectors of length 3 on each grid vertex;
using the law of biot-savart, calculate separatelyThree-dimensional current intensity vector at each grid vertex at the firstMagnetic field intensity vectors generated at the head-mounted sensor of the brain magnetic map;
will beThe magnetic field strength vectors are added to the result of the addition at +.>Projection length on normal vector of personal brain magnetic map helmet sensor coil as element +.>Is a numerical value of (2).
Further, the magnetic field signal of the scalp surface is obtained in the fourth stepConversion to equivalent current signal->The method specifically comprises the following steps:
；
wherein,the representation is based on a steering matrix->And (5) calculating a reverse matrix.
Further, in the fourth step, the performing spatial position disturbance on the forward model specifically includes: partial grid vertexes randomly translate, cerebral cortex areas wholly translate, and the magnetoencephalography helmet rotates.
Preferably, the random translation of the partial mesh vertex specifically includes:
will be equivalent current signalAccording to->The mesh vertices are divided into->A group equivalent current signal;
respectively calculating L2 norm sequence of each group of equivalent current signals：
；
Wherein,for a point in time within the signal acquisition time period, < > in->Indicated at the time point +.>A set of equivalent current signals having a current component of length 3 +.>Is L2 norm>Representing a set of time sequences;
l2 norm sequence for each set of equivalent current signalsCalculating the coefficient of variation from->Selecting the maximum value from the variation coefficients>Coefficient of variation->；
Maximizing the coefficient of variationTaking grid vertexes corresponding to the L2 norm sequences as grid vertexes to be translated, executing random translation of spatial positions of the grid vertexes to be translated in the cerebral cortex grid model one by one, and performing spatial positions of the grid vertexes to be translated after translation +.>The method comprises the following steps:
；
；
wherein,for the spatial position before the translation of the mesh vertices to be translated, < +.>For a random amount of translation of the spatial position of the mesh vertices to be translated,/->Sequence numbers of three adjacent grid vertices for the grid vertices to be translated, +.>To be the +.>Spatial position of vertices of adjacent mesh, +.>A function is generated for random numbers having absolute values less than a fixed threshold.
Preferably, the magnetoencephalography helmet rotates, and specifically comprises:
calculation of brain skinLayer mesh modelAverage value of spatial positions of the mesh vertices, coordinates of center point of rotation of the magnetoencephalography helmet +.>：
；
Wherein,is->Spatial locations of the mesh vertices;
computing a rotation matrix：
；
Wherein,three euler angles are predefined;
spatial position of rotated magnetoencephalography helmet sensorThe method comprises the following steps:
；
wherein,the original spatial position of the helmet sensor before rotation is given to the magnetoencephalography.
Further, the equivalent electricity is generated in the fourth stepStream signalPerforming numerical perturbation, specifically including: partial channel noise addition, stationary signal suppression, partial channel amplitude scaling, and scrambling of channel order.
Preferably, the partial channel noise adding specifically includes:
generatingThe length of the segment is->Is filtered, and the Gaussian white noise of each segment is +.>Is an integer and satisfies->；
Will be equivalent current signalIs divided into->A number of channels, each channel having a length +.>；
At the position ofRandom selection of +.>Individual channels and->The length of the segment is->Is paired with the gaussian white noise signal of (a);
for each pair of paired signals, the signal to noise ratio is predefinedScaling the Gaussian white noise signal, and superposing the scaled Gaussian white noise signal on the current signal of the paired channels sample by sample to obtain +.>Noise adding signals of the channels;
will beNoise adding signal of each channel, replacing equivalent current signal +.>Corresponding channels of the circuit are subjected to disturbance to obtain equivalent current signals +.>。
Preferably, the partial channel amplitude scaling specifically includes:
will be equivalent current signalAccording to->The mesh vertices are divided into->A group equivalent current signal;
respectively calculating L2 norm sequence of each group of equivalent current signals：
；
Wherein,for a point in time within the signal acquisition time period, < > in->Indicated at the time point +.>A set of equivalent current signals having a current component of length 3 +.>Is L2 norm>Representing a set of time sequences;
at the position ofRandom selection of +.>Group (S)/(S)>Is an integer and satisfies；
Calculation ofScaling parameters of the group equivalent current signal +.>：
；
Wherein,representing averaging operations,/->For standard deviation calculation->Scaling parameters for a preset L2 norm;
based on scaling parametersCalculate->A scaled current signal of the group equivalent current signal and replacing the equivalent current signal with the scaled current signal>Corresponding original signals in the obtained signal are obtained after disturbance。
One or more technical solutions provided in the embodiments of the present application at least have the following technical effects or advantages:
the method and the device realize the augmentation of the magnetoencephalography data by executing numerical disturbance on equivalent current signals of all grid vertices in the cortical grid model or executing spatial position disturbance on a forward model of the magnetoencephalography. Wherein, the numerical disturbance of equivalent current signals follows the physiological rule of neuron discharge in the brain, and more possible discharge modes can be simulated; the forward model accords with the physical rule of electromagnetic field propagation in the magnetoencephalography, and the spatial position disturbance executed on the forward model can simulate more brain morphology and position changes of a patient and position changes of wearing the magnetoencephalography helmet. The method for amplifying the magnetoencephalography data can cover more actual use scenes and increase the diversity of the magnetoencephalography data.
Drawings
Fig. 1 is a schematic alignment diagram of a magnetoencephalography helmet and a human brain template according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail below with reference to the drawings and detailed embodiments, and before the technical solutions of the embodiments of the present invention are described in detail, the terms and terms involved will be explained, and in the present specification, the components with the same names or the same reference numerals represent similar or identical structures, and are only limited for illustrative purposes.
The invention provides a method for augmenting magnetoencephalography data in machine learning, which constructs a cerebral cortex grid model, and performs numerical approximation on electrophysiological signals in the brain by using equivalent currents on all grid vertexes; the method comprises the steps of constructing a forward model to describe the relation between equivalent current signals and magnetic field signals on the scalp surface, and executing numerical disturbance on equivalent current signals of all grid vertices in a cerebral cortex grid model or executing spatial position disturbance on the forward model of a cerebral magnetic map to increase the diversity of cerebral magnetic map data so as to improve the performance of a machine learning algorithm based on the cerebral magnetic map data.
The specific implementation method of the invention is as follows:
1. establishing a forward model for electromagnetic fields of a magnetoencephalography
This step aims to build a forward model for the magnetoencephalography, describing the effect of the cortical current on the induced magnetic field at the scalp surface. A human brain template is selected and segmented to obtain a cerebral cortex region, the cerebral cortex region is divided into a fixed number of grid vertices by a finite element method, and the scalp surface of the brain template is registered and aligned with the position of a magnetoencephalography helmet sensor. And establishing a proper or overdetermined forward model according to the Piaor-savart law and the number of the brain magnetic map helmet sensors.
1.1 construction of a cortex mesh model
A general human brain template, such as MNI152, is selected and divided into a plurality of cortical areas based on the T1 structural image of brain nuclear magnetic resonance.
Dividing the cerebral cortex area into two parts by using the finite element methodGrid of number of grid vertices, use equivalent current on all grid vertices, for largeThe electrophysiological signals in the brain are subjected to numerical approximation to construct a cerebral cortex grid model, wherein the number of grid vertexes in the cerebral cortex grid model is +.>The method comprises the following steps:
；
wherein,to divide the sign completely, 3 is the number of dimensions in three-dimensional space, +.>The number of helmet sensors for the magnetoencephalography.
And registering and aligning the scalp surface of the human brain template with the position of the magnetoencephalography helmet sensor through manually setting a datum point and an ICP iterative algorithm so as to determine the space relative position of the grid vertex and the magnetoencephalography helmet sensor. The aligned effect is shown in figure 1, the distance between the magnetoencephalography helmet and the scalp surface of the human brain template is smaller, and the scalp of the human brain template is semi-surrounded by the magnetoencephalography helmet.
1.2 building a Forward model for a magnetoencephalogram
Collecting all original signals collected by the magnetoencephalography helmet sensor in a period of time, preprocessing the original signals to obtain magnetic field signals on the scalp surfaceThe period of time is defined as a signal acquisition period of time. The preprocessing of the original signal comprises: spatial separation of signals in the time domain, removal of bad channels, independent component analysis and filtering. By preprocessing the original signals, the interference of part of the external space of the brain magnetic map helmet and the interference of part of heartbeat and eye movement can be removed.
Magnetic field signal of scalp surfaceNamely the inventionAugmented magnetoencephalography data is required.
The forward model of the magnetoencephalogram is used to describe the current of all mesh vertices in the cortical mesh model, and the course of the induced magnetic field generated at the magnetoencephalogram helmet sensor at the scalp surface.
The forward model is represented as follows:
；
wherein,guide matrix for forward model, +.>Equivalent current signals for all mesh vertices in the cortical mesh model during the signal acquisition period,/for>Noise received for a magnetoencephalography helmet sensor, < ->Representing a matrix dot product operation. Magnetic field signal of scalp surface->Is +.>Is>The number of sampling points in the signal acquisition time period is the number; guide matrix->Is +.>Is a two-dimensional matrix of (a); equivalent current signal->Is oneIs a two-dimensional matrix of (a) and (b). Noise->Omitted in the present invention.
Definition of a steering matrixIs>Line, th->The element at column is +.>，/>，Element->The calculation method of (2) is as follows:
to two-dimensional equivalent current signalsThe number of sampling points in the signal acquisition time period +.>Fixed to 1, the length of one dimension is obtained>Is>Current intensity vector +.>Is>The component is set to 1 and the other components are set to 0, resulting in the +.>A unit current intensity vector;
will be the firstThe unit current intensity vector is split into +.>Three-dimensional current intensity vectors of length 3 on each grid vertex;
using the law of biot-savart, calculate separatelyThree-dimensional current intensity vector at each grid vertex at the firstMagnetic field intensity vectors generated at the head-mounted sensor of the brain magnetic map;
will beThe magnetic field strength vectors are added to the result of the addition at +.>Projection length on normal vector of personal brain magnetic map helmet sensor coil as element +.>Is a numerical value of (2).
Steering matrixIs>Length of behaviours->Is representative of->The unit current intensity vector is at +.>Magnetic field intensity components generated on the individual magnetoencephalography helmet sensor; guide matrix->Is>The column is length->Is the vector of>The unit current intensity vector is->Magnetic field intensity components generated on the individual brain magnetic map helmet sensor.
2. Spatial position disturbance
Biological systems, including the brain, all present natural, unique variability. By performing spatial position perturbation on the forward model, this actual biological variability can be simulated, bringing the forward model closer to the actual situation of the brain.
The spatial position disturbance of the forward model needs to be considered in the actual brain structure and function and the related theory of neuroscience and biophysics; the perturbation should be random to simulate the uncertainty and uniqueness of the actual biological system.
Performing spatial position perturbation operations on the forward model includes, but is not limited to: partial mesh vertex random translation, brain cortex area integral translation, brain magnetic map helmet rotation, new brain nuclear magnetic resonance T1 structural image selection and the like.
Performing spatial position perturbation on the forward model may be an operation, or a random combination of operations.
After spatial position disturbance is carried out on the forward model, a guide matrix of the forward model after spatial position disturbance is calculatedSteering matrix based on forward model after spatial position perturbation +.>Equivalent current signal +.>The magnetic field signal converted to the scalp surface is used as a part of the sample of the augmented magnetoencephalography data.
3. Equivalent current signal disturbance
The invention simulates actual biological variability by executing disturbance on equivalent current signals, so that the data distribution of the amplified brain magnetic map is closer to the actual situation. The perturbation should be random and the basic physical properties of the equivalent current signal should be maintained.
Performing equivalent current signal disturbance requires two electromagnetic conversions, a first reverse matrix based on a forward modelMagnetic field signal of scalp surface->Conversion to equivalent current signal->Equivalent current signal +.>Performing numerical disturbance to obtain equivalent current signal +.>Then performing a second electromagnetic conversion, i.e. based on the previousEquivalent current signal after disturbance to model +.>The magnetic field signal of the scalp surface is converted into the magnetic field signal of the scalp surface, and the magnetic field signal of the scalp surface obtained after conversion is used as a part of samples of the brain magnetic map data after augmentation.
3.1 generating equivalent Current Signal
Guide matrix based on forward modelSolving the inverse matrix +.>. The forward model may be a forward model in which the spatial position disturbance is not performed, or may be a forward model in which the spatial position disturbance is performed.
Inverse matrixIs->Is a two-dimensional matrix of (a) and (b). Solving the inverse matrix is common knowledge in the art, and the present invention is not described in detail.
Based on a reverse matrixMagnetic field signal of scalp surface->Conversion to equivalent current signal->：
；
Wherein,representing a matrix dot product operation.
3.2 numerical perturbation
For equivalent current signalsPerforming numerical disturbance to obtain equivalent current signal +.>。
For equivalent current signalsPerforming numerical perturbation operations includes, but is not limited to: partial channel noise addition, stationary signal suppression, partial channel amplitude scaling, scrambling of channel order, etc.
For equivalent current signalsThe numerical perturbation may be performed as one operation, or as a random combination of operations.
In the two electromagnetic conversions, one electromagnetic conversion uses a forward model which does not execute the spatial position disturbance, and the other electromagnetic conversion uses a forward model which executes the spatial position disturbance, wherein the spatial position disturbance of the forward model is introduced in the method for augmenting the magnetoencephalography data. Both electromagnetic conversions may also use a forward model that does not perform spatial position perturbation.
4. Augmenting magnetoencephalography data
The invention provides two disturbance types, namely spatial position disturbance of a forward model and numerical disturbance of an equivalent current signal, and any disturbance type or combination of the two disturbance types can be selected in the process of amplifying the magnetoencephalography data. For example, only spatial position disturbance of the forward model is performed, and the specific flow is as follows: based on a reverse matrixMagnetic field signal of scalp surface->Conversion to equivalent current signal->The method comprises the steps of carrying out a first treatment on the surface of the And then executing the space position disturbance of the forward model to obtain a guide matrix of the forward model after the space position disturbance>Guide matrix using spatially perturbed forward model>Equivalent current signal +.>The magnetic field signal converted to the scalp surface is used as a part of the sample of the augmented magnetoencephalography data.
Or, only executing the numerical disturbance of the equivalent current signal, wherein the specific flow is as follows: based on a reverse matrixMagnetic field signal of scalp surface->Conversion to equivalent current signal->The method comprises the steps of carrying out a first treatment on the surface of the For equivalent current signal->Performing numerical disturbance to obtain equivalent current signal after disturbance>The method comprises the steps of carrying out a first treatment on the surface of the Use of a steering matrix->Equivalent current signal after disturbance +.>The magnetic field signal converted to the scalp surface is used as a part of the sample of the augmented magnetoencephalography data.
Alternatively, the spatial positions of the forward models are performed simultaneouslyThe specific flow of the disturbance and the numerical disturbance of the equivalent current signal is as follows: executing the space position disturbance of the forward model to obtain a guide matrix of the forward model after the space position disturbanceSteering matrix based on forward model after spatial position perturbation +.>Solving the inverse matrix and adding the magnetic field signal to the scalp surface>Conversion to equivalent current signal->The method comprises the steps of carrying out a first treatment on the surface of the For equivalent current signal->Performing numerical disturbance to obtain equivalent current signal after disturbance>The method comprises the steps of carrying out a first treatment on the surface of the Use of a steering matrix->Equivalent current signal after disturbance +.>The magnetic field signal converted to the scalp surface is used as a part of the sample of the augmented magnetoencephalography data.
5. Spatial position perturbation embodiment of Forward model
5.1 partial mesh vertex random translation
5.1.1 calculating the L2 norm sequence of the equivalent Current Signal
Will be equivalent current signalAccording to->The mesh vertices are divided into->Equivalent current signals of each group, each equivalent current signal representing a certain grid vertex with length of +.>Is used for the current component of the (c).
Respectively calculating L2 norm sequence of each group of equivalent current signals：
；
Wherein,for a point in time within the signal acquisition time period, < > in->Indicated at the time point +.>A set of equivalent current signals having a current component of length 3 +.>Is L2 norm>Representing a set of timings.
5.1.2 random translation of mesh vertices to be translated
L2 norm sequence for each set of equivalent current signalsCalculating the coefficient of variation from->Selecting the maximum value from the variation coefficients>Coefficient of variation->. In this embodiment, the standard deviation of the L2 norm sequence and the average value of the L2 norm sequence are divided as the coefficient of variation.
Maximizing the coefficient of variationTaking grid vertexes corresponding to the L2 norm sequences as grid vertexes to be translated, executing random translation of spatial positions of the grid vertexes to be translated in the cerebral cortex grid model one by one, and performing spatial positions of the grid vertexes to be translated after translation +.>The method comprises the following steps:
；
；
wherein,for the spatial position before the translation of the mesh vertices to be translated, < +.>For a random amount of translation of the spatial position of the mesh vertices to be translated,/->Sequence numbers of three adjacent grid vertices for the grid vertices to be translated, +.>To be the +.>Void of adjacent mesh verticesInter-position(s) (i.e. the position of the room)>A function is generated for random numbers having absolute values less than a fixed threshold. The adjacent grid vertexes are three nearest grid vertexes calculated according to the space distance, and any two adjacent grid vertexes are not collinear with the grid vertexes to be translated.
After spatial position random translation is carried out on all grid vertices to be translated, a guide matrix of a forward model after spatial position disturbance is calculated based on an updated cerebral cortex grid modelSteering matrix based on forward model after spatial position perturbation +.>Equivalent current signal +.>The magnetic field signal converted to the scalp surface is used as a part of the sample of the augmented magnetoencephalography data.
5.2 rotation of the brain magnetic map helmet
5.2.1 calculating the center of rotation
Computing a mesh model of the cortexAverage value of spatial positions of the mesh vertices, coordinates of center point of rotation of the magnetoencephalography helmet +.>：
；
Wherein,is->Spatial locations of the mesh vertices.
5.2.2 calculating the rotation matrix
Defining three Euler anglesIn this embodiment, the three euler angles have the following values: />,,/>。
Defining a rotation matrixRotation matrix->Is->Is calculated as follows:
。
5.2.3 coordinate rotation
Spatial position of rotated magnetoencephalography helmet sensorThe method comprises the following steps:
；
wherein,the original spatial position of the helmet sensor before rotation is given to the magnetoencephalography.
Helmet based on rotated magnetoencephalographyThe space position of the sensor, and the guide matrix of the forward model after the disturbance of the space position is calculatedSteering matrix based on forward model after spatial position perturbation +.>Equivalent current signal +.>Is converted into a magnetic field signal of the scalp surface as a part of samples of the brain magnetic map data after the augmentation.
6. Numerical perturbation embodiment of equivalent current signal
6.1 partial channel noise addition
GeneratingThe length of the segment is->Is filtered, and the Gaussian white noise of each segment is +.>Is an integer and satisfies->. The filtering parameters for filtering the Gaussian white noise of each section are the same as the filtering parameters used in preprocessing the original signals acquired by the magnetoencephalography helmet sensor.
Equivalent current signalIs->Is to add equivalent current signals to the two-dimensional matrix of (2)>Is divided into->A number of channels, each channel having a length +.>. At->Random selection of +.>Individual channels and->The length of the segment is->Is matched to the white gaussian noise signal of the (c).
For each pair of paired signals, according to signal-to-noise ratioScaling the Gaussian white noise signal, and superposing the scaled Gaussian white noise signal on the current signal of the paired channels sample by sample to obtain +.>Noise adding signals of the channels. In this embodiment +.>。
Will beNoise adding signal of each channel, replacing equivalent current signal +.>Corresponding channels of the circuit are subjected to disturbance to obtain equivalent current signals +.>Based on a steering matrix->Equivalent current signal after disturbance +.>Is converted into a magnetic field signal of the scalp surface as a part of samples of the brain magnetic map data after the augmentation.
6.2 partial channel amplitude scaling
Will be equivalent current signalAccording to->The mesh vertices are divided into->Equivalent current signals of each group, each group is +.>Each set of equivalent current signals representing a certain grid vertex with a length of +.>Is used for the current component of the (c). />
At the position ofRandom selection of +.>Group (S)/(S)>Is an integer and satisfies。
Calculation ofScaling parameters of the group equivalent current signal +.>：
；
Wherein,l2 norm sequence for each set of equivalent current signals,/L2 norm sequence for each set of equivalent current signals>Representing averaging operations,/->For standard deviation calculation->For the preset scaling parameters of L2 norm, the setting is performed in this embodiment。
Due to the different L2 norm sequence of each set of equivalent current signals, the scaling parameters of each set of equivalent current signalsAlso different, based on the scaling parameter +.>Calculate->A scaled current signal of the group equivalent current signal and replacing the equivalent current signal with the scaled current signal>Corresponding original signal of the obtained disturbance equivalent current signal +.>Based on a steering matrix->Equivalent current signal after disturbance +.>Is converted into a magnetic field signal of the scalp surface as a part of samples of the brain magnetic map data after the augmentation.
The above embodiments are merely illustrative of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention, and various modifications and improvements made by those skilled in the art to the technical solution of the present invention should fall within the protection scope defined by the claims of the present invention without departing from the design spirit of the present invention.
Claims (10)
1. A method for augmenting magnetoencephalography data in machine learning, comprising the steps of:
step one, constructing grid vertexes based on human brain templates, wherein the number of the grid vertexes isRegistering and aligning the scalp surface of the human brain template with the position of a magnetoencephalography helmet sensor, and registering the number of grid vertices +.>The method comprises the following steps:
；
wherein,to divide the symbol wholly->The number of helmet sensors for the magnetoencephalography;
collecting original signals collected by all the magnetoencephalography helmet sensors in a signal collecting time period, and obtaining signalsThe number of sampling points in the number acquisition time period isPreprocessing the original signal to obtain magnetic field signal +.>；
Step three, constructing a forward model to describe equivalent current signalsMagnetic field signal to scalp surface->The relationship between the forward model is expressed as:
；
wherein,representing a steering matrix>Noise received for a magnetoencephalography helmet sensor, < ->Representing a matrix dot product operation;
step four, the magnetic field signal on the surface of the scalpConversion to equivalent current signal->Performing spatial position disturbance on the forward model to obtain a spatial position disturbed forward model, and based on the spatial position disturbed forward model, performing equivalent current signal +.>Converting the magnetic field signals into magnetic field signals of the scalp surface to be used as a part of samples of the amplified magnetoencephalography data;
or the magnetic field signal of the scalp surfaceConversion to equivalent current signal->Equivalent current signal +.>Performing numerical disturbance to obtain equivalent current signal +.>Equivalent current signal after disturbance is +.>Is converted into a magnetic field signal of the scalp surface as a part of samples of the brain magnetic map data after the augmentation.
2. The method for augmenting magnetoencephalography data of claim 1, wherein the noise received by the magnetoencephalography helmet sensor is ignored when constructing the forward model in step three。
3. A method of augmenting magnetoencephalography data in machine learning according to claim 2, wherein a steering matrix is definedIs>Line, th->The element at column is +.>，/>，/>Elements ofThe calculation method of (2) is as follows:
the number of sampling points in a fixed signal acquisition time period1, equivalent current signal +.>Converted to one dimension of lengthIs>Current intensity vector +.>Is>The component is set to 1 and the other components are set to 0, resulting in the +.>A unit current intensity vector;
will be the firstThe unit current intensity vector is split into +.>Three-dimensional current intensity vectors of length 3 on each grid vertex;
using the law of biot-savart, calculate separatelyThe three-dimensional current intensity vector at the top of the mesh is at +.>Magnetic field intensity vectors generated at the head-mounted sensor of the brain magnetic map;
will beThe magnetic field strength vectors are added to the result of the addition at +.>Projection length on normal vector of personal brain magnetic map helmet sensor coil as element +.>Is a numerical value of (2).
4. The method for augmenting magnetoencephalography data of claim 2, wherein in step four, the magnetic field signal of the scalp surface is measuredConversion to equivalent current signal->The method specifically comprises the following steps:
；
wherein,the representation is based on a steering matrix->And (5) calculating a reverse matrix.
5. The method for augmenting magnetoencephalography data of claim 2, wherein the performing spatial position perturbation on the forward model in step four comprises: partial grid vertexes randomly translate, cerebral cortex areas wholly translate, and the magnetoencephalography helmet rotates.
6. The method for augmenting magnetoencephalography data of claim 5, wherein the portion of the mesh vertices are randomly translated, comprising:
will be equivalent current signalAccording to->The mesh vertices are divided into->A group equivalent current signal;
respectively calculating L2 norm sequence of each group of equivalent current signals：
；
Wherein,for a point in time within the signal acquisition time period, < > in->Indicated at the time point +.>A set of equivalent current signals having a current component of length 3 +.>Is L2 norm>Representing a set of time sequences;
l2 norm sequence for each set of equivalent current signalsCalculating the coefficient of variation from->Selecting the maximum value from the variation coefficients>Coefficient of variation->；
Maximizing the coefficient of variationTaking grid vertexes corresponding to the L2 norm sequences as grid vertexes to be translated, executing random translation of spatial positions of the grid vertexes to be translated in the cerebral cortex grid model one by one, and performing spatial positions of the grid vertexes to be translated after translation +.>The method comprises the following steps:
；
；
wherein,for the spatial position before the translation of the mesh vertices to be translated, < +.>For a random amount of translation of the spatial position of the mesh vertices to be translated,/->Sequence numbers of three adjacent grid vertices for the grid vertices to be translated, +.>To be the +.>Spatial position of vertices of adjacent mesh, +.>A function is generated for random numbers having absolute values less than a fixed threshold.
7. The method of augmenting magnetoencephalography data of claim 5, wherein the magnetoencephalography helmet is rotated, comprising:
computing a mesh model of the cortexAverage value of spatial positions of the mesh vertices, coordinates of center point of rotation of the magnetoencephalography helmet +.>：
；
Wherein,is->Spatial locations of the mesh vertices;
computing a rotation matrix：
；
Wherein,three euler angles are predefined;
spatial position of rotated magnetoencephalography helmet sensorThe method comprises the following steps:
；
wherein,the original spatial position of the helmet sensor before rotation is given to the magnetoencephalography.
8. The method for augmenting magnetoencephalography data of claim 2, wherein in step four, the equivalent current signal is generatedPerforming numerical perturbation, specifically including: partial channel noise addition, stationary signal suppression, partial channel amplitude scaling, and scrambling of channel order.
9. The method for augmenting magnetoencephalography data of claim 8, wherein the partial channel noise-adding comprises:
generatingThe length of the segment is->Is filtered, and the Gaussian white noise of each segment is +.>Is an integer and satisfies->；
Will be equivalent current signalIs divided into->A number of channels, each channel having a length +.>；
At the position ofRandom selection of +.>Individual channels and->The length of the segment is->Is paired with the gaussian white noise signal of (a);
for each pair of paired signals, the signal to noise ratio is predefinedScaling the Gaussian white noise signal, and superposing the scaled Gaussian white noise signal on the current signal of the paired channels sample by sample to obtain +.>Noise adding signals of the channels;
will beNoise adding signal of each channel, replacing equivalent current signal +.>Corresponding channels of the circuit are subjected to disturbance to obtain equivalent current signals +.>。
10. The method of augmenting magnetoencephalography data of claim 8, wherein the partial channel amplitude scaling comprises:
will be equivalent current signalAccording to->The mesh vertices are divided into->A group equivalent current signal;
respectively calculating L2 norm sequence of each group of equivalent current signals：
；
Wherein,for a point in time within the signal acquisition time period, < > in->Indicated at the time point +.>A set of equivalent current signals having a current component of length 3 +.>Is L2 norm>Representing a set of time sequences;
at the position ofRandom selection of +.>Group (S)/(S)>Is an integer and satisfies；
Calculation ofScaling parameters of the group equivalent current signal +.>：
；
Wherein,representing averaging operations,/->For standard deviation calculation->Scaling parameters for a preset L2 norm;
based on scaling parametersCalculate->A scaled current signal of the group equivalent current signal and replacing the equivalent current signal with the scaled current signal>Corresponding original signal of the obtained disturbance equivalent current signal +.>。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311540863.3A CN117257312B (en) | 2023-11-20 | 2023-11-20 | Method for augmenting magnetoencephalography data in machine learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311540863.3A CN117257312B (en) | 2023-11-20 | 2023-11-20 | Method for augmenting magnetoencephalography data in machine learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117257312A CN117257312A (en) | 2023-12-22 |
| CN117257312B true CN117257312B (en) | 2024-01-26 |
Family
ID=89219951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311540863.3A Active CN117257312B (en) | 2023-11-20 | 2023-11-20 | Method for augmenting magnetoencephalography data in machine learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117257312B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105212895A (en) * | 2015-08-24 | 2016-01-06 | 中国科学院苏州生物医学工程技术研究所 | Dynamic brain source localization method |
| CN111046918A (en) * | 2019-11-21 | 2020-04-21 | 大连理工大学 | ICA-CNN classified fMRI data space pre-smoothing and broadening method |
| CN112826512A (en) * | 2021-02-05 | 2021-05-25 | 南京慧脑云计算有限公司 | Automatic detection and peak positioning method for epileptic spike |
| CN113129403A (en) * | 2021-04-19 | 2021-07-16 | 中国科学院自动化研究所 | Magnetic particle imaging system matrix image reconstruction method and system based on forward model |
| CN115541693A (en) * | 2022-08-19 | 2022-12-30 | 西安电子科技大学 | Forward model constrained neural network magnetic particle imaging reconstruction method and system |
| WO2023178737A1 (en) * | 2022-03-24 | 2023-09-28 | 中国科学院深圳先进技术研究院 | Spiking neural network-based data enhancement method and apparatus |
| CN116887752A (en) * | 2020-09-30 | 2023-10-13 | 诺丁汉大学 | Magnetoencephalography method and magnetoencephalography system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10433742B2 (en) * | 2013-08-05 | 2019-10-08 | The Regents Of The University Of California | Magnetoencephalography source imaging for neurological functionality characterizations |
| US10861226B2 (en) * | 2019-03-07 | 2020-12-08 | Ricoh Company, Ltd. | Volume surface generator using wave vectors |
-
2023
- 2023-11-20 CN CN202311540863.3A patent/CN117257312B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105212895A (en) * | 2015-08-24 | 2016-01-06 | 中国科学院苏州生物医学工程技术研究所 | Dynamic brain source localization method |
| CN111046918A (en) * | 2019-11-21 | 2020-04-21 | 大连理工大学 | ICA-CNN classified fMRI data space pre-smoothing and broadening method |
| CN116887752A (en) * | 2020-09-30 | 2023-10-13 | 诺丁汉大学 | Magnetoencephalography method and magnetoencephalography system |
| CN112826512A (en) * | 2021-02-05 | 2021-05-25 | 南京慧脑云计算有限公司 | Automatic detection and peak positioning method for epileptic spike |
| CN113129403A (en) * | 2021-04-19 | 2021-07-16 | 中国科学院自动化研究所 | Magnetic particle imaging system matrix image reconstruction method and system based on forward model |
| WO2023178737A1 (en) * | 2022-03-24 | 2023-09-28 | 中国科学院深圳先进技术研究院 | Spiking neural network-based data enhancement method and apparatus |
| CN115541693A (en) * | 2022-08-19 | 2022-12-30 | 西安电子科技大学 | Forward model constrained neural network magnetic particle imaging reconstruction method and system |
Non-Patent Citations (1)
| Title |
|---|
| 脑源成像技术及其应用研究进展;魏玉烜;《中国科学：技术科学》;1-28 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117257312A (en) | 2023-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Boto et al. | Measuring functional connectivity with wearable MEG | |
| Bastiaansen et al. | Tangential derivative mapping of axial MEG applied to event-related desynchronization research | |
| Gramfort et al. | Time-frequency mixed-norm estimates: Sparse M/EEG imaging with non-stationary source activations | |
| Glomb et al. | Connectome spectral analysis to track EEG task dynamics on a subsecond scale | |
| Giraldo-Suarez et al. | Reconstruction of neural activity from EEG data using dynamic spatiotemporal constraints | |
| JP2012179352A (en) | System and method for constructing current dipole | |
| CN113576491A (en) | Method and system for automatically analyzing frequency domain characteristics and brain network based on resting EEG | |
| CN105266804B (en) | A kind of brain-electrical signal processing method based on low-rank and sparse matrix decomposition | |
| Cao et al. | Independent component analysis for unaveraged single-trial MEG data decomposition and single-dipole source localization | |
| Janapati et al. | Towards a more theory-driven BCI using source reconstructed dynamics of EEG time-series | |
| Songdechakraiwut et al. | Dynamic topological data analysis for functional brain signals | |
| Gu et al. | Automatic coregistration of MRI and on-scalp MEG | |
| Khosropanah et al. | Fused multivariate empirical mode decomposition (MEMD) and inverse solution method for EEG source localization | |
| Wen et al. | The EEG signal analysis for spatial cognitive ability evaluation based on multivariate permutation conditional mutual information-multi-spectral image | |
| CN111543983B (en) | Electroencephalogram signal channel selection method based on neural network | |
| Zheng et al. | Classification of motor imagery electrocorticogram signals for brain-computer interface | |
| CN117257312B (en) | Method for augmenting magnetoencephalography data in machine learning | |
| Yun-Mei et al. | The abnormal detection of electroencephalogram with three-dimensional deep convolutional neural networks | |
| KR100911731B1 (en) | Cortical rhythmic activity monitering system based on EEG | |
| CN115813409A (en) | Ultra-low-delay moving image electroencephalogram decoding method | |
| Wang et al. | Methods for Improving Movement Compatibility of Wearable OPM-MEG: A Review | |
| Bruña et al. | The more, the merrier: multivariate phase synchronization methods excel pairwise ones in estimating functional brain connectivity from reconstructed neural sources | |
| JP4791797B2 (en) | Biomagnetic field measurement device | |
| Shi et al. | SCTICA: Sub-packet constrained temporal ICA method for fMRI data analysis | |
| Wang et al. | Dynamic extraction of visual evoked potentials through spatial analysis and dipole localization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |