CN111227823B - One-dimensional characteristic signal processing method, device and system with time domain characteristics - Google Patents
One-dimensional characteristic signal processing method, device and system with time domain characteristics Download PDFInfo
- Publication number
- CN111227823B CN111227823B CN202010030908.2A CN202010030908A CN111227823B CN 111227823 B CN111227823 B CN 111227823B CN 202010030908 A CN202010030908 A CN 202010030908A CN 111227823 B CN111227823 B CN 111227823B
- Authority
- CN
- China
- Prior art keywords
- signal
- neural network
- band
- layer
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/318—Heart-related electrical modalities, e.g. electrocardiography [ECG]
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Molecular Biology (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Psychiatry (AREA)
- Physiology (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Fuzzy Systems (AREA)
- Cardiology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention provides a one-dimensional characteristic signal processing method, a one-dimensional characteristic signal processing device and a one-dimensional characteristic signal processing system with time domain characteristics. The device comprises: the signal reading module is used for the one-dimensional characteristic signal with the time domain characteristic; the preprocessing module is used for preprocessing the one-dimensional characteristic signal to obtain a one-dimensional characteristic sampling signal segment with a certain length and a certain amplitude; the memory stores computer-executable instructions, and the instructions are called by the processor to analyze the one-dimensional characteristic sampling signal segment so as to obtain an analysis result of information carried by the electrocardiosignal; and the interface module is used for reading and outputting the analysis result.
Description
Technical Field
The invention belongs to the technical field of signal analysis and processing, and particularly relates to a one-dimensional characteristic signal processing method, device and system with time domain characteristics.
Background
Nowadays, the lethality of cardiovascular diseases to human beings is very large, and the lethality becomes one of the main death reasons of human beings all over the world. At present, the prevalence rate and the death rate of cardiovascular diseases in China are still in an increasing stage. The number of cardiovascular diseases is estimated to be 2.9 hundred million, the disease death rate is the top, which is higher than that of tumors and other diseases and accounts for more than 40 percent of the death rate of resident diseases. Therefore, Electrocardiogram (ECG) becomes an important physiological indication that we need to monitor daily, it can effectively reflect the cardiovascular health status of human body, and it is widely used in clinical medicine.
The electrocardiogram provides abundant cardiovascular health information for medical workers and is inconvenient. Firstly, a large amount of clinical experience is needed for accurate analysis of the electrocardiogram, related electrocardiogram diagnosis and treatment knowledge can not be obtained all the time, and misdiagnosis and missed diagnosis can be caused due to insufficient experience, so that medical accident potential exists. Secondly, with the increase of patients with cardiovascular diseases and the maturity of medical technology, the number of electrocardiograms which need to be read by medical workers every day becomes larger, and the increase of workload brings about pressure and fatigue, which may cause errors in disease diagnosis. In conclusion, the development of an automatic electrocardiogram analysis technology improves the accuracy of machine diagnosis, and is particularly important for preventing and treating cardiovascular diseases.
There have been some related studies on the automatic diagnosis technology of electrocardiogram. For example, in "study of arrhythmia detection algorithm based on deep learning," Zhang Kun et al, "medical and health Equipment," 2018, vol.39, vol.12 ", the authors use convolutional neural network and LSTM network to analyze single heart beat data, which cannot process the long-time electrocardiogram combined by multiple heart beats; in the "electrocardiographic classification research combining migration learning and deep convolutional network, snow sail, and the like," journal of chinese medical physics ", 2018, volume 35, phase 11", an author converts a one-dimensional electrocardiographic signal into two dimensions and then analyzes the two dimensions, and although this makes an electrocardiographic analysis model obtain prior knowledge from image data and a pre-training network, the calculation amount required by data analysis is greatly increased.
In the above schemes, some feature changes brought by multi-heartbeat combinations, which are concentrated on single-heartbeat analysis and ignored, are greatly increased in calculation amount in order to utilize knowledge migration from images, and there are few concerns about accurate analysis of unidimensional electrocardiographic data with indefinite length and application of a sample equalization strategy in electrocardiographic analysis.
Disclosure of Invention
In order to solve the performance evaluation problem of the mobile device in the prior art, embodiments of the present specification provide an electrocardiographic signal processing method, an electrocardiographic signal processing device, and a terminal device. The technical scheme is as follows:
in a first aspect, a method for processing a one-dimensional feature signal with time-domain characteristics is provided, which includes: reading the one-dimensional characteristic signal with the time domain characteristic; preprocessing the one-dimensional characteristic signal to obtain a one-dimensional characteristic signal sampling segment with a certain length and a certain amplitude; analyzing the one-dimensional characteristic signal sampling segment to obtain an analysis result of the information carried by the one-dimensional characteristic signal; and reading and outputting the analysis result.
In a second aspect, there is provided a one-dimensional feature signal processing apparatus having a time-domain characteristic, including: the signal reading module is used for the one-dimensional characteristic signal with the time domain characteristic; the preprocessing module is used for preprocessing the one-dimensional characteristic signal to obtain a one-dimensional characteristic sampling signal segment with a certain length and a certain amplitude; the memory stores computer-executable instructions, and the instructions are called by the processor to analyze the one-dimensional characteristic sampling signal segment so as to obtain an analysis result of information carried by the electrocardiosignal; and the interface module is used for reading and outputting the analysis result.
In a third aspect, a system for monitoring cardiac electrical signals is provided, the system comprising: the sensor is used for acquiring electrocardiosignals; the signal reading unit is in communication connection with the sensor in a wired or wireless mode and is used for reading the electrocardiosignals; the preprocessing module is used for preprocessing the electrocardiosignals to obtain electrocardio sampling signal segments with certain length and certain amplitude; the electrocardiosignal analysis device comprises a memory and a processor, wherein computer-executable instructions are stored in the memory and are called by the processor to analyze the electrocardiosignal sampling signal segment so as to obtain an analysis result of information carried by the electrocardiosignal; and the interface module is used for reading and outputting the analysis result.
In a fourth aspect, a method for processing an electrocardiographic signal is provided, the method comprising: acquiring an electrocardio sampling signal; preprocessing the electrocardio sampling signal to obtain an electrocardio sampling signal segment with certain length and amplitude; analyzing the electrocardio sampling signal segment to obtain a classification result of the electrocardio signal carrying information; and outputting the classification result.
In a fourth aspect, there is provided a computer readable storage medium, executable to perform a method comprising: acquiring an electrocardio sampling signal; preprocessing the electrocardio sampling signal to obtain an electrocardio sampling signal segment with certain length and amplitude; analyzing the electrocardio sampling signal segment to obtain a classification result of the electrocardio signal carrying information; and outputting the classification result.
The invention can achieve the following beneficial effects:
a one-dimensional neural network model which is more matched with the type of the electrocardiogram data is designed based on the periodicity of the electrocardiogram data, so that the training speed is improved, and the complexity of the neural network is reduced. The nested design of the neural network is more suitable for periodically repeated electrocardio data. The global maximum pooling layer is adaptive to the periodicity of the electrocardiosignals, is more suitable for extracting depth features, and can enable the network to adapt to output data with different lengths. And a sample equalization strategy is used for processing the problem of sample imbalance in the electrocardio data, so that the analysis precision is improved.
Drawings
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings;
FIG. 1 is a flow chart of a method for processing an ECG signal according to an embodiment of the present disclosure;
FIG. 2 is a block diagram of the basic hardware of the electrocardiosignal processing system provided by the embodiments of the present disclosure;
FIG. 3 is a block diagram of the basic hardware of the electrocardiosignal processing system provided by the embodiments of the present disclosure;
FIG. 4 is a block diagram of the basic hardware of the electrocardiosignal processing system provided by the embodiments of the present disclosure;
FIG. 5 is a block diagram of a pre-processing flow provided by embodiments of the present description;
FIG. 6 is a schematic diagram of input and output results of a preprocessing step provided in an embodiment of the present disclosure;
FIG. 7 is a schematic diagram of a neural network structure of an analysis module provided in an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure;
FIG. 9 is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure;
FIG. 10 is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure;
FIG. 11 is a functional block diagram of an apparatus for processing cardiac electrical signals provided by an embodiment of the present disclosure;
fig. 12 is a schematic diagram of a terminal including an electrocardiographic signal processing device according to an embodiment of the present specification.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The embodiment of the invention provides an electrocardiosignal processing method, as shown in fig. 1, the method can comprise the following steps:
and S110, acquiring the electrocardio sampling signal.
In the human body, the sinoatrial node emits an excitation which is transmitted to the atrium and the ventricle in sequence according to a certain path and time course, so that the whole heart is excited. Therefore, the direction, route, sequence and time of bioelectrical changes occurring during the excitation of the various parts of the heart in each cardiac cycle are regular. The bioelectrical changes are reflected on the body surface by the conductive tissue and body fluids around the heart, so that the body parts also undergo regular bioelectrical changes, i.e., cardiac potentials, in each cardiac cycle. The electrocardio sampling signal is obtained by the acquisition of the sensor in the process.
As shown in fig. 2, a basic hardware block diagram of the ecg signal processing system is shown. The cardiac signal processing system may include a sensor for collecting cardiac signal data of the user. The analog signal obtained by the sensor is sent to a sampling module of the processing terminal, the acquisition module acquires the signal and amplifies the signal by the signal amplification module, the signal amplified by the signal amplification module can perform analog-to-digital conversion on the signal by the analog-to-digital conversion module, and the data after the digital-to-analog conversion is displayed on a display device and is stored in a memory simultaneously
Referring to fig. 3 and 4, basic hardware block diagrams of an electrocardiograph signal processing system that can be used in the present specification are shown. For example, as shown in fig. 3, the collected user data may be sent to the processing terminal in a wired manner, where the processing terminal includes a collection module, a signal amplification module, and an analog-to-digital sampling module, a signal passing through the analog-to-digital conversion module is input to the analysis module, and the analysis module analyzes the data and analyzes the analysis result. In this method, the electrocardiographic signal is transmitted to the terminal device by wire, processed by the terminal device, and further transmitted to the display device and stored. The analysis device may also issue an alarm message depending on the processing result and even control the activation of other emergency or first aid devices depending on the processing result. For example, the analysis device outputs heartbeat abnormality fed back by electrocardio data of the monitored user, the life support system classifies the heartbeat abnormality according to the heartbeat abnormality, adopts preset measures to start a corresponding medical care strategy, and sends alarm information to medical care personnel at the same time. The storage device can store a large amount of electrocardio data, only needs to store the electrocardio data at an abnormal moment, is convenient for medical workers to observe and call key data, and is added into the medical file, and meanwhile, the storage device is also beneficial to researching the condition of the patient.
For example, as shown in fig. 4, the collected user data may be wirelessly transmitted to the processing terminal, and the collected user data may be wirelessly transmitted to the processing terminal, where the processing terminal includes a collection module, a signal amplification module, and an analog-to-digital sampling module, a signal passing through the analog-to-digital conversion module is input to the analysis module, and the analysis module performs data analysis on the data and analyzes an analysis result. Based on the method, the electrocardiosignals are transmitted to the terminal device through a wired mode, and the terminal device can be a mobile terminal carried by the user, is processed by the terminal device, then is further sent to the display device, and is stored. The analysis device may also send an alarm message to remind the user of the attention according to the processing result. For example, the mobile terminal feeds back the electrocardiosignal condition to the user according to the analysis result, so that the user can know the physical condition of the user. The storage device can store only the electrocardiograph data at the abnormal moment without storing a large amount of electrocardiograph data, so that the electrocardiograph data is convenient to know. The mobile terminal can also upload data to the server, and wider data sample support is provided for disease condition monitoring. Even networking means can provide positioning information of users, and provide fixed-tone guidance for first aid when an emergency occurs, so that precious first aid time is won.
Without loss of generality, the acquisition of the electrocardiographic signal may also be electrocardiographic data which is already acquired by other equipment and is stored.
In summary, the step of obtaining the electrocardiographic signal can be implemented in different system hardware environments to adapt to different application scenarios.
And S120, preprocessing the electrocardio sampling signal to obtain an electrocardio sampling signal segment with an indefinite length.
As shown in fig. 5, in the process of acquiring the electrocardiographic signal, since the electrocardiographic signal is acquired by collecting the electrical signal on the body surface of the patient through the electrode, the electrode may be affected by the life activity of the body to generate a deviation in the signal during the collection process. Meanwhile, the current of the acquisition equipment is changed, and the generated magnetic field can generate certain interference on signals. Therefore, various noises including baseline drift noise, power frequency interference and the like always exist in the electrocardiosignal.
The baseline drift is caused by poor contact between the electrode and the body surface, respiration of a patient and the like, so that the electrocardiosignal generates baseline drift, and the accurate identification and judgment of the electrocardiosignal are greatly influenced. The baseline drift is represented by the fact that the baseline of the electrocardiosignal swings up and down and changes periodically like a low-frequency sine wave. The power frequency interference is from electromagnetic radiation generated by a signal acquisition environment, the frequency of the power frequency interference is consistent with the frequency of alternating current in the environment, the frequency of the alternating current used in China is 50Hz, the power frequency interference is generally an interference signal and an extremely harmonic wave under the frequency, and the interference can have certain influence on the signal, so that certain deviation is generated on the amplitude measurement of a QRS wave group.
In order to overcome the influence of power frequency noise and baseline noise simultaneously, a band-pass filter and a band-stop filter which are connected in series are used. The band-pass filter is used for reserving signals in a preset frequency band, and the band-stop filter is used for filtering specific noise frequency points in the frequency band reserved by the band-pass filter.
In an alternative embodiment, the sampling rate of the electrocardiosignal used is 500Hz, the amplitude unit is μ V, the calculation is carried out in a data length of 30s, and a single data comprises 15000 data points. Because the original electrocardiosignal has noise such as baseline drift, power frequency interference and the like, and has positive amplitude and negative amplitude. This type of data can increase the pressure of the analysis module, be unfavorable for model convergence, lead to being difficult to learn key electrocardio characteristic. Therefore, a digital band-pass filter, such as a butterworth filter, is used to solve the baseline wander problem, taking into account the low frequency characteristics of the baseline wander. Generally speaking, the frequency range of the electrocardiosignals is between 0.05 Hz and 100Hz, and 90 percent of the spectral energy of the electrocardiosignals is concentrated between 0.25 Hz and 35Hz according to the measured data, so the upper and lower limit ranges of the frequency to be reserved can be specifically questioned according to the requirements of an analysis model. For example, through multiple experiments, it is found that the retention of 1.25-100Hz electrocardiogram data is the best in model performance, so we use a digital Butterworth band-pass filter with an upper cut-off frequency of 1.25Hz and a lower cut-off frequency of 100Hz to filter out low-frequency baseline drift and simultaneously filter out unwanted high-frequency information.
In an alternative embodiment, the invention employs a digital band-stop filter, implemented by a butterworth filter, for filtering the power frequency interference. The power frequency interference is from electromagnetic radiation generated by a signal acquisition environment, the frequency of the power frequency interference is 50Hz in China, the power frequency interference can be absorbed by a human body and acts on a distributed capacitor formed by body fluid and tissues, and noise pollution is caused to signals. The invention uses a digital Butterworth band elimination filter with the upper cut-off frequency of 49Hz and the lower cut-off frequency of 51Hz to filter the power frequency interference.
Without loss of generality, the type of filter and the cut-off frequency can also be chosen, for example with a chebyshev filter, the frequencies of the band-pass and band-stop can be adjusted. After the noise filtering task is finished, amplitude adjustment is carried out on the electrocardiogram data, the amplitude of the data is limited to be in the range of [0,1] as much as possible, the original amplitude is divided by 1000 as the amplitude unit of the original electrocardiogram data is mu V, and the unit is adjusted to mV.
After the amplitude adjustment is finished, in consideration of the continuity of the electrocardiographic data, in order to reduce the processing pressure of the analysis module, the data may be down-sampled and the amount of input data may be halved. For example, let a piece of data contain data points that fall from 15000 originally to 7500.
According to the difference of the lengths of the input signals, after the preprocessing step, the electrocardiogram sampling data segments with indefinite lengths can be generated. As shown in fig. 6, after filtering, amplitude-dividing and down-sampling two pieces of input electrocardiographic data with the length of L1 and L2, respectively, the signal length does not change, and the input lengths are still L1 and L2. Therefore, according to the length of the input, a sampling signal segment with a certain length is output.
In summary, the preprocessing step may use a first band-pass filter and a second band-stop filter to process the electrocardiosignal, where the first band-pass filter has a first band-pass frequency band determined by a first upper cutoff frequency and a first lower cutoff frequency; the second band-stop filter is provided with a second band-stop frequency band determined by a second upper cut-off frequency and a second lower cut-off frequency; the second band-stop frequency band is within the first band-pass frequency band; the preprocessing step further comprises a whole amplitude step, which is used for carrying out whole amplitude to a uniform amplitude range on the amplitude value of the electrocardio sampling signal; and a down-sampling step for reducing the sampling rate of the electrocardio sampling signal. After the preprocessing step, the input electrocardiosignals are converted into electrocardio sampling signal segments with certain length and amplitude.
S130, analyzing the electrocardio sampling signal segments to obtain the classification result of the electrocardio signals carrying information.
In an alternative embodiment, since the preprocessed cardiac electrical signal is converted into a sampled signal segment with a certain length and amplitude, which can be regarded as a feature map (feature map) with a certain width but different lengths, the cardiac electrical signal can be identified by means of artificial intelligence technology, for example, the cardiac electrical signal can be processed by using a deep neural network. As shown in fig. 7, the deep neural network may include a plurality of neural network sub-modules, each including convolutional neural network elements and a max pooling layer (maxporoling), and a forward shortcut connection that enables addition between convolutional neural network inputs and outputs. As shown in fig. 8, each neural network sub-module includes a batch normalization layer (BN), an activation function layer (RELU, ELU), and a one-dimensional convolutional neural network layer (1-D CNN) connected in sequence. The reason for using the one-dimensional convolutional neural network is that the input electrocardiosignal is a continuous sampling signal in a time domain, a second sampling value does not appear at the same time point, the complexity of the network can be reduced by using the one-dimensional convolutional neural network, and the training and operation speed of the network is increased at the same time. The input data is convolved with the one-dimensional convolution neural network layer, and the one-dimensional convolution layer is used as a feature extractor of the electrocardio data and used for sliding extraction and generating features of higher layers.
As shown in fig. 9, a plurality of neural network sub-modules are connected in series, and a maximum pooling layer is connected behind each module for reducing feature dimensions and extracting key information.
In a preferred embodiment, all convolution kernels have a size of 1 × 9 and step size of 1, that is, one convolution kernel can extract information of 9 feature points, and higher-level features are generated by continuously sliding and operating the convolution kernels on lower-level features. As shown in fig. 10, 6 maximum pooling layers were used. The pooling size is gradually increased, for example 6 pooling levels are sized {2, 2, 4, 4, 4, 4}, respectively. The reason for setting the convolution kernel and the maximum pooling layer is that after 6 convolutional neural network modules, the model's receptive field can be covered within one second, taking a 500Hz sampling rate as an example, and 250 sampling points are included every minute after sampling, so the model's receptive field needs to include data covering the 250 samples. A larger convolution kernel and pooling size is used. In the first three convolutional neural network modules, the number of convolutional kernels in all convolutional layers is 16, and in the last three convolutional neural network modules, the number of convolutional kernels in all convolutional layers is 64. The arrangement is that when the low-level features are extracted, the number of required feature vectors is small, and when the model is evolved to the high level, data points contained in the generated feature vectors are further reduced under the action of the maximum pooling layer, and meanwhile, the number of convolution kernels is increased, so that the high-level features have more feature vectors to express more different types of high-level features.
Since the ecg sampling segments as input data may have different lengths, some ecg sampling signals have 1500 input points, and some ecg sampling signals have more points. In order to enable the neural network to process the electrocardio-sampled signals with different lengths, as shown in fig. 10, the analysis module further comprises a global maximum pooling layer arranged behind the plurality of neural network sub-modules. Considering the periodicity of the electrocardiogram data and the framework of a network nested network, a global maximum pooling layer is added after the neural network modules of each hierarchy to acquire the required high-dimensional electrocardiogram characteristics. Under the framework of a network nested network, the model can be understood that a sub-network exists on each electrocardiogram subregion to extract the characteristics of the electrocardiogram subregion, and the extracted characteristics can also change periodically due to certain periodic characteristics of the electrocardiogram data, so that the characteristics which have key effects on the electrocardiogram information discrimination in the periodic repeated characteristics can be extracted by using a global maximum pooling based on the whole situation to carry out accurate electrocardiogram analysis. Meanwhile, the global maximum pooling can effectively reduce the feature dimension and greatly reduce the operation amount. And because the global maximum pooling is characterized in that the length of any long feature is reduced to 1, and the use of a convolutional neural network is combined, the model can process the electrocardiographic data of variable length. After global max pooling, a regression result and a plurality of binary results are output using a plurality of fully connected layers and activation functions (e.g., sigmoid). In a preferred embodiment, the output result of the global max-pooling layer is input to a first fully-connected layer; the output result of the first full connection layer is input into a second full connection layer, the second connection layer at least has a numerical value output channel, and the numerical value channel is used for outputting a heart rate numerical value. The output of the global maximum pooling layer is input to a first fully connected layer; the output of the first full connection layer is input into a second full connection layer, the second connection layer at least has a plurality of two-classification result output channels, and the two-classification result output channels are used for outputting at least one of four classification results of normal, premature beat, block and interference.
As shown in FIG. 10, after the classification result is obtained in the model training process, it can be processed by using a sample equalization strategy. Because the proportion of the electrocardiographic positive abnormal data is greatly different, and the number of the abnormal data of different types is also greatly different, the problem of serious data imbalance of the electrocardiographic data can be seen, the model can be biased to a plurality of types in the training process, and the accuracy of electrocardiographic analysis is reduced. To this end, the present invention addresses this problem using a loss function based sample equalization strategy. The idea is that the loss generated by each batch of data in training is adjusted through two parameters, the influence of a small number of electrocardio information types on a model is increased, and therefore analysis errors caused by imbalance are eliminated.
In an optional embodiment, there is also a serious sample imbalance problem in the electrocardiographic data, and in consideration of the imbalance, a sample equalization strategy in the two-dimensional target detection problem is adjusted and applied to the one-dimensional electrocardiographic data to solve the same type of problem, and a loss function in the training process needs to be considered:
on the first hand, the proportion of loss values generated by simple samples and difficult samples is adjusted, the simple samples refer to samples which are easy to classify the model, the difficult samples refer to samples which are difficult to classify the model, and the model can be better fitted by increasing the proportion of the loss values of the difficult samples;
in the second aspect, the proportion between the loss values generated by the types with fewer samples and the types with more samples is adjusted, so that the model can be better fitted to the types with fewer samples, and the accuracy of automatic analysis is improved.
In a preferred design under the idea, the loss function can be designed as:
l(y，y′)＝-αy(1-y′)γlogy′-y′y(1-α)(1-y)log(1-y′)
wherein y represents a data label, y' represents two parameters of which the predicted values gamma and alpha of the model are loss functions, and the specific gravities of the simple samples and the difficult samples and the specific gravities of the few types and the most types are respectively controlled. γ is an empirical value that can be set, for example, to 2, and α follows the following rule:
wherein C isiIndicating the proportion of positive samples in category i.
In electrocardiographic data, considering the sample imbalance, C is generally the normiLess than 0.5, alphaiRepresenting the proportion of the total loss value that is contributed by the positive samples in class i. Can not convert alpha to alphaiSet to a value of 1 or very close to 1, let α be such that it would cause serious inaccuracies due to the model completely ignoring the negative examples' featuresiIs in the range of [0.5,0.9 ]]Therefore, the influence caused by sample imbalance can be effectively adjusted.
After the neural network model and the loss function are set, training is started, and parameters of the neural network model are trained based on a back propagation algorithm through electrocardiosignal samples serving as training samples and the indexing results of the electrocardiosignal samples. After training with enough samples, the model is verified by validating the sample set. After the model training is completed, the trained model can be deployed to the terminal device using the neural network model. The terminal device may be a ward monitoring system or a terminal device held by a user, and the like, which can receive the electrocardiosignals and process the electrocardiosignals.
After the trained neural network model is deployed in the equipment, the corresponding equipment can process the input electrocardio sampling signal segment and output the classification result of the electrocardio signal carrying information. The input result, such as the output result of the pre-designed network structure, can output at least one of the four classification results of normal, premature beat, block and interference. The numerical channel is used for outputting heart rate numerical values, and the heart rate output numerical values can be compared with a threshold value to obtain output results of bradycardia and tachycardia.
In an alternative embodiment, to evaluate the accuracy of the model in the electrocardiographic analysis, the F1 score is used as an index, because the extreme sample imbalance problem in the electrocardiographic data is not suitable for directly judging the accuracy of the model by using the accuracy rate, and the F1 score is more suitable. The F1 score is an index used for measuring the accuracy of the two classification models in statistics and is composed of precision ratio and recall ratio, the precision ratio represents the accuracy of the model for judging the positive samples, the recall ratio represents the proportion of the positive samples in all the positive samples judged by the model, and the F1 score is a function of the two and can be regarded as a harmonic mean value of the two, and the value range of the F3578 score is [0,1 ].
According to the neural network model, a neural network can be built. And then training the model by using the electrocardiosignals and the indexing results corresponding to the electrocardiosignals, wherein the training process comprises a back propagation algorithm.
In conclusion, the neural network is trained based on the built neural network. The electrocardiosignals may be processed and different classification results output, as well as results of heart rate bradycardia/tachycardia based on heart rate.
And S140, outputting the classification result.
As shown in fig. 10, the input data is analyzed and different results are output. For example, HR represents heart rate value; IF represents interference; PB represents premature beat; BL represents hysteresis; tachycardia (TC) and Bradycardia (BC) conditions can also be obtained from the HR values. These results can be output in a readable manner via a human-machine interface, for example as a speech output or a display output. In some cases, an alarm process may also be performed. The output data can also be stored locally or in the cloud.
In summary, based on the above steps, the morphological feature in the electrocardiographic information can be extracted and identified. Global pooling is used, based on the periodic nature of the electrocardiogram, for extracting the most efficient features for identification. Meanwhile, the use of global pooling enables the method to process the electrocardiogram data with different lengths; and a sample equalization strategy is used for processing the problem of sample imbalance in the electrocardio data, so that the analysis precision is improved. The electrocardiosignal analysis method based on the global pooling and sample equalization strategy is realized.
Fig. 11 is a schematic diagram of an apparatus for performing a method for processing an electrocardiographic signal according to the foregoing embodiment, which includes: the signal reading unit is used for reading the electrocardiosignals; the preprocessing module is used for preprocessing the electrocardiosignals to obtain electrocardio sampling signal segments with certain length and certain amplitude; the electrocardiosignal analysis device comprises a memory and a processor, wherein computer-executable instructions are stored in the memory and are called by the processor to analyze the electrocardiosignal sampling signal segment so as to obtain an analysis result of information carried by the electrocardiosignal; and the interface module is used for reading and outputting the analysis result. The module further includes a sub-module for correspondingly implementing each function in the foregoing method, and repeated parts with the foregoing embodiment are not described again.
The preprocessing module comprises a first band-pass filter and a second band-stop filter which are connected in series, wherein the first band-pass filter is provided with a first band-pass frequency band determined by a first upper cut-off frequency and a first lower cut-off frequency; the second band-stop filter is provided with a second band-stop frequency band determined by a second upper cut-off frequency and a second lower cut-off frequency; the second band-stop frequency band is within the first band-pass frequency band. A pre-processing module comprising: the whole amplitude submodule is used for carrying out whole amplitude on the amplitude value of the electrocardio sampling signal to a uniform amplitude range; and/or the down-sampling sub-module is used for reducing the sampling rate of the electrocardio sampling signal. The analysis module comprises a plurality of neural network sub-modules, and each neural network sub-module comprises a convolutional neural network unit and a maximum pooling layer; and a forward shortcut connection for performing an addition operation between the input and the output of the convolutional neural network. The neural network submodule comprises a batch standardization layer, an activation function layer and a one-dimensional convolution neural network layer which are sequentially connected. The analysis module further comprises a global max pooling layer disposed after the plurality of neural network sub-modules, the global max pooling layer for preserving a maximum in each of the feature vectors. The output result of the global maximum pooling layer is input into a first full-connection layer; the output result of the first full connection layer is input into a second full connection layer, the second connection layer at least has a numerical value output channel, and the numerical value channel is used for outputting a heart rate numerical value.
According to the method of the present illustrative embodiment, in addition to processing the cardiac electrical signal, it is also possible to process a sound signal having characteristics similar to those of the cardiac electrical signal. The sound signal, after being converted into an electrical signal by a transducing device such as a microphone, is also a one-dimensional signal having a time-series characteristic, and thus can be processed by the method and apparatus in the present embodiment. The neural network is trained only by using pre-indexed sound signals, network parameters are optimized based on a back propagation algorithm, the trained network is deployed into terminal equipment, the terminal equipment can input the pre-deployed neural network after preprocessing the sound signals, the neural network identifies the voice, and parameters such as the type, tone and scale of the voice signals are output. Of course, without loss of generality, the signal may be processed by other models and the corresponding result may be output.
Referring to fig. 12, a schematic structural diagram of a terminal according to an embodiment of the present invention is shown. The terminal is configured to implement the method for evaluating the performance of the mobile device on the sender client side provided in the foregoing embodiment, specifically:
the terminal 1100 may include RF (Radio Frequency) circuitry 110, memory 120 including one or more computer-readable storage media, an input unit 130, a display unit 140, a video sensor 150, audio circuitry 160, a WiFi (wireless fidelity) module 170, a processor 180 including one or more processing cores, and a power supply 190. Those skilled in the art will appreciate that the terminal structures shown in the figures are not intended to be limiting of the terminal, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the RF circuit 110 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receives downlink information from a base station and then sends the received downlink information to the one or more processors 180 for processing; in addition, data relating to uplink is transmitted to the base station. In general, the RF circuitry 110 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, and the like. In addition, the RF circuitry 110 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communications), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), and the like.
The memory 120 may be used to store software programs and modules, and the processor 180 executes various functional applications and data processing by operating the software programs and modules stored in the memory 120. The memory 120 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as video data, a phone book, etc.) created according to the use of the terminal 1100, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 120 may further include a memory controller to provide the processor 180 and the input unit 130 with access to the memory 120.
The input unit 130 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, the input unit 130 may include an image input device 131 and other input devices 132. The image input device 131 may be a camera or a photoelectric scanning device. The input unit 130 may include other input devices 132 in addition to the image input device 131. In particular, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 140 may be used to display information input by or provided to a user and various graphical user interfaces of the terminal 1100, which may be made up of graphics, text, icons, video, and any combination thereof. The Display unit 140 may include a Display panel 141, and optionally, the Display panel 141 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode, 15 Organic Light-Emitting Diode), or the like.
WiFi belongs to a short-distance wireless transmission technology, and the terminal 1100 can help a user send and receive e-mails, browse web pages, access streaming media, and the like through the WiFi module 70, and it provides a wireless broadband internet access for the user. Although the WiFi module 170 is shown in the drawing, it is understood that it does not belong to the essential constitution of the terminal 1100 and can be omitted entirely as needed within the scope not changing the essence of the invention.
The processor 180 is a control center of the terminal 1100, connects various parts of the entire mobile phone using various interfaces and lines, and performs various functions of the terminal 1100 and processes data by operating or executing software programs and/or modules stored in the memory 120 and calling data stored in the memory 120, thereby performing overall monitoring of the mobile phone. Optionally, processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications.
It will be appreciated that the modem processor described above may not be integrated into the processor 180.
The terminal 1100 also includes a power supply 190 (e.g., a battery) for powering the various components, which may preferably be logically coupled to the processor 180 via a power management system that may be used to manage charging, discharging, and power consumption. The power supply 190 may also include any component including one or more of a dc or ac power source, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.
Although not shown, the terminal 1100 may further include a bluetooth module or the like, which is not described in detail herein.
In this embodiment, the terminal 1100 further comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for performing the method on the sender client side or the receiver client side.
The memory also includes one or more programs stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the method on the backend server side, including:
it should be understood that the reference to "a plurality" in the present embodiment means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (17)
1. A method of one-dimensional feature signal processing having time-domain characteristics, comprising:
reading the one-dimensional characteristic signal with the time domain characteristic;
preprocessing the one-dimensional characteristic signal to obtain a one-dimensional characteristic signal sampling segment with a certain length and a certain amplitude;
analyzing the one-dimensional characteristic signal sampling segment to obtain an analysis result of the information carried by the one-dimensional characteristic signal;
reading and outputting the analysis result;
analyzing the one-dimensional characteristic signal sampling segment by using a deep neural network, wherein the deep neural network comprises a plurality of neural network sub-modules, and each neural network sub-module comprises a convolution neural network unit and a maximum pooling layer connected with the convolution neural network unit; and forward shortcut connection for performing addition operation between the input and output of the convolutional neural network;
the loss function of the training basis of the neural network sub-module is as follows:
l(y，y′)＝-αy(1-y′)γlogy′-y′γ(1-α)(1-y)log(1-y′)
wherein y represents a data label, y' represents a predicted value, and gamma and alpha are two parameters of a loss function, and the specific gravities of the simple samples and the difficult samples and the specific gravities of the minority types and the majority types are respectively controlled;
γ is an empirical value;
the setting of α follows the following function:
wherein C isiIndicating the proportion of positive samples in category i.
2. The method of claim 1, wherein preprocessing the one-dimensional signature signal comprises passing the one-dimensional signature signal through a first band-pass filter and a second band-stop filter in series, the first band-pass filter having a first band-pass band determined by a first upper cutoff frequency and a first lower cutoff frequency; the second band-stop filter is provided with a second band-stop frequency band determined by a second upper cut-off frequency and a second lower cut-off frequency; the second band-stop frequency band is within the first band-pass frequency band.
3. The method of claim 1, wherein pre-processing the one-dimensional feature signal comprises:
the amplitude value of the one-dimensional characteristic signal is amplified to a preset amplitude range;
and/or reducing the sampling rate of the one-dimensional characteristic signal.
4. The method of claim 1, wherein the neural network sub-module comprises a batch normalization layer, an activation function layer, and a one-dimensional convolutional neural network layer connected in sequence.
5. The method of claim 1, wherein a global max pooling layer is set after the plurality of neural network sub-modules, the global max pooling layer being used to preserve a maximum value in each feature vector.
6. The method of claim 5, wherein the output of the global max-pooling layer is input to a first fully-connected layer; the output result of the first full connection layer is input into a second full connection layer, the second full connection layer at least has a numerical value output channel, and the numerical value output channel is used for outputting a heart rate numerical value.
7. The method of claim 5, wherein the output of the global max-pooling layer is input to a first fully-connected layer; and the output of the first full connection layer is input into a second full connection layer, and the second full connection layer at least has a plurality of binary result output channels.
8. The method according to one of claims 1 to 7, characterized in that the one-dimensional feature signal having time-domain characteristics is an electrocardiogram signal, an electroencephalogram signal, a pulse wave signal, or a speech signal.
9. A one-dimensional feature signal processing apparatus having a time-domain characteristic, comprising:
the signal reading module is used for the one-dimensional characteristic signal with the time domain characteristic;
the preprocessing module is used for preprocessing the one-dimensional characteristic signal to obtain a one-dimensional characteristic sampling signal segment with a certain length and a certain amplitude;
the device comprises a memory and a processor, wherein computer executable instructions are stored in the memory and are called by the processor to analyze the one-dimensional characteristic sampling signal segment so as to obtain an analysis result of information carried by the one-dimensional characteristic signal;
the interface module is used for reading and outputting the analysis result;
analyzing the one-dimensional characteristic signal sampling segment by using a deep neural network, wherein the deep neural network comprises a plurality of neural network sub-modules, and each neural network sub-module comprises a convolution neural network unit and a maximum pooling layer connected with the convolution neural network unit; and forward shortcut connection for performing addition operation between the input and output of the convolutional neural network;
the loss function of the training basis of the neural network sub-module is as follows:
l(y，y′)＝-αy(1-y′)γlogy′-y′γ(1-α)(1-y)log(1-y′)
wherein y represents a data label, y' represents a predicted value, and gamma and alpha are two parameters of a loss function, and the specific gravities of the simple samples and the difficult samples and the specific gravities of the minority types and the majority types are respectively controlled;
γ is an empirical value;
the setting of α follows the following function:
wherein C isiIndicating the proportion of positive samples in category i.
10. The apparatus of claim 9, wherein the pre-processing module comprises a first band-pass filter and a second band-stop filter connected in series, the first band-pass filter having a first band-pass frequency band determined by a first upper cutoff frequency and a first lower cutoff frequency; the second band-stop filter is provided with a second band-stop frequency band determined by a second upper cut-off frequency and a second lower cut-off frequency; the second band-stop frequency band is within the first band-pass frequency band.
11. The apparatus of claim 9, wherein the preprocessing module comprises:
the whole amplitude submodule is used for carrying out whole amplitude on the amplitude value of the electrocardio sampling signal to a uniform amplitude range;
and/or the down-sampling sub-module is used for reducing the sampling rate of the electrocardio sampling signal.
12. The apparatus of claim 9, wherein the neural network sub-module comprises a batch normalization layer, an activation function layer, and a one-dimensional convolutional neural network layer connected in sequence.
13. The apparatus of claim 9, further comprising a global max pooling layer disposed after the plurality of neural network sub-modules, the global max pooling layer configured to preserve a maximum value in each feature vector.
14. The apparatus of claim 13, wherein the output of the global max-pooling layer is input to a first fully-connected layer; the output result of the first full connection layer is input into a second full connection layer, the second full connection layer at least has a numerical value output channel, and the numerical value output channel is used for outputting a heart rate numerical value.
15. The apparatus of claim 13, wherein an output of the global max-pooling layer is input to a first fully-connected layer; the output of the first full connection layer is input into a second full connection layer, the second full connection layer at least has a plurality of two-classification result output channels, and the two-classification result output channels are used for outputting at least one of four classification results of normal, premature beat, block and interference.
16. The apparatus according to one of claims 9 to 15, characterized in that the one-dimensional feature signal having time-domain characteristics is an electrocardiogram signal, an electroencephalogram signal, a pulse wave signal, or a speech signal.
17. A cardiac signal processing system comprising:
the sensor is used for acquiring electrocardiosignals;
the one-dimensional characteristic signal processing apparatus with time-domain characteristics as claimed in one of claims 9 to 15, wherein the signal reading module is connected in communication with the sensor in a wired or wireless manner for reading the electrocardiographic signal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010030908.2A CN111227823B (en) | 2020-01-13 | 2020-01-13 | One-dimensional characteristic signal processing method, device and system with time domain characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010030908.2A CN111227823B (en) | 2020-01-13 | 2020-01-13 | One-dimensional characteristic signal processing method, device and system with time domain characteristics |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111227823A CN111227823A (en) | 2020-06-05 |
| CN111227823B true CN111227823B (en) | 2021-05-18 |
Family
ID=70867026
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010030908.2A Active CN111227823B (en) | 2020-01-13 | 2020-01-13 | One-dimensional characteristic signal processing method, device and system with time domain characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111227823B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113536676B (en) * | 2021-07-15 | 2022-09-27 | 重庆邮电大学 | Lithium battery health condition monitoring method based on feature transfer learning |
| CN113780106B (en) * | 2021-08-24 | 2024-02-27 | 电信科学技术第五研究所有限公司 | Deep learning signal detection method based on radio waveform data input |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101658425B (en) * | 2009-09-11 | 2011-06-01 | 西安电子科技大学 | Device and method for detecting attention focusing degree based on analysis of heart rate variability |
| CN110037684A (en) * | 2019-04-01 | 2019-07-23 | 上海数创医疗科技有限公司 | Device based on the identification rhythm of the heart type for improving convolutional neural networks |
| CN109864714A (en) * | 2019-04-04 | 2019-06-11 | 北京邮电大学 | A kind of ECG Signal Analysis method based on deep learning |
-
2020
- 2020-01-13 CN CN202010030908.2A patent/CN111227823B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111227823A (en) | 2020-06-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108478209B (en) | Electrocardio information dynamic monitoring method and dynamic monitoring system | |
| Gradl et al. | Real-time ECG monitoring and arrhythmia detection using Android-based mobile devices | |
| Pecchia et al. | Remote health monitoring of heart failure with data mining via CART method on HRV features | |
| CN105796096B (en) | A kind of heart rate variance analyzing method, system and terminal | |
| WO2016024495A1 (en) | Biological-signal measurement system, biological-information measurement device, and method for changing biological-information extraction algorithm | |
| Zhang et al. | ECG quality assessment based on a kernel support vector machine and genetic algorithm with a feature matrix | |
| CN101385644A (en) | 12 lead wireless remote electrocardiograph monitoring system | |
| CN111227823B (en) | One-dimensional characteristic signal processing method, device and system with time domain characteristics | |
| CN109288515B (en) | Periodicity monitoring method and device based on premature beat signal in wearable electrocardiosignal | |
| CN110353704B (en) | Emotion evaluation method and device based on wearable electrocardiogram monitoring | |
| CN112971800B (en) | Deep learning-based electrocardio premature beat signal processing method, module and device | |
| CN112869752A (en) | Electrocardiosignal acquisition device and quality grade evaluation and QRS wave detection method | |
| Yen et al. | A portable real-time ECG recognition system based on smartphone | |
| Hadiyoso et al. | Arrhythmia detection based on ECG signal using Android mobile for athlete and patient | |
| Yoshimoto et al. | Recent progress of biomedical processor soc for wearable healthcare application: A review | |
| Zhao et al. | Towards deep learning-based detection scheme with raw ECG signal for wearable telehealth systems | |
| CN106264522A (en) | A kind of Noninvasive Blood Glucose Detection Methods based on Bioelectrical impedance analysis and device thereof | |
| Secerbegovic et al. | The research mHealth platform for ECG monitoring | |
| Marhoon et al. | A new real-time resource-efficient algorithm for ECG denoising, feature extraction and classification-based wearable sensor network | |
| CN205251538U (en) | Intelligent underwear of heart electrograph guardianship function based on bluetooth | |
| CN113317795B (en) | Signal measurement method and device | |
| Goh et al. | Issues in implementing a knowledge-based ECG analyzer for personal mobile health monitoring | |
| Plesnik et al. | ECG signal acquisition and analysis for telemonitoring | |
| Kumar et al. | Remote health monitoring system for detecting cardiac disorders | |
| Jokić et al. | An efficient ECG modeling for heartbeat classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |