CN113361596B - Sensor data augmentation method, system and storage medium - Google Patents
Sensor data augmentation method, system and storage medium Download PDFInfo
- Publication number
- CN113361596B CN113361596B CN202110623634.2A CN202110623634A CN113361596B CN 113361596 B CN113361596 B CN 113361596B CN 202110623634 A CN202110623634 A CN 202110623634A CN 113361596 B CN113361596 B CN 113361596B
- Authority
- CN
- China
- Prior art keywords
- data
- drift
- sensor
- matrix
- sensing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01K—MEASURING TEMPERATURE; MEASURING QUANTITY OF HEAT; THERMALLY-SENSITIVE ELEMENTS NOT OTHERWISE PROVIDED FOR
- G01K15/00—Testing or calibrating of thermometers
- G01K15/005—Calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Testing Or Calibration Of Command Recording Devices (AREA)
Abstract
The invention discloses a method and a system for augmenting sensor data and a storage medium, and belongs to the technical field of sensor data processing. The method includes the steps that a random cutting method is adopted for data without drift to achieve augmentation of data samples without drift, a plurality of function models which accord with drift characteristics of a sensor are selected as trend items to construct a non-stable random walk process containing the trend items to simulate the data drift process of the sensor, drift amount simulation is achieved by setting a drift probability threshold of the sensor and determining a maximum drift range according to data characteristics, and the augmented data samples without drift are augmented to achieve augmentation. According to the method, the time and space characteristics of the drift data of the sensor are considered comprehensively by fusing the data of each adjacent sensor in the sensing field, so that the correctness of the drift amount simulation and the diversity of the characteristics of the augmented data are ensured, and the problem of poor generalization capability of the model caused by the shortage of training samples in the sensor drift calibration model is solved.
Description
Technical Field
The invention relates to the technical field of sensor data processing, in particular to a sensor data augmentation method, a system, a storage medium and a sensor data drift calibration method.
Background
Drift amount of sensor data generated along with time in the agricultural internet of things is a key restriction factor influencing data acquisition quality of a sensor, intelligent regulation and control of internet of things equipment and effective analysis of data cannot be guaranteed due to the fact that the sensor drift phenomenon exists, independent unloading and recalibration of the sensor are often difficult to achieve due to the fact that the sensor is usually deployed in a large scale and for a long time, and therefore calibration of the sensor is more and more important under the condition that no drift real signal cannot be obtained.
At present, the inventors have attracted their attention to calibration of sensors based on deep learning. The Convolutional Neural Network (CNN) is a subset of deep learning, has been applied to other fields, and achieves good results. Research on the application of CNN to the time series field is also growing. For example, the electroencephalogram signal is denoised, the method can establish accurate mapping from the noise signal to the electroencephalogram signal, real-time denoising is realized, and the efficiency and quality of the electroencephalogram signal denoising can be effectively improved. However, this method is limited to denoising in a single time series of data, and is not suitable for drift calibration of multiple sensors.
Furthermore, bao et al, 2018, in Structural Health Monitoring,18 (2), 401-421, convert time series signals into image vectors, segment-render them in grayscale images, and input training data sets consisting of randomly selected and manually labeled image vectors into a deep neural network or a set of deep neural networks trained by stacked autocoders and greedy hierarchical training techniques, which can more accurately detect time series data, including multi-modal anomalies including data drift, but this method can only detect and classify anomalous patterns in time series data, and does not involve calibration of sensor drift. Tian et al propose a novel deep learning model in IEEE Access,8,121385-121397 in 2020, and combine unsupervised technology and supervised technology to realize drift compensation of an electronic nose, but the method is complex and does not involve training set construction method research, and the applicable scene is limited.
Therefore, the existing method cannot effectively meet the requirement of sensor drift calibration based on the analysis. The deep learning has the capability of automatically learning features from a large amount of data, and the deep learning method has great application prospect when being applied to the field of sensor drift calibration. Therefore, deep learning is adopted to automatically extract drift characteristics in the sensing data, and it is very meaningful work to realize drift calibration of the sensor. However, training of deep learning models such as neural networks requires a large number of data samples, and how to construct sufficient drift-containing data samples and iteratively input the appropriate size of the training samples is an urgent problem to be solved by the person skilled in the art; on the other hand, how to ensure the accuracy of extracting the drift features and the accuracy of calibration is also a difficult problem.
Disclosure of Invention
1. Problems to be solved
Aiming at the problems that a large number of data samples are needed for training deep learning models such as a neural network and the like in the prior art, and how to construct sufficient drift-containing data samples and how to iteratively input proper sizes of the training samples, the invention provides a sensor data augmentation method. A squeezing-exciting (SE) module is embedded in a Residual block of a Residual network (ResNet), and a drift calibration method based on the squeezing-exciting Residual network (SE-ResNet) is constructed to realize sensor drift data calibration and effectively improve the efficiency and quality of sensor drift calibration.
2. Technical scheme
In order to solve the above problems, the present invention adopts the following technical solutions.
A first aspect of the invention provides a method of augmenting sensor data, the method comprising:
and SA: deploying the calibrated sensor to construct a target sensing field;
SB: acquiring first sensing data of calibrated sensors in a target sensing field, cutting the first sensing data into a plurality of data matrixes by using a random cutting method, selecting partial row data in the data matrixes from the single cut data matrixes according to the positions of the sensors corresponding to each row in the matrix in the sensing field to construct data matrixes of adjacent sensors, and determining first augmented data;
and (3) SC: and according to the data matrix cut by the first perception data, simulating the drifting process of the sensor data by using a non-stationary random walk process containing a trend item to obtain second perception data, selecting a drifting amount matrix with the same position and size as the first augmented data from the second perception data, and determining the drifting amount matrix as the second augmented data.
In some embodiments, the first sensing data is determined according to the continuous L sampling points collected by the N calibrated sensors in the target sensing field, and is recorded as an N × L drift-free data matrix X, where N matrix row data in the drift-free data matrix X respectively correspond to data collected by the N sensors in the sensing field;
randomly cutting a plurality of data matrixes with the size of NxL from the drift-free data matrix X, and recording any one of the cut data matrixes as X l Said data matrix X l Number of columns of said drift-free data matrix X, from a single clipped data matrix X l From the data matrix X in dependence on the position in the sensing field of the sensor corresponding to each matrix row l N rows of data are selected to construct an nxl-sized data matrix X of the adjacent sensors b And recording as first augmented data; specifically, it can be expressed as:
wherein
The initial cutting position of the sampling point is defined as l, the column number of a cutting data matrix is defined as the number of continuous sampling points of the sensor corresponding to each matrix row; 1,
represents taking a matrix of
To
Column data; n is from the clipped data matrix X l The selected number of rows.
Constructing a drift amount simulation matrix D with the size of Nxl by adopting a non-stationary random walk process containing trend items l According to from X l Selecting n rows of data to construct X b Method from D l Constructing drift amount matrix D with size of nxl by n rows of data with same row position selected b As second augmented data; specifically, the method can be represented as follows:
wherein d is the drift amount of each sensor at different time.
In some embodiments, the number n of the proximity sensors is selected by:
adaptively determining the size of the number n of proximity sensors from the data matrix X clipped according to the data augmentation application scenario l Randomly selecting a matrix row as a reference, sequentially selecting n-1 adjacent sensors from small to large according to Euclidean distance between other sensors in a sensing field and the reference sensor corresponding to the matrix row, and finally selecting a data matrix X corresponding to the n sensors l The matrix row data in (1) is the n row data to be selected.
In some embodiments, the length l of the clipped data matrix is determined by the sensor data acquisition time interval and the characteristic period of the data distribution by:
wherein, T is a characteristic period of regular distribution of sensor data stream, Δ T is a time interval of data acquisition by the sensor, λ is a positive rational number, and a suitable positive integer is generally selected according to the required training sample size.
In some embodiments, a non-stationary random walk process with trend terms is used to construct a sensor drift amount simulation matrix D of size NxL l According to from X l Selecting n rows of data to construct X b Method from D l Constructing the n rows of data with the same row position as a drift amount matrix D with the size of n multiplied by l b And recording as second augmentation data;
setting a sensor drift probability threshold, and performing drift process simulation on sensor data by adopting a non-stationary random walk process containing a trend item, wherein the simulated sensor data can be represented as follows:
wherein, y i,t For the drift-containing data, x, of the sensor i at time t after simulation i,t For sensor i at time tNo drift data of d i,t For the drift amount generated by the sensor i at the time t through simulation, rand (0, 1) is a random floating point number between 0 and 1, alpha is a floating point number and alpha epsilon (0, 1) is a probability threshold value of the sensor drifting when rand (0, 1)>When alpha is reached, the sensor i does not drift, and the generated drift amount is zero; otherwise, drift occurs;
when drift amount simulation is carried out on the data of each sensor, one of a linear function, an exponential function, a square root function and a sine function is selected in a self-adaptive mode to serve as a trend item in the non-stationary random walk process containing the trend item, and the trend item is used for generating a corresponding drift trend; the drift amount simulation specific mode is as follows:
in the linear drift trend, the drift amount of the sensor i at the time t can be represented as:
in the exponential drift trend, the drift amount of the sensor i at the time t can be represented as:
in the square root drift trend, the drift amount of the sensor i at the time t can be represented as:
in the sinusoidal drift trend, the drift amount of the sensor i at the time t can be represented as:
wherein e is the maximum drift amount in each trend term, u i,t For the random walk data amount, r is an angular velocity parameter.
In some embodiments, the method for selecting the drift amount simulation parameters of each trend item is as follows:
the value of the maximum drift e in each trend item is determined by the collected data characteristics and the column number of the cutting data matrix, and is positively correlated with the column number of the data matrix, specifically
Wherein s is a standard deviation of the normalized data in the characteristic period T, and Δ T is a sensor data sampling interval. Amount of random walk data u i,t ～iid(0,σ 2 ) Wherein
The angular velocity parameter r is used to adjust the sinusoidal period, and
a second aspect of the invention provides a method of calibrating drift of a sensor, the method comprising:
and acquiring sensor data to be calibrated, inputting the sensor data to be calibrated into the trained squeeze-excitation residual error network model, and outputting corresponding drift calibration data.
Wherein the squeeze-excitation residual error network model is obtained by training a plurality of groups of data samples; the plurality of sets of data samples are data sets constructed from first augmented data and second augmented data.
In some embodiments, the data set constructed from the first augmented data and the second augmented data comprises:
taking the first augmented data as a drift-free data sample X b Taking the second augmentation data as a drift amount simulation data sample D b ；
Performing matrix addition on the drift-free data sample and the drift amount simulation data sample to obtain a drift-containing data sample Y b (ii) a Using a plurality of drift-free data samples X b Composing a drift-free data set X M Forming a drift-containing data set Y by using a plurality of drift-containing data samples M (ii) a Will contain the drift data set Y M And drift-free data set X M As a data set for training the squeeze-to-excite residual network model. A third aspect of the present invention provides a sensor data augmentation system, comprising:
the perception field construction module is used for deploying the calibrated sensor and constructing a target perception field;
the non-drift data sample construction module is used for collecting first sensing data of a calibrated sensor in a target sensing field, cutting the first sensing data into a plurality of data matrixes by using a random cutting method, and selecting partial rows in the data matrixes to construct adjacent sensor data matrixes as first augmented data according to the positions of the sensors corresponding to the rows in the matrixes in the sensing field from the single cut data matrix, wherein the first augmented data are non-drift data samples;
and the drift amount simulation data sample construction module is used for carrying out drift process simulation on the sensor data by utilizing a non-stationary random walk process containing a trend item according to a data matrix cut by the first sensing data to obtain second sensing data, selecting a drift amount matrix with the same position and size as the first augmented data from the second sensing data, and determining the drift amount matrix as second augmented data, wherein the second augmented data is a drift amount simulation data sample.
A fourth aspect of the invention provides a readable storage medium having stored thereon a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method as described above.
3. Advantageous effects
Compared with the prior art, the invention has obvious technical advantages that:
(1) According to the data amplification method, the sensor data drifting process is established by adopting the non-stationary random walk process containing the trend item, the amplification operation of the sensor data is realized by utilizing the method of randomly cutting and selecting the data of the adjacent sensor, and the characteristic diversity of the amplification data is increased; the data augmentation method can fuse time and space characteristics of sensor drift data, and solves the problem of poor model generalization capability caused by insufficient training samples in a sensor drift calibration model.
(2) According to the data augmentation method, a sensor drifting process is established by adopting a non-stationary random walk process containing a trend item, a probability threshold value of sensor drifting is set according to uncertainty of sensor drifting, and meanwhile, the range of the maximum drifting amount of the trend item is set according to characteristics of sensing data to control the drifting amplitude of the sensing data, so that controllability is improved; 4 types of function models conforming to the drift characteristics are used as the trend item to simulate the drift characteristics of the sensor, and each drift quantity simulation parameter is selected based on the data characteristics, so that the applicability of the drift data sample construction method in different application scenes and different data types is ensured.
(3) According to the data augmentation method provided by the invention, the data of each adjacent sensor in the sensing field is fused in the same training sample, and the size of the cutting data matrix is determined as the iterative input value of the sensor drift calibration network model according to the sampling interval and the data characteristic period of the sensor data, so that the problem of data characteristic loss caused by iterative input of the data sample can be avoided, the input size of the neural network can be ensured to be moderate, and the operation cost is effectively reduced.
(4) The invention provides a sensor data drift calibration method, which adopts an extrusion-excitation residual error network to construct a sensor drift calibration model, wherein an extrusion-excitation residual error module contained in the sensor drift calibration model can extract time and space characteristics of data to realize calibration of sensor drift data by utilizing the correlation of adjacent sensor data; the data augmentation method can fuse the time and space characteristics of the sensor drift data and provide data guarantee for training the deep neural network.
(5) The sensor data drift calibration method provided by the invention can provide guarantee for accurately extracting the drift characteristics in the data of a plurality of sensors, and ensures the reliability of simultaneously extracting the data characteristics of the plurality of sensors in the sensing field and compensating the drift. By adopting the squeeze-excitation residual error network, the correlation of the data of the adjacent sensors can be effectively utilized, the network depth is increased, and the drift calibration quality of the data of the sensors is effectively improved.
(6) The data augmentation method provided by the invention has strong expandability, the data of a plurality of sensors in a sensing field and the data of a single sensor in different time periods can be used as basic data required by data augmentation, and the basic data is used for acquiring a required data set by the data augmentation method.
Drawings
Fig. 1 is a flowchart of a data augmentation method according to an embodiment of the present invention;
fig. 2 is a block diagram of a data augmentation system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data set expansion method provided by the present invention;
FIG. 4 is a flow chart of data set construction provided by an embodiment of the present invention;
FIG. 5 is a flow chart of a method for calibrating sensor drift provided by the present invention;
FIG. 6 is a schematic diagram of a SE-ResNet residual module provided by the present invention;
FIG. 7 is a histogram of SE-ResNet and ResNet drift calibration errors under different drift trends provided by an embodiment of the present invention;
Detailed Description
Hereinafter, embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments of the present application, and it should be understood that the present application is not limited by the embodiments described herein.
Example 1
As shown in fig. 1, the embodiment of the present application discloses a method for augmenting sensor data, which comprises the following general processes:
and SA: and deploying the calibrated sensors to construct a target perception field.
Specifically, in soil environment information monitoring, calibrated sensors are deployed to construct a target sensing field, in this example, the sensors arranged in the target sensing field are soil temperature and humidity sensors, the number of the sensors is 20, the sampling interval of the sensors is 5 minutes, the characteristic period of sensing data flow regular distribution is 24 hours, and soil temperature data of 90 days, namely, 2 months, 1 day to 4 months, 30 days after 2020 years of sensor calibration, is taken as initial acquisition data of the sensors. The periodic characteristics refer to that the distribution rule of the sensing data changes periodically along with time.
This data can be considered drift-free by collecting initial sensing data of calibrated sensors within the sensing field, and this data is used as a basis for drift data samples and calibration data required for building a sensor drift calibration network model.
SB: the method comprises the steps of collecting first sensing data of calibrated sensors in a target sensing field, cutting the first sensing data into a plurality of data matrixes by using a random cutting method, selecting partial row data in the data matrixes from the single cut data matrixes according to the positions of the sensors corresponding to all rows in the matrixes in the sensing field to construct data matrixes of adjacent sensors, and determining first augmented data.
Specifically, the method comprises the following steps: SB1: modeling the first perception data as a drift-free data matrix.
And selecting the soil temperature data of 90 days in total from 1 day 2 month to 30 days 4 months in 2020, which is acquired by the calibrated soil temperature sensor from the target sensing field as first sensing data. The number N of the sensors is 20, the sampling interval of the sensors is 5 minutes, the characteristic period is 24 hours, and the sensors are recorded as a drift-free data matrix X with the size of 20 multiplied by 25920, wherein 20 matrix row data in the drift-free data matrix X respectively correspond to data collected by 20 sensors in a sensing field;
SB2: cutting the drift-free data matrix X into a plurality of data matrixes, wherein the cut data matrixes X l Is smaller than the number L of columns of the drift-free data matrix X.
Specifically, in order to avoid the loss of data characteristics, a drift-free data matrix needs to be cut into a data matrix with a smaller size according to data acquisition characteristics in a specific monitoring scene, and a data matrix with the size of nxl is randomly cut from the drift-free data matrix X and is marked as X l 。
SB3: and selecting a part of row data in the data matrix from the cut data matrix according to the positions of the sensors corresponding to the rows in the matrix in the sensing field from the single cut data matrix to construct a data matrix adjacent to the sensors so as to determine first augmented data.
From the data matrix X l Selecting n rows of data to construct a proximity sensor data matrix X with the size of nxl b And recording as first augmentation data, the specific mode is as follows:
wherein
And l is the column number of a cutting data matrix, namely the number of continuous sampling points of the sensor corresponding to each matrix row. 1,
represents taking a matrix of
To
Column data. n is from the clipped data matrix X l The selected number of rows.
As a variation, the method for selecting the number n of sensors in this example is:
determining the size of the number n of adjacent sensors according to the neural network receptive field, and obtaining the data matrix X from the cut data matrix l Randomly selecting a matrix row as a reference, sequentially selecting n-1 adjacent sensors from small to large according to Euclidean distances between other sensors in a sensing field and the reference sensors corresponding to the matrix row, and finally selecting a data matrix X corresponding to the n sensors l Matrix row data in (1)I.e. the n rows of data that need to be selected. In this example n is chosen to be 10, where n is determined by the neural network receptive field. It should be noted that the size of each data sample is determined by selecting the number of sensors, the neural network receptive field is determined by the size and step size of the convolution kernel, and the convolution kernel slides on the data sample to obtain the features.
The column number l of the cut data matrix is determined by the time interval of data acquisition of the sensor and the characteristic period of data distribution, and the specific calculation mode is as follows:
wherein, T is a characteristic period of regular distribution of sensor data flow, delta T is a time interval of data acquisition of the sensor, lambda is a positive rational number, and a proper positive integer is generally selected according to the size of a required training sample.
In this example, λ select 8 is calculated by equation (3) to obtain 2304 as column number l of data matrix, and data matrix X is then obtained l Can be represented as a two-dimensional matrix with a shape of 20 × 2304, and the drift-free data sample X with a shape of 10 × 2304 is constructed by extracting the neighboring 10 sensor data according to equation (2) b 。
SC: method for constructing drift amount simulation matrix D with size of Nxl by adopting non-stationary random walk process containing trend item l According to from X l N matrix rows are selected to construct X b From D to l N matrix row data are selected to construct a drift amount matrix D with the size of nxl b As second augmented data, wherein the second sensing data is a drift amount simulation matrix; the second augmentation data is a drift amount simulation data sample, which can be specifically expressed as:
where d is the drift amount of each sensor at different times.
Specifically, when a sensor drift process is constructed, drift occurring among different sensors is generally independent, a sensor drift probability threshold is set in the example, a non-stationary random walk process containing a trend item is adopted to simulate the drift process of sensor data, and the simulated sensor data can be represented as follows:
wherein, y i,t For the drift-containing data, x, of the sensor i at time t after simulation i,t For drift-free data at time t for sensor i, d i,t The amount of drift generated at time t is simulated for sensor i. rand (0, 1) is a random floating point number between 0 and 1, α is a floating point number and α ∈ (0, 1) is a probability threshold for the sensor to drift, when rand (0, 1)>When alpha is reached, the sensor i does not drift, and the generated drift amount is zero; otherwise, drift occurs;
and when the drift amount simulation is carried out on the data of each sensor, one of a linear function, an exponential function, a square root function and a sine function is selected in a self-adaptive mode to serve as a trend item in the non-stationary random walk process containing the trend item, so that the corresponding drift trend is generated. The drift amount simulation concrete mode is as follows:
in the linear drift trend, the drift amount of the sensor i at the time t can be expressed as follows:
in the exponential drift trend, the drift amount of the sensor i at the time t can be represented as:
in the square root drift trend, the drift amount of the sensor i at the time t can be represented as:
in the sinusoidal drift trend, the drift amount of the sensor i at the time t can be expressed as:
wherein e is the maximum drift amount in each trend term, u i,t For random walk data amount, r i Is an angular velocity parameter.
In some embodiments, the method for selecting the drift amount simulation parameters of each trend item is as follows:
the value of the maximum drift e in each trend item is determined by the characteristics of the acquired data and the column number of the cutting data matrix, and is positively correlated with the column number of the data matrix, specifically
Wherein s is a standard deviation of the normalized data in the characteristic period T, and Δ T is a sensor data sampling interval. Amount of random walk data u i,t ～iid(0,σ 2 ) Wherein
The angular velocity parameter r is used to adjust the sine period, an
In this example, in order to ensure the sufficiency of the drift samples, the threshold value of the probability of drift occurring in each sensor in the same time period in which the number of sampling points is l is set to 0.5, i.e., the probability of drift occurring in 10 sensors of the same data matrix is 50%. The 4-class drift amount simulation mode can adjust various drift occurrence probabilities according to application scenarios, and in this example, 4-class data sets are respectively constructed by adopting the 4-class sensor drift trends. The maximum drift e to E in each trend item is calculatedU (2.97, 5.94), amount of random walk data U i,t ～iid(0,σ 2 ) Middle sigma-U (0.03, 0.06); angular velocity parameters r-U (2, 4).
In one possible embodiment, the first augmented data and the second augmented data are constructed into a data set as follows:
SD: will not drift data sample X b And drift amount simulation data sample D b Performing matrix addition to obtain drift-containing data samples Y b 。
Specifically, there will be no drift data samples X b And drift amount simulation data sample D b The drift-containing data samples obtained by performing the matrix addition should be:
Y b ＝X b +D b (11)
and SE: selecting multiple drift data samples X b And a plurality of drift-containing data samples Y b Separately form a drift-free data set X M And a drift-containing data set Y M (ii) a Will have no drift data set X M Data set Y containing drift M And respectively dividing one part of the training set into a network training set and the other part of the training set into a network test set, wherein the network training set and the network test set are not crossed.
Specifically, a measurement matrix of 20 × 25920 is obtained from the perceptual field by randomly clipping 8000 data matrices X of size 20 × 2304 according to equation (1) l Selecting 10 proximity sensors to obtain 8000 drift-free data samples X with the size of 10X 2304 according to the formula (2) b As a drift-free data set, 8000 drift-containing data samples Y of size 10X 2304 were likewise obtained in this manner b As a drift-containing data set Y M . In summary, in the sensor data sample construction process, the training sample size input in each iteration is determined according to the characteristic period and the sampling interval of data in the data stream of the internet of things, the drift process of the sensor is simulated by adopting the non-stationary random walk process containing the trend item, large-scale high-quality basic data are provided for the drift calibration of the sensor, the problem of insufficient training samples when the sensor is calibrated by adopting a deep learning method can be solved, and the data acquisition quality is guaranteed.
Example 2
As shown in fig. 5, on the basis of embodiment 1, the present embodiment discloses a sensor drift calibration method, which includes the following general processes:
acquiring sensor data to be calibrated, inputting the sensor data to be calibrated into a trained squeeze-excitation residual error network model, and outputting corresponding drift calibration data;
wherein the squeeze-excitation residual error network model is obtained by training a plurality of groups of data samples; the plurality of sets of data samples are data sets constructed from first augmented data and second augmented data.
Specifically, referring to step SF, the squeeze-excitation residual network model training step is as follows: an extrusion-excitation module is added into a one-dimensional convolution residual error network to fully utilize data correlation between adjacent sensors to realize sensor drift calibration, and as shown in fig. 6, the method specifically comprises the following steps:
SF1: the 1 one-dimensional convolutional layer connected with the network input is used as the input of the residual error module, and the 1 one-dimensional convolutional layer connected with the output of the residual error module is used for adjusting the output result to be the same as the input size. The input of the residual error module is connected with 3 one-dimensional convolutional layers, each one-dimensional convolutional layer comprises convolution operation, batch normalization and activation function mapping output, the output of the last one-dimensional convolutional layer comprises two network branches, one network branch is connected with the extrusion-excitation module, and the other network branch is multiplied by the output result of the extrusion-excitation module. The extrusion-excitation module mainly comprises a global pooling layer and two full-connection layers, wherein the first full-connection layer adopts an activation function Relu as output, and the second full-connection layer adopts a Sigmoid function as output of the activation function.
In the example, the network input sample is a data matrix with 10 rows, so that the number of convolution kernels of a 1 st convolution layer input into the sensor drift calibration network in a channel 10 mode is 32, the size of the convolution kernels is 1 × 5, the input of a residual error module is connected with 3 one-dimensional convolution layers, the number of convolution kernels of each one-dimensional convolution layer is 32, the size of the convolution kernels is 1 × 3, the number of channels of a squeezing-actuating module in an SE-ResNet model is reduced by half through the number of channels of a first full connection layer, and the number of channels is reduced to 32 through a next full connection layer after being activated by a Relu activation function; the last one-dimensional convolutional layer sets the number of channels to 10 to keep the same as the initial input size.
SF2: respectively adopting 4 drift trends to construct 4 drift-containing data sets and non-drift data sets, and selecting a drift-containing data set Y M 6400 drift-containing data samples in the data sample group, namely the first 80% of the data samples are used as a training set of the neural network; the other 1600 data samples, namely the last 20%, are taken as a neural network test set, and no intersection exists between the network training set and the network test set.
SF3: will contain the drift data set Y M Drift-containing data samples Y in partitioned training set b Iteratively inputting the data into a drift calibration network for training, and adopting corresponding drift-free data samples X b As the true output value of the neural network, the loss of the neural network adopts the mean square error function to achieve the purpose of data drift compensation, and the loss of the designed calibration module is as follows:
wherein m is the number of training samples,
for the output calibration data, X i Corresponding drift-free data samples.
The Adam optimizer is adopted for iterative training in the example, the Size of Batch Size is set to be 200, the number of iterations is 5000, and the network learning rate is set to be 0.001, so that a trained squeeze-excitation residual network model is obtained.
SG: calibrating drift sensor data, inputting a drift data-containing sample of the test set as data to be calibrated into the trained squeeze-excitation residual error network model, and outputting corresponding drift calibration data.
The convolutional neural network extracts time and space characteristics in data containing drift from historical sensing data through convolutional layer characteristic learning capacity, and drift calibration of the sensor is achieved. However, the perception data collected from a specific sensor network is limited, the data containing drift amount is more deficient, and the requirement of the neural network on training samples is difficult to guarantee.
This example employs, as the first sensing data, soil temperature sensing data of 90 days from 1 day on 2 months to 30 days on 4 months in 2020. The sample data augmentation method provided by the invention meets the sample data required by neural network training, and the sample data obtained by the method has no obvious difference with drift data in a real scene. The method has the advantages that the neural network model is trained by adopting sample data with different drift trends, experimental results show that the method can effectively solve the problem of data drift in data streams of the Internet of things, root mean square errors serve as evaluation standards, 20 total data sets are adopted and are respectively tested for 20 times, the average value of the experimental results is shown in figure 7, and the total drift calibration error is only about 1/2 of that of the traditional calibration method based on the residual error neural network.
Example 3
As shown in fig. 2, the present embodiment provides a sensor data augmentation system, which includes:
a perception field construction module 20 for deploying the calibrated sensors to construct a target perception field; the perception field construction module also comprises a proximity sensor data determination module which adaptively determines the number n of proximity sensors according to the data augmentation application scene and cuts the data matrix X l Randomly selecting a matrix row as a reference, sequentially selecting n-1 adjacent sensors from small to large according to Euclidean distance between other sensors in a sensing field and the reference sensor corresponding to the matrix row, and finally selecting n sensor data in a data matrix X l The corresponding n rows of data in the data collection are the n rows of proximity sensor data required to be selected.
The drift-free data sample construction module 30 is configured to collect first sensing data of a calibrated sensor in a target sensing field, cut the first sensing data into a plurality of data matrices by using a random cutting method, select a part of rows of data in a single cut data matrix according to positions of sensors corresponding to rows in the matrix in the sensing field, construct a data matrix of an adjacent sensor, and determine first augmented data.
Specifically, modeling the first sensing data into a drift-free data matrix, wherein a plurality of matrix rows in the drift-free data matrix respectively correspond to data collected by a plurality of sensors in the sensing field; cutting the drift-free data matrix X into a plurality of data matrices, wherein the data matrices X l Is less than the column number L of the drift-free data matrix; selecting partial row data from a single cut data matrix according to the position of a sensor corresponding to each matrix row in a sensing field to construct an adjacent sensor data matrix, and determining first augmentation data, wherein the first augmentation data are drift-free data samples; the drift-free data sample construction module also comprises a data matrix column number cutting calculation module which is used for determining the column number l of a data matrix to be cut according to the time interval of data acquisition of the sensor and the characteristic period of data distribution, and the calculation mode is as follows:
and the drift amount simulation data sample construction module 40 is used for performing drift process simulation on the sensor data by utilizing a non-stationary random walk process containing a trend item according to the data matrix cut by the first sensing data to obtain second sensing data, selecting a drift amount matrix with the same position and size as the first augmented data from the second sensing data, and determining the drift amount matrix as second augmented data, wherein the second augmented data is a drift amount simulation data sample.
Example 4
The present embodiments provide an exemplary computer program product and computer-readable storage medium
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the decision-making method according to various embodiments of the present application described in the "exemplary methods" section of this specification above.
The computer program product may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages, for carrying out operations according to embodiments of the present application. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a decision-making method according to various embodiments of the present application described in the "exemplary methods" section above of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present application have been described above with reference to specific embodiments, but it should be noted that advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and the advantages, effects, etc. must not be considered to be possessed by various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is provided for purposes of illustration and understanding only, and is not intended to limit the application to the details which are set forth in order to provide a thorough understanding of the present application.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (8)
1. A method of sensor data augmentation, the method comprising:
and SA: deploying the calibrated sensor, and constructing a target sensing field;
SB: acquiring first sensing data of calibrated sensors in a target sensing field, cutting the first sensing data into a plurality of data matrixes by using a random cutting method, selecting partial row data in the data matrixes from the single cut data matrixes according to the positions of the sensors corresponding to all rows in the matrixes in the sensing field to construct data matrixes of adjacent sensors, and determining first augmented data;
SC: according to the data matrix cut by the first sensing data, performing drift process simulation on the sensor data by using a non-stationary random walk process containing a trend item to obtain second sensing data, selecting a drift amount matrix with the same position and size as the first augmented data from the second sensing data, and determining the second augmented data;
the determining first augmented data comprises:
determining first sensing data according to continuous L sampling points acquired by calibrated N sensors in a target sensing field, and recording the first sensing data as a non-drifting data matrix X with the size of NxL, wherein N matrix row data in the non-drifting data matrix X respectively correspond to data acquired by the N sensors in the sensing field;
randomly cutting a plurality of data matrixes with the size of NxL from the drift-free data matrix X, and recording any one of the cut data matrixes as X l Said data matrix X l Is smaller than the number of columns L of the drift-free data matrix X, from a single clipped data matrix X l From the data matrix X in dependence on the position in the sensing field of the sensor corresponding to each matrix row l Selecting n rows of data to construct n multiplied by l adjacent sensor numberAccording to matrix X b And recorded as the first augmented data.
2. The method of claim 1, wherein the specific formula for determining the first augmented data is:
wherein
The initial cutting position of the sampling point is defined as l, the column number of a cutting data matrix is defined as the number of continuous sampling points of the sensor corresponding to each matrix row; 1,
to express taking the matrix
To
Column data; n is a secondary cut data matrix X l The selected number of rows.
3. The sensor data augmentation method of claim 2, wherein the n-row data selection method comprises:
adaptively determining the size of the number n of proximity sensors from the data matrix X clipped according to the data augmentation application scenario l In the method, a matrix row is randomly selected as a reference, and other sensors in a sensing field and reference sensors corresponding to the matrix row are arranged between the other sensorsSequentially selecting n-1 adjacent sensors from small to large according to Euclidean distance, and finally selecting data matrix X corresponding to the n sensors l The matrix row data in (1) is the n row data to be selected.
4. The method of claim 2, wherein the number of columns/of the clipped data matrix is determined by the characteristic period of the data distribution and the time interval of the data acquisition of the sensor, and is calculated by:
wherein, T is a characteristic period of regular distribution of sensor data flow, delta T is a time interval of data acquisition of the sensor, lambda is a positive rational number, and a proper positive integer is generally selected according to the size of a required training sample.
5. The method according to claim 2, wherein the second sensing data is a drift amount simulation matrix D with size of NxL constructed by a non-stationary random walk process with trend terms l According to from X l Selecting n rows of data to construct X b Method from D l Constructing a drift amount matrix D with the size of nxl from n rows of data with the same row position b And recording as second augmentation data;
setting a sensor drift probability threshold, and performing drift process simulation on sensor data by adopting a non-stationary random walk process containing a trend item, wherein the simulated sensor data can be expressed as follows:
wherein, y i,t For the drift-containing data, x, of the sensor i at time t after simulation i,t For drift-free data of sensor i at time t, d i,t For sensor i, at time t by simulationThe amount of drift produced; rand (0, 1) is a random floating point number between 0 and 1, alpha is a floating point number and alpha epsilon (0, 1) is a probability threshold value of the sensor drifting when rand (0, 1)>When alpha is reached, the sensor i does not drift, and the generated drift amount is zero; otherwise, drift occurs;
and when drift amount simulation is carried out on the data of each sensor, one of a linear function, an exponential function, a square root function and a sine function is selected in a self-adaptive mode to serve as a trend item in the non-stationary random walk process containing the trend item, and the trend item is used for generating a corresponding drift trend.
6. A sensor drift calibration method is characterized by comprising the following steps:
acquiring sensor data to be calibrated, inputting the sensor data to be calibrated into a trained extrusion-excitation residual error network model, and outputting corresponding drift calibration data;
wherein the squeeze-excitation residual error network model is obtained by training a plurality of groups of data samples; the plurality of sets of data samples are data sets constructed from first augmented data and second augmented data;
the step of constructing a data set from the first augmented data and the second augmented data comprises:
taking the first augmented data as a drift-free data sample X b Taking the second augmentation data as a drift amount simulation data sample D b ；
The drift-free data sample X b And drift amount simulation data sample D b Performing matrix addition to obtain drift-containing data samples Y b (ii) a Using a plurality of drift-free data samples X b Composing a drift-free data set X M (ii) a Using a plurality of drift-containing data samples Y b Forming a drift-containing data set Y M (ii) a Will contain the drift data set Y M And drift-free data set X M As a training data set for the squeeze-excitation residual network model.
7. A sensor data augmentation system, comprising:
the perception field construction module is used for deploying the calibrated sensor and constructing a target perception field;
the drift-free data sample construction module is used for collecting first sensing data of a calibrated sensor in a target sensing field, cutting the first sensing data into a plurality of data matrixes by using a random cutting method, selecting part of row data in the data matrixes from the single cut data matrixes according to the positions of the sensors corresponding to all rows in the matrixes in the sensing field to construct adjacent sensor data matrixes, and determining first augmented data, wherein the first augmented data are drift-free data samples;
the drift amount simulation data sample construction module is used for simulating the drift process of the sensor data by utilizing a non-stationary random walk process containing a trend item according to a data matrix cut by the first sensing data to obtain second sensing data, selecting a drift amount matrix with the same position and size as the first augmented data from the second sensing data, and determining the drift amount matrix as second augmented data, wherein the second augmented data is a drift amount simulation data sample;
the determining first augmented data comprises:
determining first sensing data according to continuous L sampling points acquired by N calibrated sensors in a target sensing field, and recording the first sensing data as a non-drifting data matrix X with the size of NxL, wherein N matrix row data in the non-drifting data matrix X respectively correspond to data acquired by the N sensors in the sensing field;
randomly cutting a plurality of data matrixes with the size of NxL from the drift-free data matrix X, and recording any one of the cut data matrixes as X l Said data matrix X l Is smaller than the number of columns L of the drift-free data matrix X, from a single clipped data matrix X l From the data matrix X in dependence on the position in the sensing field of the sensor corresponding to each matrix row l N rows of data are selected to construct an nxl-sized data matrix X of the adjacent sensors b And recorded as first augmented data.
8. A readable storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110623634.2A CN113361596B (en) | 2021-06-04 | 2021-06-04 | Sensor data augmentation method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110623634.2A CN113361596B (en) | 2021-06-04 | 2021-06-04 | Sensor data augmentation method, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113361596A CN113361596A (en) | 2021-09-07 |
| CN113361596B true CN113361596B (en) | 2022-10-04 |
Family
ID=77532136
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110623634.2A Active CN113361596B (en) | 2021-06-04 | 2021-06-04 | Sensor data augmentation method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113361596B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105938116A (en) * | 2016-06-20 | 2016-09-14 | 吉林大学 | Gas sensor array concentration detection method based on fuzzy division and model integration |
| CN109684314A (en) * | 2018-12-17 | 2019-04-26 | 南京旅游职业学院 | A kind of wireless sensor network missing value estimation method based on space structure |
| CN109993088A (en) * | 2019-03-22 | 2019-07-09 | 江南大学 | A kind of blind calibration method of wireless sensor network data wander |
| CN110109470A (en) * | 2019-04-09 | 2019-08-09 | 西安电子科技大学 | Joint method for determining posture based on Unscented kalman filtering, satellite attitude control system |
| CN110288028A (en) * | 2019-06-27 | 2019-09-27 | 北京邮电大学 | ECG detecting method, system, equipment and computer readable storage medium |
| CN110906933A (en) * | 2019-11-06 | 2020-03-24 | 中国海洋大学 | AUV (autonomous Underwater vehicle) auxiliary navigation method based on deep neural network |
| CN112836439A (en) * | 2019-11-25 | 2021-05-25 | 罗伯特·博世有限公司 | Method and apparatus for processing sensor data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7705724B2 (en) * | 2007-07-25 | 2010-04-27 | Honeywell International Inc. | Augmented security system |
| CN108961308B (en) * | 2018-06-01 | 2021-07-02 | 南京信息工程大学 | Residual error depth characteristic target tracking method for drift detection |
-
2021
- 2021-06-04 CN CN202110623634.2A patent/CN113361596B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105938116A (en) * | 2016-06-20 | 2016-09-14 | 吉林大学 | Gas sensor array concentration detection method based on fuzzy division and model integration |
| CN109684314A (en) * | 2018-12-17 | 2019-04-26 | 南京旅游职业学院 | A kind of wireless sensor network missing value estimation method based on space structure |
| CN109993088A (en) * | 2019-03-22 | 2019-07-09 | 江南大学 | A kind of blind calibration method of wireless sensor network data wander |
| CN110109470A (en) * | 2019-04-09 | 2019-08-09 | 西安电子科技大学 | Joint method for determining posture based on Unscented kalman filtering, satellite attitude control system |
| CN110288028A (en) * | 2019-06-27 | 2019-09-27 | 北京邮电大学 | ECG detecting method, system, equipment and computer readable storage medium |
| CN110906933A (en) * | 2019-11-06 | 2020-03-24 | 中国海洋大学 | AUV (autonomous Underwater vehicle) auxiliary navigation method based on deep neural network |
| CN112836439A (en) * | 2019-11-25 | 2021-05-25 | 罗伯特·博世有限公司 | Method and apparatus for processing sensor data |
Non-Patent Citations (3)
| Title |
|---|
| A Drift-Compensating Novel Deep Belief Classification Network to Improve Gas Recognition of Electronic Noses;YUTONG TIAN 等;《IEEE Access》;20200731;第8卷;第121385-121397页 * |
| 基于长短期记忆神经网络的数据中心空调系统传感器故障诊断;王路瑶 等;《化工学报》;20181231;第69卷(第S2期);第252-259页 * |
| 遥感图像目标检测的数据增广研究;张磊 等;《测绘科学技术学报》;20191231;第36卷(第5期);第505-510页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113361596A (en) | 2021-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110674604B (en) | Transformer DGA data prediction method based on multi-dimensional time sequence frame convolution LSTM | |
| Kannan et al. | Prediction of daily rainfall state in a river basin using statistical downscaling from GCM output | |
| Broennimann et al. | Measuring ecological niche overlap from occurrence and spatial environmental data | |
| Bocco et al. | Comparison of regression and neural networks models to estimate solar radiation | |
| CN113919448A (en) | Method for analyzing influence factors of carbon dioxide concentration prediction at any time-space position | |
| CN110427654B (en) | Landslide prediction model construction method and system based on sensitive state | |
| CN110880369A (en) | Gas marker detection method based on radial basis function neural network and application | |
| Tsarouchi et al. | Historical and future land-cover changes in the Upper Ganges basin of India | |
| CN113361194B (en) | Sensor drift calibration method based on deep learning, electronic equipment and storage medium | |
| Maisano Delser et al. | Demographic inferences after a range expansion can be biased: the test case of the blacktip reef shark (Carcharhinus melanopterus) | |
| CN112287601B (en) | Method, medium and application for constructing tobacco leaf quality prediction model by using R language | |
| Yu et al. | Enhancing streamflow simulation using hybridized machine learning models in a semi-arid basin of the Chinese loess Plateau | |
| CN110852415B (en) | Vegetation index prediction method, system and equipment based on neural network algorithm | |
| Farahani et al. | Evaluating ecohydrological model sensitivity to input variability with an information-theory-based approach | |
| CN114324336A (en) | Nondestructive measurement method for biomass of soybean in whole growth period | |
| Lilleleht et al. | Spatial forest structure reconstruction as a strategy for mitigating edge-bias in circular monitoring plots | |
| CN113962333A (en) | Model training method, fine particulate matter concentration prediction device and electronic equipment | |
| CN110852475B (en) | Extreme gradient lifting algorithm-based vegetation index prediction method, system and equipment | |
| CN113361596B (en) | Sensor data augmentation method, system and storage medium | |
| CN117407707A (en) | Odor identification model training method and device | |
| CN115907178B (en) | Clean ecosystem CO 2 Exchange amount prediction method | |
| CN117036781A (en) | Image classification method based on tree comprehensive diversity depth forests | |
| CN116958809A (en) | Remote sensing small sample target detection method for feature library migration | |
| CN114998719A (en) | Forest fire prediction method based on deep learning and multi-source remote sensing data | |
| Qing-Dao-Er-Ji et al. | Research on data mining algorithm of meteorological observation based on data quality control algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |