CN116702076A - Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion - Google Patents

Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion Download PDF

Info

Publication number
CN116702076A
CN116702076A CN202310596134.3A CN202310596134A CN116702076A CN 116702076 A CN116702076 A CN 116702076A CN 202310596134 A CN202310596134 A CN 202310596134A CN 116702076 A CN116702076 A CN 116702076A
Authority
CN
China
Prior art keywords
cnn
fault diagnosis
training
model
feature fusion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310596134.3A
Other languages
Chinese (zh)
Inventor
于迪
傅海越
解志杰
詹昊
吕景亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northeast Forestry University
Original Assignee
Northeast Forestry University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northeast Forestry University filed Critical Northeast Forestry University
Priority to CN202310596134.3A priority Critical patent/CN116702076A/en
Publication of CN116702076A publication Critical patent/CN116702076A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/243Classification techniques relating to the number of classes
    • G06F18/2433Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/2163Partitioning the feature space
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/25Fusion techniques
    • G06F18/253Fusion techniques of extracted features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T90/00Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)

Abstract

A small sample transfer learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion relates to the field of rotary machinery fault diagnosis. The method solves the problems that the existing deep learning method cannot effectively process small samples and is difficult to be used for fault diagnosis of the rolling bearing. The method comprises the following steps: collecting vibration signals of the rolling bearing under an original working condition and a target working condition, and dividing the signals to construct a source domain sample data set and a target domain sample data set; extracting time-frequency characteristics of signals and constructing a time-frequency image dataset; constructing a CNN model of feature fusion; carrying out normalization processing on the original vibration signal and the corresponding time-frequency image, and carrying out CNN model training by utilizing source domain data to obtain a CNN training model; adjusting the CNN training model according to part of samples in the target domain to obtain a CNN fault diagnosis fine tuning model; and diagnosing the residual sample of the target domain by adopting the fine tuning model, and obtaining a fault diagnosis result. The intelligent detection method is applied to the field of intelligent detection of the rolling bearing.

Description

Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion
Technical Field
The invention relates to the field of fault diagnosis of rotary machinery, in particular to a small sample migration learning fault diagnosis method based on CNN feature fusion.
Background
Rolling bearings are key fundamental components of rotary machine transmissions, which are prone to failure when operated for long periods of time under extreme and complex operating conditions, resulting in economic losses and safety hazards. Therefore, in order to avoid economic losses and safety accidents caused by faults of the rolling bearings, it is necessary to monitor the operating conditions of the rolling bearings using fault diagnosis techniques.
The current fault diagnosis method of the rolling bearing mainly comprises a traditional method based on a signal processing technology and a method based on deep learning. Most of the traditional fault diagnosis methods need manual selection of characteristics, and considerable expertise is needed when analyzing a complex system, and the diagnosis result is very uncertain and poor in generalization. The deep learning method can greatly reduce the requirement of feature extraction on professional knowledge and reduce uncertainty caused by manual participation, and can directly complete end-to-end intelligent fault diagnosis.
The raw vibration signal of the bearing is usually collected to have non-stationary characteristics and contains a lot of background noise and noise generated by other components, so that the diagnosis result is poor. Time-frequency analysis is an effective tool for processing non-stationary signals. The deep learning method based on the time-frequency image is deeper, and the number of required training parameters is increased along with the increase of the number and the size of the hidden layers. This results in a large amount of uniformly distributed marker data often required when training an image-based deep learning network, and the existing deep learning method is based on a single time domain signal, which results in poor quality of extracted features, difficulty in training a deep learning model, and difficulty in application when the target domain marker sample is insufficient.
In most practical industrial scenarios, the training data is severely limited, especially failures tend to occur only at the end of bearing life. Moreover, as the working conditions of the rolling bearing are complex and various, the fault data distribution learned and predicted in the diagnosis model is different. The conventional deep learning method cannot well cope with the case of a small sample, and is difficult to be used for fault diagnosis of the rolling bearing.
Disclosure of Invention
The invention provides a small sample transfer learning (TL-MTCN) fault diagnosis method based on CNN feature fusion, which aims to solve the problem that the existing deep learning method cannot well process a small sample and is difficult to be used for fault diagnosis of a rolling bearing.
The technical scheme of the invention is as follows:
a small sample transfer learning fault diagnosis method based on CNN feature fusion, the method comprising:
s1: collecting vibration signals of the rolling bearing under an original working condition and a target working condition, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
s2: extracting time-frequency characteristics of the original vibration signals, and constructing a time-frequency image dataset;
s3: constructing a CNN model with feature fusion according to the original vibration signal and the time-frequency image dataset;
s4: normalizing the original vibration signal and the corresponding time-frequency image, and training a CNN model according to the normalized source domain data to obtain a CNN training model;
s5: according to the target domain sample data set, a CNN training model is adjusted to obtain a CNN fault diagnosis fine tuning model;
s6: and diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
Further, there is also provided a preferred manner, the constructing a CNN model of feature fusion according to the original vibration signal and the time-frequency image dataset, including:
extracting the characteristics of the time-frequency image data set by adopting a ConvNeXt network of a standard convolution module, wherein the input characteristic size of the ConvNeXt network is 224 multiplied by 224, and the output characteristic size is 768 multiplied by 1;
constructing a 1D-CNN network structure combined with ECA-Net, and extracting characteristics of an original vibration signal, wherein the length of the original vibration signal input by the 1D-CNN network structure is 1024, and the output characteristic size is 96 multiplied by 1;
and fusing the extracted time-frequency image features and the original vibration signal features to obtain a CNN model with fused features.
Further, a preferred mode is also provided, and the 1D-CNN network structure specifically comprises:
the 1D-CNN network consists of 4 convolution layers, wherein the number of convolution kernels in each convolution layer is 256, 128, 64 and 32 respectively, the convolution kernel of the first convolution layer is 64, the rest is 3, and the step length is 1;
after each convolution layer there is a maximum pooling layer with a pooling kernel size of 4 x 1 and a pooling movement step size of 4. And neither the convolution layer nor the pooling layer performs a filling operation; at the end of the network, two full connection layers are added, 96 and 10 neurons are respectively arranged;
the ECA-Net module is added after the last two convolution layers, and in addition, the nonlinear activation function used after each convolution layer is a scaling index linear unit SeLu, and the classification layer uses Softmax.
Further, there is provided a preferred mode, the convolution calculating method is as follows:
wherein ,for the output of the ith neuron, +.>For the input of the ith neuron, f (·) is the activation function, ω ij For input signal +.>Connection weight to jth neuron, b j For output bias.
Further, there is provided a preferred manner, wherein the normalizing the original vibration signal in the dataset and the corresponding time-frequency image further includes:
selecting 70% of samples in the source domain sample data set as a training set for pre-training, and using 30% of samples as a verification set to evaluate a source domain training result and keep optimal parameters; and when the preset iteration times are reached, storing the optimal source domain training parameters according to the performance of the verification set, and migrating the optimal source domain training parameters to the target domain.
Further, there is provided a preferred manner, the adjusting a CNN training model according to a portion of samples in the target domain sample dataset, to obtain a CNN fault diagnosis fine tuning model, including:
migrating the pre-trained samples in the source domain sample dataset to the whole TL-MTCN depth convolution network, and determining whether to delete the last full connection layer and parameters thereof according to migration tasks;
and (3) taking samples within 10% as training sets on a target domain, pre-training and adjusting a CNN training model, and taking the rest samples as verification sets.
Further, a preferred mode is also provided, and the normalization processing of the time-frequency image specifically includes:
the mean and variance of the ImageNet dataset were normalized, with the mean (0.485,0.456,0.406) and the variance (0.229,0.224,0.225).
Based on the same inventive concept, the invention also provides a small sample transfer learning fault diagnosis system based on CNN feature fusion, which comprises:
module one: the method comprises the steps of acquiring vibration signals under an original working condition and a target working condition of a rolling bearing, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
and a second module: the time-frequency characteristic is used for extracting the time-frequency characteristic of the original vibration signal, and a time-frequency image dataset is constructed;
and a third module: the CNN model is used for constructing feature fusion according to the original vibration signal and the time-frequency image dataset;
and a fourth module: the method comprises the steps of carrying out normalization processing on the original vibration signals and corresponding time-frequency images, and carrying out CNN model training according to the normalized source domain data to obtain a CNN training model;
and a fifth module: the method comprises the steps of adjusting a CNN training model according to a part of samples in the target domain sample data set to obtain a CNN fault diagnosis fine tuning model;
and a sixth module: and the method is used for diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
Based on the same inventive concept, the invention also provides a computer readable storage medium for storing a computer program, wherein the computer program executes a small sample migration learning fault diagnosis method based on CNN feature fusion.
Based on the same inventive concept, the invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a small sample migration learning fault diagnosis method based on CNN feature fusion.
The invention has the advantages that: the invention solves the problems that the existing deep learning method can not well process small samples and is difficult to be used for fault diagnosis of the rolling bearing.
The invention provides a rolling bearing fault diagnosis method for feature fusion by combining original vibration signals with time-frequency images, and a source domain fault diagnosis model is finely tuned by utilizing a small quantity of training samples in a target domain through a migration learning technology, so that high-precision intelligent fault diagnosis between different working conditions and different machine structures is realized. The method has better effect under the condition of not carrying out source domain training. The training cost of the source domain diagnostic model can be effectively reduced by initializing ConvNeXt by using the pre-training weight. The 1D-CNN designed based on the attention mechanism suppresses overfitting while maintaining a small number of trainable parameters. The method has excellent feature extraction capability under the environment of low sample number and high noise.
The invention is applied to the field of intelligent detection of rolling bearings.
Drawings
Fig. 1 is a flowchart of a small sample transfer learning fault diagnosis process based on CNN feature fusion according to an embodiment;
fig. 2 is an overall frame diagram of a TL-MTCN deep convolutional network according to the second embodiment;
FIG. 3 is a graph showing a comparison of a confusion matrix of fine tuning results of task A-D according to an eleventh embodiment;
FIG. 4 is a graph showing the visualization of the fine tuning result T-SNE of the data set E after noise addition according to the eleventh embodiment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments.
Embodiment one, this embodiment will be described with reference to fig. 1. The small sample migration learning fault diagnosis method based on CNN feature fusion of the embodiment comprises the following steps:
s1: collecting vibration signals of the rolling bearing under an original working condition and a target working condition, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
s2: extracting time-frequency characteristics of the original vibration signals, and constructing a time-frequency image dataset;
s3: constructing a CNN model with feature fusion according to the original vibration signal and the time-frequency image dataset;
s4: normalizing the original vibration acceleration signal and the corresponding time-frequency characteristic image, and performing CNN model training according to the normalized source domain data to obtain a CNN training model;
s5: according to the target domain sample data set, a CNN training model is adjusted to obtain a CNN fault diagnosis fine tuning model;
s6: and diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
The original vibration signal in this embodiment is the vibration signal under the original working condition and the target working condition of the rolling bearing.
According to the small sample migration learning fault diagnosis method based on CNN feature fusion, a migration learning technology is adopted, a small quantity of training samples in a target domain are utilized to conduct fine adjustment on a source domain fault diagnosis model, and high-precision intelligent fault diagnosis between different working conditions and different machine structures is achieved.
The CNN training model according to this embodiment continuously adjusts model parameters until the parameters converge, obtains the optimal source domain training parameters, and stores the training parameters, so as to obtain the optimal training model and ensure the accuracy of fault diagnosis.
Embodiment two, this embodiment will be described with reference to fig. 2. The present embodiment is further defined to a small sample transfer learning fault diagnosis method based on CNN feature fusion according to the first embodiment, where the constructing a CNN model with feature fusion according to the original vibration signal and the time-frequency image dataset includes:
extracting the characteristics of the time-frequency image data set by adopting a ConvNeXt network of a standard convolution module, wherein the input characteristic size of the ConvNeXt network is 224 multiplied by 224, and the output characteristic size is 768 multiplied by 1;
constructing a 1D-CNN network structure combined with ECA-Net, and extracting characteristics of an original vibration signal, wherein the length of the original vibration signal input by the 1D-CNN network structure is 1024, and the output characteristic size is 96 multiplied by 1;
and fusing the extracted time-frequency image features and the original vibration signal features to obtain a CNN model with fused features.
The overall framework of the TL-MTCN deep convolutional network according to this embodiment is shown in fig. 2, where the TL-MTCN deep convolutional network according to this embodiment is a CNN model for feature fusion.
In this embodiment, an improved 1D-CNN network structure and a ConvNeXt network structure are respectively established to perform deep feature extraction on the combined data, and then a full connection layer is used to classify the fused features to implement pattern recognition of the bearing state, specifically:
the CNN model of feature fusion in the embodiment is constructed by a ConvNeXt network which is completely dependent on a standard convolution module, features of time-frequency images are extracted through the ConvNeXt network, the input feature size of the ConvNeXt network is 224 multiplied by 224, the output feature size is 768 multiplied by 1, and the last classification layer of the ConvNeXt network is deleted; and constructing a 1D-CNN network structure combined with ECA-Net, extracting the characteristics of an original vibration signal, wherein the length of the input original vibration acceleration signal is 1024, the output characteristic size is 96 multiplied by 1, and deleting the last classification layer in the 1D-CNN network structure. And then splicing the extracted features and inputting the spliced features into a full-connection layer, wherein the activation function adopts Softmax, so that the classification of various rolling bearing faults is realized.
In the third embodiment, the present embodiment is further defined by a small sample transfer learning fault diagnosis method based on CNN feature fusion in the second embodiment, where the 1D-CNN network structure specifically includes:
the 1D-CNN network consists of 4 convolution layers, wherein the number of convolution kernels in each convolution layer is 256, 128, 64 and 32 respectively, the convolution kernel of the first convolution layer is 64, the rest is 3, and the step length is 1;
after each convolution layer, there is a maximum pooling layer with a pooling kernel size of 4 x 1 and a pooling movement step size of 4; and neither the convolution layer nor the pooling layer performs a filling operation; adding two full-connection layers at the tail end of the network, wherein the two full-connection layers are respectively provided with 96 neurons and 10 neurons;
the ECA-Net module is added after the last two convolution layers, and in addition, the nonlinear activation function used after each convolution layer is a scaling index linear unit SeLu, and the classification layer uses Softmax.
In this embodiment, in order to fully utilize the characteristics of different channels, ECA-Net modules are added after the last two convolution layers. The embodiment builds a new 1D-CNN network structure, and the feature extraction is more accurate.
In a fourth embodiment, the present embodiment is further defined by a small sample migration learning fault diagnosis method based on CNN feature fusion according to the third embodiment, where the convolution calculation method is:
wherein ,for the output of the ith neuron, +.>For the input of the ith neuron, f (·) is the activation function, ω ij For input signal +.>Connection weight to jth neuron, b j For output bias.
The present embodiment will be described with reference to the second and third embodiments.
Firstly, constructing parallel convolution layers with different dimensions by using ConvNeXt network and improved 1D-CNN network, and extracting key state characteristic information from data with two different dimensions. The first set of two-dimensional convolution layers operates on the time-frequency image to adaptively extract the useful features. The second set of one-dimensional convolution layers extracts critical information from the original vibration signal. Finally, the extracted features with two different dimensions are fused to carry out the next classification. The pre-training model with different parameter numbers exists on the basis of the pre-trained ConvNeXt network, and the Tiny model with the least parameter is selected in the embodiment.
The 1D-CNN network model is constructed by 4 convolution layers, and the number of convolution kernels in each convolution layer is 256, 128, 64 and 32 respectively. The convolution kernel of the first convolution layer has a size of 64, the rest of 3, and the step sizes are all 1. After each convolution layer there is a maximum pooling layer with a pooling kernel size of 4 x 1 and a pooling movement step size of 4. And neither the convolutional layer nor the pooling layer performs the padding operation. At the end of the network, two fully connected layers were added, 96 and 10 neurons each. The convolution calculation method is as follows:
wherein ,for the output of the ith neuron, +.>For the input of the ith neuron, f (·) is the activation function, ω ij For input signal +.>Connection weight to jth neuron, b j For output bias. The maximum pooling calculation method comprises the following steps:
wherein ,for inputting the input value of the j-th neuron of the feature plane, f max (. Cndot.) is the maximum of the function, Y i out Is the output value of the ith neuron of the output feature plane.
To fully exploit the characteristics of the different channels, ECA-Net modules are added after the last two convolutional layers. In addition, the nonlinear activation function used after each convolution layer is a scaling exponential linear unit (SeLu), and the classification layer uses Softmax. The specific structure is shown in table 1:
TABLE 1D-CNN model Structure
Finally, the feature map of the last convolution operation is merged into one in the channel dimension by merging the feature maps of two different shapes. Then, the new feature map is flattened and then input into a full-connection layer, the mapping relation between feature extraction and bearing fault states is calculated, and finally fault classification is carried out through Softmax. The calculation formula of the full connection layer is as follows:
wherein ,a weight matrix representing the j-th neuron of layer l and the i-th neuron of layer l+1,>for the corresponding bias term +.>Represents the j-th neuron of layer i. The Softmax function is a normalized exponential function, and is used for calculating the probability of the original vibration signal corresponding to each fault class, and is defined as:
wherein ,si Representing the number of samples of the current vibration signal corresponding to the ith fault class, p i Representing the probability that the current vibration signal corresponds to the ith fault class, and the sum of the probabilities that the current sample corresponds to each fault class is 1, namely
An fifth embodiment is further defined by the small sample transfer learning fault diagnosis method based on CNN feature fusion in the first embodiment, wherein the normalizing the original vibration acceleration signal and the corresponding time-frequency image further includes:
selecting 70% of samples in the source domain sample data set as a training set for pre-training, and using 30% of samples as a verification set to evaluate a source domain training result and keep optimal parameters; and when the preset iteration times are reached, storing the optimal source domain training parameters according to the performance of the verification set, and migrating the optimal source domain training parameters to the target domain.
Specifically, S4 further includes: and deleting the full connection layer parameters in the pretrained network when the network parameters pretrained based on the ImageNet are migrated to the source domain ConvNeXt deep convolutional network. Pretraining is performed on the source domain sample dataset with 70% of the data as the training set, and the source domain training results are evaluated using 30% of the samples as the validation set and the optimal parameters are retained. And when the preset iteration times are reached, storing the optimal source domain training parameters according to the performance of the verification set, and migrating the optimal source domain training parameters to the target domain.
An embodiment six, this embodiment is further defined by the small sample transfer learning fault diagnosis method based on CNN feature fusion in the embodiment five, where the adjusting a CNN training model according to a portion of samples in the target domain sample dataset, to obtain a CNN fault diagnosis fine tuning model includes:
migrating the pre-trained samples in the source domain sample dataset to the whole TL-MTCN depth convolution network, and determining whether to delete the last full connection layer and parameters thereof according to migration tasks;
and (3) taking samples within 10% as training sets on a target domain, pre-training and adjusting a CNN training model, and taking the rest samples as verification sets.
The present embodiment will be described with reference to the fifth embodiment, specifically:
and migrating the network parameters based on source domain pre-training to the whole TL-MTCN deep convolutional network, and determining whether to delete the last full connection layer and the parameters thereof according to migration tasks. And pre-training is carried out on the target domain by taking data within 10% as a training set, the rest samples are taken as a final verification set, and the verification set is not used for evaluating the training effect in the training process so as to avoid data leakage. And when the preset iteration times are reached, testing the test set sample by using the finally stored parameters to obtain a fault diagnosis result of the target domain.
Further, a TL-MTCN fault diagnostic model trained on the source domain is used to diagnose faults on the target domain. And a parameter migration method is adopted and a fault diagnosis model of the fine tuning target domain is combined. The method of parameter migration is an effective and efficient solution to overcome the tag data deficiency of the target domain.
In a seventh embodiment, the present embodiment is further defined by the small sample transfer learning fault diagnosis method based on CNN feature fusion in the first embodiment, where the normalizing process of the time-frequency image specifically includes:
the mean and variance of the ImageNet dataset were normalized, with the mean (0.485,0.456,0.406) and the variance (0.229,0.224,0.225). The original vibration signal is normalized by normalizing the training set and the validation set using the mean and variance of the training set.
An eighth embodiment is a small sample migration learning fault diagnosis system based on CNN feature fusion, where the system includes:
module one: the method comprises the steps of acquiring vibration acceleration signals under an original working condition and a target working condition of a rolling bearing, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
and a second module: the time-frequency characteristic is used for extracting the time-frequency characteristic of the original vibration signal, and a time-frequency image dataset is constructed;
and a third module: the CNN model is used for constructing feature fusion according to the original vibration signal and the time-frequency image dataset;
and a fourth module: the method comprises the steps of carrying out normalization processing on the original vibration signals and corresponding time-frequency images, and carrying out CNN model training according to normalized source domain data to obtain a CNN training model;
and a fifth module: the method comprises the steps of adjusting a CNN training model according to a part of samples in the target domain sample data set to obtain a CNN fault diagnosis fine tuning model;
and a sixth module: and the method is used for diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
The original vibration signal in this embodiment is the vibration signal under the original working condition and the target working condition of the rolling bearing.
The computer readable storage medium according to the ninth embodiment is used for storing a computer program, and the computer program executes the small sample migration learning fault diagnosis method based on CNN feature fusion according to any one of the first to seventh embodiments.
Embodiment ten, this embodiment will be described with reference to fig. 3 and 4. A computer device according to this embodiment includes a memory and a processor, the memory storing a computer program, and the processor executing a small sample migration learning failure diagnosis method based on CNN feature fusion according to any one of the first to seventh embodiments when the processor runs the computer program stored in the memory.
An eleventh embodiment is a specific example provided for the small sample transfer learning fault diagnosis method based on CNN feature fusion in the first embodiment, and is also used for explaining the second embodiment to the seventh embodiment, specifically:
firstly, respectively collecting vibration acceleration signals of a bearing under the existing working condition and the target working condition, and dividing the original vibration acceleration signals to construct a source domain sample data set and a target domain sample data set;
step two, extracting time-frequency characteristics of an original vibration signal by using synchronous compression wavelet transformation, and constructing a corresponding image data set;
thirdly, constructing a feature fusion CNN model initialized by different modes;
step four, taking an original vibration signal and a corresponding time-frequency image in a source domain sample data set as input data, normalizing the input data, inputting a fault diagnosis model for training, and storing training parameters;
and fifthly, using a few marked sample fine tuning models in the marked domain sample data set, and finally inputting the rest samples of the target domain to be tested into the network after fine tuning to obtain a bearing fault diagnosis result.
In step four of the present embodiment, the original vibration signal of the source domain sample data and the corresponding time-frequency image are used as input data, so as to implement source domain training of the model.
The data used in this embodiment are from the bearing vibration database of the american society of mechanical fault prevention technology, kesixi Chu Da, usa, respectively.
The bearing data platform of the university of Kassi storage in America consists of 1.5KW of motor, a torque sensor, a power tester and an accelerometer. The system comprises two test bearings which are respectively positioned at the motor driving end and the motor fan end, and the types of the used bearings are as follows: 6205-2RS JEM SKF. The ball, the inner ring and the outer ring are subjected to single-point damage through electric spark machining, so that the bearing has four states, namely a normal state, a ball failure state, an inner ring failure state and an outer ring failure state. The three fault conditions, in turn, contain signals of three levels of fault, namely, fault diameters of 0.007, 0.014 and 0.021 inches, respectively, depending on the fault diameter. Vibration data was collected using accelerometers placed near the bearings, and signals were acquired at a sampling frequency of 12 kHz. The data used in the experiments of this embodiment are drive end bearing data.
The american society of mechanical fault prevention technology data set includes three sets of experimental bearing vibration data, namely, baseline bearing data, outer ring fault data for various loads, and inner ring fault data for various loads. Three normal data were collected at a sampling frequency of 97656Hz and a 270lbs load; 7 outer ring fault data were collected at 48828Hz sampling frequency at 25, 50, 100, 150, 200, 250 and 300 lbs. loads, respectively, and 7 inner ring fault data were collected at 48828Hz sampling frequency at 0, 50, 100, 150, 200, 250 and 300 lbs. loads, respectively. The vibration data under normal conditions was downsampled to 48828Hz to match other fault conditions. Since the sampling rate is higher relative to CWRU, less fault information is contained for the same sample length, resulting in increased diagnostic difficulty.
Table 2 bearing experimental data set
According to the problem of bearing fault diagnosis of the cross working condition and the cross equipment, migration tasks such as A, B, A, C, A, D, B, C, B, D, C, D, A, E, B, E, C, E, D, E and the like are adopted to verify the effectiveness of the method, wherein A, B represent that the knowledge of a source domain data set A is migrated to a target domain data set B. In the experiment, the ratio of the training set of the source domain to the testing set is set to be 70:30, the training set of the target domain is not more than 10%, and the rest samples are used as final verification. All comparative experimental results in the invention are the average diagnostic accuracy of 10 repeated experiments. The training set, test set, and validation set randomly extracted from the dataset are balanced for different categories. To ensure fairness of the experiment, all training was performed under the same set of randomly selected samples.
The normalization process of the original vibration signal is as follows:
wherein ,for the normalized training data set, +.>For normalized test data set, x af Representing training data set, x ae Representing a test dataset, sigma f For standard deviation of training data set, A is training set sample number, +.>Is the mean of the training dataset.
The optimizer selects AdamW because its learning rate can be adaptively optimized. The neural network weight can be continuously and iteratively updated according to the training data, so that local optimum can be avoided, and the learning rate of each parameter can be dynamically adjusted. The learning rate of AdamW algorithm was set to 0.0005 and the weight decay was set to 0.05. The number of iterations was chosen to be 50 and the batch size was 32. And when the preset iteration times are reached, storing the optimal source domain training parameters according to the performance of the verification set, and migrating the optimal source domain training parameters to the target domain.
In the fifth step, the model is initialized by using the source domain training parameters, so that fine tuning training of a small amount of sample data on the model is realized in the target domain.
Firstly, experiments are carried out under the cross working condition, and fine adjustment is carried out on the model under the target working condition so as to execute diagnostic tasks under different working conditions. Specifically, a total of 20 samples over the target domain are randomly selected to fine tune the entire pre-training network, with the remaining samples used for final verification. The current detection methods of several relative main flows are selected for experimental comparison with the method, namely ResNet-50 without freezing parameters and VGG-16 networks with freezing three shallow blocks, the classification task uses the average accuracy of ten repeated experiments as an evaluation index, and the experimental results of migration tasks under different working conditions are shown in table 3.
TABLE 3 experimental results for different migration tasks across operating conditions
As can be seen from the table 3, the average accuracy of the method identification is more than 99.11%, so that knowledge of different working conditions of the rolling bearing can be fully explained to perform mutual migration, that is, mutual diagnosis of different working conditions of the rolling bearing can be realized through knowledge migration, and the method has a good effect. Wherein the confusion matrix for the a-D task is shown in figure 3.
In order to further verify the effectiveness of the model, experimental comparison is performed on the cross-equipment transfer learning task of the rolling bearing. A series of comparative experiments were performed on the proposed method with other models. In this section, the data set a in the CWRU is selected as the source domain, the MFPT data set E is selected as the target domain, and the data preprocessing mode and the super parameter selection are the same as those in the above experiment. But the number of bearing categories is different due to the CWRU dataset and MFPT. Therefore, no migration of the classification layer parameters is performed during the trimming process. Wherein the fine-tuned sample ratio of the target domain is set to 2.5%, 5%, 7.5% and 10% in this order. The experimental results of the different migration tasks are shown in table 4.
Table 4 comparison of diagnostic methods accuracy
As can be seen from table 4, even if only 2.5% of the target samples are selected for fine tuning, the diagnostic accuracy of the method on the migration path is above 98.21%, which indicates that our model has a nominal migration across devices. The mobility accuracy is continuously improved along with the increase of the proportion of the fine-tuning sample, and when the proportion reaches 5%, the diagnosis accuracy of each mobility path reaches 99.01%. When the ratio reaches 10%, the accuracy reaches 99.49% more, but this also brings about greater calculation cost. Therefore, 5% is selected as our fine tuning ratio in the subsequent experiments, which can use lower calculation cost while ensuring high accuracy.
Next, in order to explore having good migration performance between different devices. Pretraining was performed based on different source domains (A, B, C and D) and fine-tuned by 5% of small samples. The training results are shown in Table 5, and the classification properties thereof were 99.01% (A.fwdarw.E), 98.37% (B.fwdarw.E), 98.51% (C.fwdarw.E) and 98.63% (D.fwdarw.E), respectively. From the results, it can be seen that mobility accuracy is very close to the performance of training models with sufficient labeling samples. Therefore, the method can better solve the problem of insufficient target samples on the premise of maintaining the diagnosis accuracy.
Table 5 comparison results of different model migration methods
In addition, in consideration of the fact that in the engineering environment, vibration and mutual friction among parts inevitably generate noise, bearing vibration signals acquired by the sensor are extremely easy to pollute by the noise, and fault information in the vibration signals is covered. To verify the anti-noise performance of the proposed method, noise in the actual engineering environment is simulated by adding gaussian white noise of different intensities to the test samples. Wherein the data samples for training do not add noise. The signal-to-noise ratio (SNR) is the ratio of signal power to noise power, and is typically measured in decibels (dB) as follows.
in the formula ,Ps And P n Respectively representing the power of the signal and the power of the noise. The experiment designs five types of noise adding signals with the signal to noise ratios of-4 dB, -2dB, 0dB, 2dB and 4 dB. The greater the signal-to-noise ratio, the less the signal is disturbed by noise. Signals with low SNR are typically more complex, so it is very important to improve the robustness of the model. Table 6 shows the results of comparisons of different methods under different signal-to-noise conditions for migration tasks A-E.
TABLE 6 comparison results under different SNR conditions
The TL-MTCN model has no significant degradation in accuracy at signal to noise ratios of 4dB and 2 dB. For noise data with signal-to-noise ratios of 0dB and-2 dB, the classification accuracy of the model is slightly reduced, but the average accuracy of more than 96.73% can still be maintained. For noisy data with a signal to noise ratio of-4 dB, the TL-MTCN model accuracy is affected to a certain extent. The ResNet-50 model, which performs well under noiseless conditions, exhibits a significant performance degradation with increasing noise intensity.
T-SNE is a data dimension reduction technology, and the output layer characteristics of each model are visualized to reveal the characteristic classification capability of different models. FIG. 4 shows a T-SNE depth profile at a signal-to-noise ratio of-4 dB across a dataset migration task, where D2 and D1 represent the reduced-dimension principal components. Experimental comparison results further demonstrate that TL-MTCN has better intra-class compactness and inter-class separability and has stronger fault diagnosis migration capability than other models.
While the preferred embodiments of the present disclosure have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the disclosure.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit or scope of the disclosure. Thus, the present disclosure is intended to include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
It will be appreciated by those skilled in the art that embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical aspects of the present disclosure and not for limiting the scope thereof, and although the present disclosure has been described in detail with reference to the above embodiments, it should be understood by those of ordinary skill in the art that: various alterations, modifications, and equivalents may be suggested to the specific embodiments of the invention, which would occur to persons skilled in the art upon reading the disclosure, are intended to be within the scope of the appended claims.

Claims (10)

1. The small sample transfer learning fault diagnosis method based on CNN feature fusion is characterized by comprising the following steps:
s1: collecting vibration signals of the rolling bearing under an original working condition and a target working condition, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
s2: extracting time-frequency characteristics of the original vibration signals, and constructing a time-frequency image dataset;
s3: constructing a CNN model with feature fusion according to the original vibration signal and the time-frequency image dataset;
s4: normalizing the original vibration signal and the corresponding time-frequency image, and training a CNN model according to the normalized source domain data to obtain a CNN training model;
s5: according to the target domain sample data set, a CNN training model is adjusted to obtain a CNN fault diagnosis fine tuning model;
s6: and diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
2. The CNN feature fusion-based small sample transfer learning fault diagnosis method according to claim 1, wherein the constructing a feature fusion CNN model according to the original vibration signal and the time-frequency image dataset comprises:
extracting the characteristics of the time-frequency image data set by adopting a ConvNeXt network of a standard convolution module, wherein the input characteristic size of the ConvNeXt network is 224 multiplied by 224, and the output characteristic size is 768 multiplied by 1;
constructing a 1D-CNN network structure combined with ECA-Net, and extracting characteristics of an original vibration signal, wherein the length of the original vibration signal input by the 1D-CNN network structure is 1024, and the output characteristic size is 96 multiplied by 1;
and fusing the extracted time-frequency image features and the original vibration signal features to obtain a CNN model with fused features.
3. The small sample transfer learning fault diagnosis method based on the CNN feature fusion according to claim 2, wherein the 1D-CNN network structure is specifically:
the 1D-CNN network consists of 4 convolution layers, wherein the number of convolution kernels in each convolution layer is 256, 128, 64 and 32 respectively, the convolution kernel of the first convolution layer is 64, the rest is 3, and the step length is 1;
after each convolution layer, there is a maximum pooling layer with a pooling kernel size of 4 x 1 and a pooling movement step size of 4; and neither the convolution layer nor the pooling layer performs a filling operation; at the end of the network, two full connection layers are added, 96 and 10 neurons are respectively arranged;
the ECA-Net module is added after the last two convolution layers, and in addition, the nonlinear activation function used after each convolution layer is a scaling index linear unit SeLu, and the classification layer uses Softmax.
4. The small sample transfer learning fault diagnosis method based on CNN feature fusion of claim 3, wherein the convolution calculation method is as follows:
wherein ,for the output of the ith neuron, +.>For the input of the ith neuron, f (·) is the activation function, ω ij For input signal +.>Connection weight to jth neuron, b j For output bias.
5. The CNN feature fusion-based small sample transfer learning fault diagnosis method according to claim 1, wherein the normalizing process is performed on the original vibration acceleration signal and the corresponding time-frequency image in the data set, and further comprises:
selecting 70% of samples in the source domain sample data set as a training set for pre-training, and using 30% of samples as a verification set to evaluate a source domain training result and keep optimal parameters; and when the preset iteration times are reached, storing the optimal source domain training parameters according to the performance of the verification set, and migrating the optimal source domain training parameters to the target domain.
6. The method for small sample transfer learning fault diagnosis based on CNN feature fusion according to claim 5, wherein the step of adjusting a CNN training model according to a portion of samples in the target domain sample dataset to obtain a CNN fault diagnosis fine tuning model comprises the steps of:
migrating the pre-trained samples in the source domain sample dataset to the whole TL-MTCN depth convolution network, and determining whether to delete the last full connection layer and parameters thereof according to migration tasks;
and (3) taking samples within 10% as training sets on a target domain, pre-training and adjusting a CNN training model, and taking the rest samples as verification sets.
7. The small sample transfer learning fault diagnosis method based on CNN feature fusion of claim 1, wherein the normalization processing of the time-frequency image is specifically:
the mean and variance of the ImageNet dataset were normalized, with the mean (0.485,0.456,0.406) and the variance (0.229,0.224,0.225).
8. A CNN feature fusion-based small sample transfer learning fault diagnosis system, the system comprising:
module one: the method comprises the steps of acquiring vibration signals under an original working condition and a target working condition of a rolling bearing, dividing the original vibration signals, and constructing a source domain sample data set and a target domain sample data set according to the divided signals;
and a second module: the time-frequency characteristic is used for extracting the time-frequency characteristic of the original vibration signal, and a time-frequency image dataset is constructed;
and a third module: the CNN model is used for constructing feature fusion according to the original vibration signal and the time-frequency image dataset;
and a fourth module: the method comprises the steps of carrying out normalization processing on the original vibration signals and corresponding time-frequency images, and carrying out CNN model training according to the normalized source domain data to obtain a CNN training model;
and a fifth module: the method comprises the steps of adjusting a CNN training model according to a part of samples in the target domain sample data set to obtain a CNN fault diagnosis fine tuning model;
and a sixth module: and the method is used for diagnosing the residual sample of the target domain by adopting the CNN fault diagnosis fine tuning model to obtain a bearing fault diagnosis result.
9. A computer readable storage medium for storing a computer program for executing a CNN feature fusion-based small sample transfer learning fault diagnosis method according to any one of claims 1 to 7.
10. A computer device, characterized by: comprising a memory and a processor, the memory having stored therein a computer program, which when executed by the processor performs a small sample migration learning fault diagnosis method based on CNN feature fusion according to any one of claims 1-7.
CN202310596134.3A 2023-05-25 2023-05-25 Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion Pending CN116702076A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310596134.3A CN116702076A (en) 2023-05-25 2023-05-25 Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310596134.3A CN116702076A (en) 2023-05-25 2023-05-25 Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion

Publications (1)

Publication Number Publication Date
CN116702076A true CN116702076A (en) 2023-09-05

Family

ID=87831934

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310596134.3A Pending CN116702076A (en) 2023-05-25 2023-05-25 Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion

Country Status (1)

Country Link
CN (1) CN116702076A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116894190A (en) * 2023-09-11 2023-10-17 江西南昌济生制药有限责任公司 Bearing fault diagnosis method, device, electronic equipment and storage medium
CN117404765A (en) * 2023-12-14 2024-01-16 山东省人工智能研究院 Air conditioner system fan fault diagnosis method and system under weak supervision condition and air conditioner

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116894190A (en) * 2023-09-11 2023-10-17 江西南昌济生制药有限责任公司 Bearing fault diagnosis method, device, electronic equipment and storage medium
CN116894190B (en) * 2023-09-11 2023-11-28 江西南昌济生制药有限责任公司 Bearing fault diagnosis method, device, electronic equipment and storage medium
CN117404765A (en) * 2023-12-14 2024-01-16 山东省人工智能研究院 Air conditioner system fan fault diagnosis method and system under weak supervision condition and air conditioner
CN117404765B (en) * 2023-12-14 2024-03-22 山东省人工智能研究院 Air conditioner system fan fault diagnosis method and system under weak supervision condition and air conditioner

Similar Documents

Publication Publication Date Title
CN109947086B (en) Mechanical fault migration diagnosis method and system based on counterstudy
Hoang et al. Rolling element bearing fault diagnosis using convolutional neural network and vibration image
Cheng et al. A convolutional neural network based degradation indicator construction and health prognosis using bidirectional long short-term memory network for rolling bearings
Ravikumar et al. Gearbox fault diagnosis based on Multi-Scale deep residual learning and stacked LSTM model
CN112084974B (en) Multi-label rolling bearing fault diagnosis method based on meta-learning
CN112763214B (en) Rolling bearing fault diagnosis method based on multi-label zero-sample learning
CN116702076A (en) Small sample migration learning fault diagnosis method, system, computer and storage medium based on CNN feature fusion
Li et al. Unsupervised rotating machinery fault diagnosis method based on integrated SAE–DBN and a binary processor
CN110657984B (en) Planetary gearbox fault diagnosis method based on reinforced capsule network
CN113111820B (en) Rotary part fault diagnosis method and device based on improved CNN and relation module
CN111562108A (en) Rolling bearing intelligent fault diagnosis method based on CNN and FCMC
CN113673346B (en) Motor vibration data processing and state identification method based on multiscale SE-Resnet
CN114970605A (en) Multi-mode feature fusion neural network refrigeration equipment fault diagnosis method
CN112560948B (en) Fundus image classification method and imaging method under data deviation
CN109946080A (en) A kind of mechanical equipment health status recognition methods based on embedded recirculating network
CN114170184A (en) Product image anomaly detection method and device based on embedded feature vector
CN116502175A (en) Method, device and storage medium for diagnosing fault of graph neural network
CN115290326A (en) Rolling bearing fault intelligent diagnosis method
CN115859077A (en) Multi-feature fusion motor small sample fault diagnosis method under variable working conditions
CN115170874A (en) Self-distillation implementation method based on decoupling distillation loss
Wang et al. A novel rolling bearing fault diagnosis method based on adaptive denoising convolutional neural network under noise background
CN115239034B (en) Method and system for predicting early defects of wind driven generator blade
CN116227586A (en) Meta learning fault diagnosis method and device based on depth residual error shrinkage prototype network
Techane et al. Rotating machinery prognostics and application of machine learning algorithms: Use of deep learning with similarity index measure for health status prediction
CN116026569A (en) Mechanical equipment unsupervised migration intelligent fault diagnosis method for releasing source data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination