CN111476757A - Coronary artery patch data detection method, system, storage medium and terminal - Google Patents

Coronary artery patch data detection method, system, storage medium and terminal Download PDF

Info

Publication number
CN111476757A
CN111476757A CN202010160279.5A CN202010160279A CN111476757A CN 111476757 A CN111476757 A CN 111476757A CN 202010160279 A CN202010160279 A CN 202010160279A CN 111476757 A CN111476757 A CN 111476757A
Authority
CN
China
Prior art keywords
network
sample
layer
dimensional
coronary artery
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010160279.5A
Other languages
Chinese (zh)
Other versions
CN111476757B (en
Inventor
宋小磊
吴斌
赵凤军
范思琪
陈一兵
朱元强
贺小伟
侯榆青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwestern University
Original Assignee
Northwestern University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwestern University filed Critical Northwestern University
Priority to CN202010160279.5A priority Critical patent/CN111476757B/en
Publication of CN111476757A publication Critical patent/CN111476757A/en
Application granted granted Critical
Publication of CN111476757B publication Critical patent/CN111476757B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention belongs to the technical field of medical image processing and computer-aided diagnosis, and discloses a coronary artery plaque data detection method, a system, a storage medium and a terminal, wherein a three-dimensional image block is extracted along a coronary artery central line point on a three-dimensional medical image as a sample, and a sample data set is divided; taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network; sending the training data set into a network, and calculating a loss function to train a network model; and predicting the trained network model on the test set to complete the coronary plaque detection task. The invention combines non-labeled data to perform semi-supervised learning, completes the plaque detection only by depending on a small amount of labeled data, reduces the difficulty of labeled data and improves the detection precision. The invention realizes the detection of the coronary plaque, and has the characteristics of no need of pre-dividing the blood vessel, high accuracy and dependence on a small amount of labeled data.

Description

Coronary artery patch data detection method, system, storage medium and terminal
Technical Field
The invention belongs to the technical field of medical image processing and computer-aided diagnosis, and particularly relates to a coronary plaque data detection method, a system, a storage medium and a terminal.
Background
At present, coronary artery disease is one of the most common cardiovascular diseases, and the generation of coronary atherosclerotic plaques is mainly responsible for coronary artery disease. Early detection and identification of coronary plaques is of great importance for the prevention and diagnosis of coronary artery disease. Imaging techniques such as intravascular ultrasound and optical coherence tomography provide detailed visualization of the intraluminal and plaque morphology, and allow reliable quantitative analysis of atherosclerotic burden and its components, with good discrimination of plaque properties, but these techniques are invasive and costly, time consuming and difficult to operate, present considerable patient risk, and can only be performed in the proximal vessels, are not suitable for detecting plaque progression throughout the coronary tree in a short period of time, and have certain limitations for clinical deployment. With the development of computer tomography, coronary artery CT angiography has become a well-known method for diagnosing and excluding suspected coronary heart disease patients due to its advantages of being non-invasive, three-dimensional, high resolution, etc. Currently clinically, the task of plaque detection using cardiac CTA images is usually done based on visual assessment, or semi-automatically segmenting the lumen and artery walls before defining the presence of vascular plaque. However, visual assessment usually has large inter-observer differences, and semi-automatic segmentation of blood vessels is not only time-consuming and labor-consuming, but also inaccurate in segmentation seriously affects plaque detection results. In view of the importance of plaque detection for early prevention and diagnostic intervention of coronary artery disease, a variety of computer-assisted coronary plaque detection and quantification methods are proposed. The plaque detection method based on the threshold value is simple in principle and easy to operate, but has the problem of attenuation intensity overlapping, the intensity of the blood vessel plaque is possibly similar to that of surrounding tissues, and the intensity values of the same type of tissues between different CTA images are possibly greatly different due to different equipment and contrast agent intensities, so that the blood vessel plaque cannot be accurately segmented from the CTA images by the threshold value method; the performance of the method depending on the blood vessel segmentation depends on the accurate segmentation of the coronary artery, and the current coronary artery segmentation method is still not accurate enough at the far end of the blood vessel and is easily influenced by serious calcification; in recent years, automatic or semi-automatic coronary plaque detection is increasingly researched by using a machine learning method, the method needs manual feature design to represent images, and the design of distinguishing features is time-consuming and labor-consuming; the deep learning method is also applied to the plaque detection task, but the training model needs a large amount of labeled data, the manual labeling of coronary plaque by experts is expensive, the error among observers is easy to exist, and a large amount of finely labeled data is still difficult to obtain.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) the existing threshold value method has the problems of intensity overlapping and inaccurate detection.
(2) The existing method depending on coronary artery segmentation is difficult to generate a fine coronary artery tree due to the influences of calcification, excessively thin far end of a blood vessel and the like.
(3) The manual design of features by traditional machine learning methods is time consuming and labor intensive.
(4) Existing depth model training requires a large amount of annotation data.
The difficulty in solving the above problems and defects is: in view of the diversity of data features, it is difficult to manually design discriminative features in the application of the conventional machine learning method; the deep learning method has certain difficulty in obtaining the marking data, and the manual marking of experts is time-consuming and labor-consuming, so that the error between observers is easy to occur.
The significance of solving the problems and the defects is as follows:
(1) the coronary artery does not need to be segmented, and the error caused by inaccurate segmentation of the coronary artery is avoided.
(2) The patch detection is carried out by using the convolutional neural network, the complexity of manually designing features is avoided, effective features are automatically learned, and the detection precision is improved.
(3) Training is performed by combining a large amount of label-free data, so that the precision of a classification task is improved, and the label-free data is easy to obtain.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a coronary plaque data detection method, a system, a storage medium and a terminal.
The invention is realized in this way, a coronary artery patch data detection method, the coronary artery patch data detection method extracts a three-dimensional image block as a sample along a central line point of a coronary artery on a three-dimensional medical image, and divides a sample data set; taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network; sending the training data set into a network, and calculating a loss function to train a network model; and predicting the trained network model on the test set to complete the coronary plaque detection task.
Further, the coronary plaque data detection method comprises the following steps:
firstly, resampling an original three-dimensional medical image to ensure that the resolution of each datum is the same; manually marking or automatically generating a coronary artery central line for the data after resampling; taking a central line point of the coronary artery as a center, taking a tangent line of the central line as a central axis, and extracting a three-dimensional image block as a sample; extracting three-dimensional image blocks of all medical images to form a sample data set; dividing a training set, a verification set and a test set; carrying out random angle rotation and mirror image on the labeled samples in the training data set, and amplifying sample information; the coronary artery central line point randomly moves a certain amplitude in any direction vertical to the blood vessel direction of the coronary artery central line point, and training data of different blood vessel central lines are added;
step two, taking the coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output, and constructing a three-dimensional step convolution neural network, wherein the network structure mainly comprises an encoder network and a decoder network; constructing an encoder network, and learning the encoding information of the blood vessel sample; adding noise to the input and each hidden layer as a noise channel, and learning robust reconstruction representation; taking a channel without any noise as a clean channel, and training reconstruction information of each layer of the network; constructing a decoder network, and reconstructing the coding characteristics into an original image; adding jump connection, and reconstructing the decoder convolution layer output;
step three, the training data set is sent into the constructed three-dimensional step convolution neural network; calculating the classified cross entropy loss between the input sample label and the noise channel output of the encoder network as a supervision loss function; calculating the weighted sum of reconstruction errors between reconstruction outputs of each convolution layer of the decoder and the output characteristic diagram of the convolution layer corresponding to the clean channel of the encoder as an unsupervised loss function; reversely propagating the error layer by layer to an input layer by using a back propagation algorithm, and updating network parameters by a gradient descent method until the network converges; sending the verification data set into a training model, evaluating the performance of the model, and selecting model parameters with the best effect;
step four, the test data set is sent into a trained model, and a clean channel without noise of an encoder is used for predicting whether the three-dimensional blood vessel image contains plaques or not; voting the test result according to the blood vessel section, acquiring a blood vessel section grade label, and evaluating the network performance.
Further, the first step is to extract a three-dimensional image block as a sample along a coronary centerline point on the three-dimensional medical image, and dividing the sample data set includes:
(1) resampling the original three-dimensional medical image to ensure that the resolution of each datum is the same;
(2) manually marking or automatically generating a coronary artery central line for the data after resampling;
(3) taking a central line point of the coronary artery as a center, taking a tangent line of the central line as a central axis, and extracting a three-dimensional image block as a sample;
(4) extracting three-dimensional image blocks of all medical image data along the central line of the coronary artery to form a sample data set;
(5) in the sample data set, a marked data set is randomly divided into a training set, a verification set and test set data, and unmarked data is added into the training set;
(6) carrying out random angle rotation and mirror image on the training data set, and amplifying sample information;
(7) the coronary artery central line point of the training data set sample randomly moves a certain amplitude to any direction vertical to the blood vessel direction of the coronary artery central line point, training data for central lines of different blood vessels are added, and errors caused by inaccurate central line extraction are reduced.
Further, the second step takes the coronary artery three-dimensional image block sample as input, and whether the image contains the plaque as output, and constructs a three-dimensional step convolution neural network, and the method is carried out according to the following steps:
(1) taking a coronary artery three-dimensional image block sample as input, and whether the image contains a plaque or not as output, constructing a three-dimensional step convolution neural network, wherein the network structure mainly comprises an encoder network and a decoder network;
(2) constructing an encoder network, learning encoding information of a blood vessel sample, wherein the encoder network comprises three-dimensional convolutional layers and two full-connection layers, each convolutional layer learns nonlinear characteristics by using a Re L u activation function, then performing characteristic dimension reduction on a three-dimensional maximum pooling layer, the first full-connection layer is used as a characteristic encoding layer for reconstructing the sample by a decoder network, the second full-connection layer is a softmax output layer, and outputting blood vessel sample category information;
(3) on the basis of the structure of an encoder, adding isotropic Gaussian noise to an input layer and each hidden layer to serve as a noise channel of a network, enabling the encoder to learn robust reconstruction representation, and simultaneously taking a channel without adding any noise as a clean channel of the network to train reconstruction information of each layer of the network;
(4) constructing a decoder network, reconstructing the coding characteristics into an original image, wherein the decoder network consists of a full-connection layer and three-dimensional convolutional layers, the characteristic dimension is changed into the size corresponding to the corresponding layer of the encoder by using up-sampling before each convolutional layer, and the output of the last convolutional layer is a reconstructed image of the input sample of the encoder;
(5) the method comprises the following steps that jump connection is added between network layers corresponding to an encoder and a decoder, the output characteristic diagram of a previous layer of a decoder network and the output characteristic diagram of a noise channel corresponding layer of the encoder are reconstructed through a denoising function, the input of a current convolutional layer is calculated, and the reconstruction function is as follows:
Figure BDA0002405526260000051
wherein ,
Figure BDA0002405526260000052
for the output of the l +1 th layer of the decoder,
Figure BDA0002405526260000053
for the l-th layer output of the encoder noise channel,
Figure BDA0002405526260000054
for the decoder l layer output, g (,) is the reconstruction function.
Further, the third step is to send the training data set to the network, calculate the loss function to train the network model, and perform the following steps:
(1) sending the training data set into the constructed three-dimensional step convolution neural network;
(2) calculating the classified cross entropy loss between the input sample label and the noise channel output of the encoder network as a supervision loss function, evaluating the consistency degree of the network prediction result and the real label, and determining the probability of the network prediction result and the real label when the input x is giveniIn the case of i ∈ (1...., N), the supervised loss function is:
Figure BDA0002405526260000055
wherein ,
Figure BDA0002405526260000056
as output of the noise channel, yiInputting a corresponding label, wherein N is the number of samples;
(3) reconstructing the unmarked data by a decoder, and calculating the reconstruction error weighted sum of the reconstruction output of each convolution layer of the decoder and the convolution layer characteristic diagram corresponding to the clean channel of the encoder as the unsupervised lossLoss function, evaluating learning ability of network characterization input sample, unsupervised reconstruction loss function CdThe calculation formula is as follows:
Figure BDA0002405526260000057
wherein L is the number of network layers, z(l)Is the feed-forward output of the l-th layer of the encoder clean channel,
Figure BDA0002405526260000058
for de-noised output of the first layer of the decoder, lambdalLoss weight for layer l reconstruction error;
(4) the total loss function is weighted sum of supervision loss and unsupervised loss, errors are reversely propagated to the input layer by using a back propagation algorithm, and network parameters are updated by a gradient descent method until the network converges to local optimum;
(5) and sending the verification data set into a training model, evaluating the performance of the model, and selecting the model parameters with the best effect as the model to be finally used.
Further, the network model trained in the fourth step predicts on a test set, and the task of detecting coronary plaque is completed by the method comprising the following steps:
(1) sending the test data set into a trained model, and predicting whether the three-dimensional blood vessel image contains plaque or not by a clean channel without noise of an encoder;
(2) voting the test result according to the blood vessel section, acquiring a blood vessel section grade label, and evaluating the network performance.
It is another object of the present invention to provide a program storage medium for receiving user input, the stored computer program causing an electronic device to perform the steps comprising: extracting a three-dimensional image block along a coronary artery central line point on a three-dimensional medical image as a sample, and dividing a sample data set; taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network; sending the training data set into a network, and calculating a loss function to train a network model; and predicting the trained network model on the test set to complete the coronary plaque detection task.
Another object of the present invention is to provide a coronary plaque data detecting system for implementing the coronary plaque data detecting method, the coronary plaque data detecting system comprising:
the sample data set dividing module is used for extracting a three-dimensional image block along a coronary artery central line point on the three-dimensional medical image as a sample and dividing a sample data set;
the three-dimensional step convolution neural network construction module is used for constructing a three-dimensional step convolution neural network by taking the coronary artery three-dimensional image block sample as input and taking whether the image contains a plaque as output;
the network model training module is used for sending the training data set into a network and calculating a loss function to train a network model;
and the network model prediction module is used for predicting the trained network model on the test set to complete the coronary plaque detection task.
The invention also aims to provide a terminal, and the terminal is provided with the coronary artery plaque data detection system.
The invention also aims to provide application of the coronary plaque data detection method in an image detection system and a computed tomography system.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention solves the problems that the target and the background are difficult to distinguish in the existing coronary plaque detection, the detection precision is low, and a large amount of labeled data is needed for training. The method depends on the central line of the coronary artery to detect the plaque, does not need to pre-divide the coronary artery, and avoids the error caused by dividing the coronary artery; by using the structure of the three-dimensional step convolution neural network, three-dimensional image block representation patch information is extracted, the global information and the local information of the patch are effectively learned, and the complexity of manually designing features in the traditional method is avoided; by using a semi-supervised learning strategy, learning unmarked data helps a supervised network to learn more information, so that the detection precision is improved, and the problem of insufficient marked data is solved; and data enhancement is carried out on the data set, so that the generalization capability of the model is effectively improved. The invention realizes the detection of the coronary plaque, and has the characteristics of no need of pre-dividing the blood vessel, high accuracy and dependence on a small amount of labeled data.
Drawings
Fig. 1 is a flowchart of a coronary plaque data detection method according to an embodiment of the present invention.
FIG. 2 is a schematic structural diagram of a coronary plaque data detection system according to an embodiment of the present invention;
in the figure: 1. a sample data set dividing module; 2. a three-dimensional step convolution neural network construction module; 3. a network model training module; 4. and a network model prediction module.
Fig. 3 is a schematic diagram of the overall structure of the three-dimensional ladder convolution neural network according to the embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a three-dimensional ladder convolutional neural network encoder network provided in an embodiment of the present invention.
Fig. 5 is a schematic diagram of a three-dimensional ladder convolutional neural network decoder network structure according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a coronary plaque data detection method, a system, a storage medium, and a terminal, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the coronary artery patch data detection method provided by the present invention includes:
s101: extracting a three-dimensional image block along a coronary artery central line point on a three-dimensional medical image as a sample, and dividing a sample data set;
s102: taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network;
s103: sending the training data set into a network, and calculating a loss function to train a network model;
s104: and predicting the trained network model on the test set to complete the coronary plaque detection task.
As shown in fig. 2, the coronary plaque data detection system provided by the present invention includes:
the sample data set dividing module 1 is used for extracting a three-dimensional image block along a coronary artery central line point on a three-dimensional medical image as a sample and dividing a sample data set.
And the three-dimensional step convolution neural network construction module 2 is used for constructing the three-dimensional step convolution neural network by taking the coronary artery three-dimensional image block sample as input and taking whether the image contains the plaque as output.
And the network model training module 3 is used for sending the training data set into a network and calculating a loss function to train the network model.
And the network model prediction module 4 is used for predicting the trained network model on the test set to complete the coronary plaque detection task.
The technical solution of the present invention is further described below with reference to the accompanying drawings.
The coronary artery plaque data detection method provided by the embodiment of the invention specifically comprises the following steps:
(1) on a three-dimensional medical image, extracting a three-dimensional image block along a coronary artery central line point as a sample, and dividing a sample data set, wherein the specific process comprises the following steps:
(1a) the data used in the example are heart CTA images, 18 heart CTA images are selected to manually mark coronary plaque for supervised learning, and 20 unlabeled heart CTA images are selected for semi-supervised learning;
(1b) the size of the cardiac CTA image is 512 × 521 × C, with a resolution between 0.25 and 0.5 the resampled resolution is [0.3,0.3,0.3] for all images;
(1c) manually marking the resampled data to generate a coronary artery central line;
(1d) dividing the coronary artery central line into 17 sections according to medical standard;
(1e) taking a central line point of the coronary artery as a center, taking a tangent line of the central line as a central axis, and extracting a three-dimensional image block with the size of 24 × 24 × 24 as a sample;
(1f) extracting three-dimensional image blocks of all heart CTA data along the central line of coronary artery to form a sample data set;
(1g) randomly dividing the marked data set into a training set, a verification set and a test set according to the ratio of 6:2:2, and adding the rest unmarked data into the training data set;
(1h) carrying out random angle rotation and mirror image on the training data set, and amplifying sample information;
(1i) the coronary artery central line point of the training data set sample randomly moves a certain amplitude (<3 voxels) towards any direction vertical to the blood vessel direction of the coronary artery central line point, training data for different blood vessel central lines are added, and errors caused by inaccurate central line extraction are reduced.
(2) Taking the coronary artery three-dimensional image block sample as input, and whether the image contains a plaque as output, constructing a three-dimensional step convolution neural network, and referring to fig. 3, the specific process is as follows:
(2a) taking a coronary artery three-dimensional image block sample as input, and whether the image contains a plaque as output, constructing a three-dimensional step convolution neural network, wherein the input size of the sample is 24 × 24 × 24, the output dimension is 2, and the network structure mainly comprises an encoder network and a decoder network;
(2b) constructing an encoder network, learning encoding information of a blood vessel sample, wherein the encoder network comprises three-dimensional convolutional layers and two fully-connected layers, the convolutional kernel size of each convolutional layer is [3,3,3], the step length is 1, the number of the convolutional kernels is 32, 64 and 128 respectively, each convolutional layer learns nonlinear characteristics by using an Re L u activation function, then, the three-dimensional maximum pooling of 2 × 2 × 2 is adopted for characteristic dimension reduction, the step length of the pooling layer is 2, the first fully-connected layer is used as a characteristic encoding layer for reconstructing a sample by a decoder network, the number of channels is 1024, the second fully-connected layer is a softmax output layer, and whether the blood vessel sample contains a plaque or not is predicted;
(2c) adding obedience distribution N (0, 0.4) to the input and hidden layers on the basis of the encoder structure2) As a net of Gaussian noiseA noise channel of the network enables an encoder to learn robust reconstruction representation, and meanwhile, the channel without any noise is used as a clean channel of the network and used for training reconstruction information of each layer of the network;
(2d) constructing a decoder network, reconstructing the coding characteristics into an original image, wherein the decoder network is composed of a full-connection layer and three-dimensional convolutional layers, the sizes of convolutional kernels of all convolutional layers are [3,3,3], the step length is 1, the number of the convolutional kernels is 64,32,1 respectively, linear interpolation upsampling is used before each convolutional layer to change the characteristic dimension into the size corresponding to the corresponding layer of the encoder, and the output of the last convolutional layer is a reconstructed image of an input sample of the encoder;
(2f) the method comprises the following steps that jump connection is added between network layers corresponding to an encoder and a decoder, the output characteristic diagram of a previous layer of a decoder network and the output characteristic diagram of a noise channel corresponding layer of the encoder are reconstructed through a denoising function, the input of a current convolutional layer is calculated, and the reconstruction function is as follows:
Figure BDA0002405526260000101
wherein ,
Figure BDA0002405526260000102
for the output of the l +1 th layer of the decoder,
Figure BDA0002405526260000103
for the l-th layer output of the encoder noise channel,
Figure BDA0002405526260000104
for the decoder l layer output, g (·,) is the reconstruction function;
(3) sending the training data set into the network, calculating a loss function to train the network model, as shown in fig. 3, the specific process is as follows:
(3a) sending the training data set into the constructed three-dimensional step convolution neural network;
(3b) calculating a categorical cross-entropy loss between an input sample label and an encoder network noise channel output as a supervised loss function at a given inputxiIn the case of i ∈ (1...., N), the supervised loss function is:
Figure BDA0002405526260000105
wherein ,
Figure BDA0002405526260000106
as output of the noise channel, yiInputting a corresponding label, wherein N is the number of samples;
(3c) and calculating the reconstruction error weighted sum of the reconstruction output of each convolution layer of the decoder and the convolution layer characteristic diagram corresponding to the clean channel of the encoder as an unsupervised loss function:
Figure BDA0002405526260000107
wherein L is the number of network layers set to 5, defining layer 0 as the input layer, z(l)Is the feed-forward output of the l-th layer of the encoder clean channel,
Figure BDA0002405526260000108
for de-noised output of the first layer of the decoder, lambdalThe loss weights for the l-th layer reconstruction error are set to [10,0.1,0.1,0.1]；
(3d) The total loss function is weighted sum of supervised loss and unsupervised loss, C ═ Cs+αCdα is a weight factor for unsupervised loss set at 0.0001;
(3e) reversely propagating the error layer by layer to an input layer by using a back propagation algorithm, and updating network parameters by a gradient descent method until the network converges to local optimum;
(3f) sending the verification data set into a training model, evaluating the performance of the model, and selecting model parameters with the best effect as a model to be finally used;
(4) the trained network model predicts on a test set to complete a coronary plaque detection task, and the specific process is as follows:
(4a) sending the test data set into a trained model, and predicting whether the three-dimensional blood vessel image contains plaque or not by a clean channel without noise of an encoder;
(4b) voting the test result according to the blood vessel section, acquiring a blood vessel section grade label, and evaluating the network performance.
The technical effects of the present invention will be described in detail with reference to experiments.
The evaluation criteria Precision (Precision), Recall (Recall), F1 score, and Accuracy (Accuracy) of the method proposed in the evaluation examples are defined as follows:
Figure BDA0002405526260000111
where TP represents the number of true positive samples predicted as positive samples. FP represents the number of true negative samples predicted as positive samples. TN denotes the number of true negative samples predicted as negative samples. FN represents the number of true positive samples predicted as negative samples. P denotes the number of true positive samples and N denotes the number of true negative samples. The evaluation index results are all between 0 and 1, and the closer to 1, the better the detection result. The test results on the test data were accuracy between [0.67,0.79], recall between [0.76,0.88], F1 score between [0.71,0.79], and accuracy between [0.75,0.82 ].
The effects of the present invention can be further illustrated by the following experiments:
the performance of the invention is compared with the performance of a fully supervised three-dimensional convolutional neural network and a convolutional recurrent neural network, and the same data set and sample size are adopted for testing, and the experimental results are shown in the following table:
Figure BDA0002405526260000121
as can be seen from the table, under the same training sample, the classification results of the test data set of the invention are all higher than those of the network using other supervised learning.
In conclusion, the semi-supervised learning strategy is used, the learning of unmarked data helps the supervised network to learn more information, the detection precision is improved, the problem of insufficient marked data is solved, and the generalization capability of the model is effectively improved.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided on a carrier medium such as a disk, CD-or DVD-ROM, programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.

Claims (10)

1. A coronary artery patch data detection method is characterized in that a three-dimensional image block is extracted along a central line point of a coronary artery on a three-dimensional medical image to serve as a sample, and a sample data set is divided; taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network; sending the training data set into a network, and calculating a loss function to train a network model; and predicting the trained network model on the test set to complete the coronary plaque detection task.
2. The coronary plaque data detection method of claim 1, comprising:
firstly, resampling an original three-dimensional medical image to ensure that the resolution of each datum is the same; manually marking or automatically generating a coronary artery central line for the data after resampling; taking a central line point of the coronary artery as a center, taking a tangent line of the central line as a central axis, and extracting a three-dimensional image block as a sample; extracting three-dimensional image blocks of all medical images to form a sample data set; dividing a training set, a verification set and a test set; carrying out random angle rotation and mirror image on the labeled samples in the training data set, and amplifying sample information; the coronary artery central line point randomly moves a certain amplitude in any direction vertical to the blood vessel direction of the coronary artery central line point, and training data of different blood vessel central lines are added;
step two, taking the coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output, and constructing a three-dimensional step convolution neural network, wherein the network structure mainly comprises an encoder network and a decoder network; constructing an encoder network, and learning the encoding information of the blood vessel sample; adding noise to the input and each hidden layer as a noise channel, and learning robust reconstruction representation; taking a channel without any noise as a clean channel, and training reconstruction information of each layer of the network; constructing a decoder network, and reconstructing the coding characteristics into an original image; adding jump connection, and reconstructing the decoder convolution layer output;
step three, the training data set is sent into the constructed three-dimensional step convolution neural network; calculating the classified cross entropy loss between the input sample label and the noise channel output of the encoder network as a supervision loss function; calculating the weighted sum of reconstruction errors between reconstruction outputs of each convolution layer of the decoder and the output characteristic diagram of the convolution layer corresponding to the clean channel of the encoder as an unsupervised loss function; reversely propagating the error layer by layer to an input layer by using a back propagation algorithm, and updating network parameters by a gradient descent method until the network converges; sending the verification data set into a training model, evaluating the performance of the model, and selecting model parameters with the best effect;
step four, the test data set is sent into a trained model, and a clean channel without noise of an encoder is used for predicting whether the three-dimensional blood vessel image contains plaques or not; voting the test result according to the blood vessel section, acquiring a blood vessel section grade label, and evaluating the network performance.
3. The coronary plaque data detecting method of claim 2, wherein said step one of extracting a three-dimensional image block as a sample along a coronary centerline point on a three-dimensional medical image, and dividing the sample data set comprises:
(1) resampling the original three-dimensional medical image to ensure that the resolution of each datum is the same;
(2) manually marking or automatically generating a coronary artery central line for the data after resampling;
(3) taking a central line point of the coronary artery as a center, taking a tangent line of the central line as a central axis, and extracting a three-dimensional image block as a sample;
(4) extracting three-dimensional image blocks of all medical image data along the central line of the coronary artery to form a sample data set;
(5) in the sample data set, a marked data set is randomly divided into a training set, a verification set and test set data, and unmarked data is added into the training set;
(6) carrying out random angle rotation and mirror image on the training data set, and amplifying sample information;
(7) the coronary artery central line point of the training data set sample randomly moves a certain amplitude to any direction vertical to the blood vessel direction of the coronary artery central line point, training data for central lines of different blood vessels are added, and errors caused by inaccurate central line extraction are reduced.
4. The coronary artery patch data detection method according to claim 2, wherein the second step is performed by taking the coronary artery three-dimensional image patch samples as input and whether the image contains the patch as output to construct a three-dimensional step convolution neural network, and comprises the following steps:
(1) taking a coronary artery three-dimensional image block sample as input, and whether the image contains a plaque or not as output, constructing a three-dimensional step convolution neural network, wherein the network structure mainly comprises an encoder network and a decoder network;
(2) constructing an encoder network, learning encoding information of a blood vessel sample, wherein the encoder network comprises three-dimensional convolutional layers and two full-connection layers, each convolutional layer learns nonlinear characteristics by using a Re L u activation function, then performing characteristic dimension reduction on a three-dimensional maximum pooling layer, the first full-connection layer is used as a characteristic encoding layer for reconstructing the sample by a decoder network, the second full-connection layer is a softmax output layer, and outputting blood vessel sample category information;
(3) on the basis of the structure of an encoder, adding isotropic Gaussian noise to an input layer and each hidden layer to serve as a noise channel of a network, enabling the encoder to learn robust reconstruction representation, and simultaneously taking a channel without adding any noise as a clean channel of the network to train reconstruction information of each layer of the network;
(4) constructing a decoder network, reconstructing the coding characteristics into an original image, wherein the decoder network consists of a full-connection layer and three-dimensional convolutional layers, the characteristic dimension is changed into the size corresponding to the corresponding layer of the encoder by using up-sampling before each convolutional layer, and the output of the last convolutional layer is a reconstructed image of the input sample of the encoder;
(5) the method comprises the following steps that jump connection is added between network layers corresponding to an encoder and a decoder, the output characteristic diagram of a previous layer of a decoder network and the output characteristic diagram of a noise channel corresponding layer of the encoder are reconstructed through a denoising function, the input of a current convolutional layer is calculated, and the reconstruction function is as follows:
Figure FDA0002405526250000031
wherein ,
Figure FDA0002405526250000032
for the output of the l +1 th layer of the decoder,
Figure FDA0002405526250000033
for the l-th layer output of the encoder noise channel,
Figure FDA0002405526250000034
for the decoder l layer output, g (,) is the reconstruction function.
5. The coronary plaque data detection method of claim 2, wherein said step three is sending the training data set into the network, calculating the loss function training network model, and proceeding according to the following steps:
(1) sending the training data set into the constructed three-dimensional step convolution neural network;
(2) calculating the classified cross entropy loss between the input sample label and the noise channel output of the encoder network as a supervision loss function, evaluating the consistency degree of the network prediction result and the real label, and determining the probability of the network prediction result and the real label when the input x is giveniIn the case of i ∈ (1...., N), the supervised loss function is:
Figure FDA0002405526250000035
wherein ,
Figure FDA0002405526250000036
as output of the noise channel, yiInputting a corresponding label, wherein N is the number of samples;
(3) reconstructing unmarked data by a decoder, calculating the reconstruction error weighted sum of each convolution layer reconstruction output of the decoder and the convolution layer characteristic diagram corresponding to the clean channel of the encoder as an unsupervised loss function, evaluating the learning ability of the network characterization input sample, and obtaining the unsupervised reconstruction loss function CdThe calculation formula is as follows:
Figure FDA0002405526250000037
wherein L is the number of network layers, z(l)Is the feed-forward output of the l-th layer of the encoder clean channel,
Figure FDA0002405526250000038
is a decoderDe-noised output of layer l, λlLoss weight for layer l reconstruction error;
(4) the total loss function is weighted sum of supervision loss and unsupervised loss, errors are reversely propagated to the input layer by using a back propagation algorithm, and network parameters are updated by a gradient descent method until the network converges to local optimum;
(5) and sending the verification data set into a training model, evaluating the performance of the model, and selecting the model parameters with the best effect as the model to be finally used.
6. The coronary plaque data detection method of claim 2, wherein the network model trained in step four predicts on a test set, and completing a coronary plaque detection task comprises:
(1) sending the test data set into a trained model, and predicting whether the three-dimensional blood vessel image contains plaque or not by a clean channel without noise of an encoder;
(2) voting the test result according to the blood vessel section, acquiring a blood vessel section grade label, and evaluating the network performance.
7. A program storage medium for receiving user input, the stored computer program causing an electronic device to perform the steps comprising: extracting a three-dimensional image block along a coronary artery central line point on a three-dimensional medical image as a sample, and dividing a sample data set; taking a coronary artery three-dimensional image block sample as input, and taking whether the image contains a plaque as output to construct a three-dimensional step convolution neural network; sending the training data set into a network, and calculating a loss function to train a network model; and predicting the trained network model on the test set to complete the coronary plaque detection task.
8. A coronary plaque data detection system for implementing the coronary plaque data detection method according to any one of claims 1 to 6, the coronary plaque data detection system comprising:
the sample data set dividing module is used for extracting a three-dimensional image block along a coronary artery central line point on the three-dimensional medical image as a sample and dividing a sample data set;
the three-dimensional step convolution neural network construction module is used for constructing a three-dimensional step convolution neural network by taking the coronary artery three-dimensional image block sample as input and taking whether the image contains a plaque as output;
the network model training module is used for sending the training data set into a network and calculating a loss function to train a network model;
and the network model prediction module is used for predicting the trained network model on the test set to complete the coronary plaque detection task.
9. A terminal equipped with the coronary plaque data detection system according to claim 8.
10. Use of the coronary plaque data detection method according to any of claims 1 to 6 in image detection systems and computed tomography systems.
CN202010160279.5A 2020-03-10 2020-03-10 Coronary plaque data detection method, system, storage medium and terminal Active CN111476757B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010160279.5A CN111476757B (en) 2020-03-10 2020-03-10 Coronary plaque data detection method, system, storage medium and terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010160279.5A CN111476757B (en) 2020-03-10 2020-03-10 Coronary plaque data detection method, system, storage medium and terminal

Publications (2)

Publication Number Publication Date
CN111476757A true CN111476757A (en) 2020-07-31
CN111476757B CN111476757B (en) 2023-05-23

Family

ID=71748167

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010160279.5A Active CN111476757B (en) 2020-03-10 2020-03-10 Coronary plaque data detection method, system, storage medium and terminal

Country Status (1)

Country Link
CN (1) CN111476757B (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112184748A (en) * 2020-09-30 2021-01-05 陕西科技大学 Deformable context coding network model and segmentation method for liver and liver tumor
CN112446877A (en) * 2020-12-14 2021-03-05 清华大学 Multi-branch tubular structure segmentation and marking method in three-dimensional image
CN112785591A (en) * 2021-03-05 2021-05-11 杭州健培科技有限公司 Method and device for detecting and segmenting costal fracture in CT image
CN113192062A (en) * 2021-05-25 2021-07-30 湖北工业大学 Arterial plaque ultrasonic image self-supervision segmentation method based on image restoration
CN113256621A (en) * 2021-06-25 2021-08-13 腾讯科技（深圳）有限公司 Image processing method, image processing device, computer equipment and storage medium
CN113313714A (en) * 2021-05-25 2021-08-27 西安交通大学第二附属医院 Coronary artery OCT image lesion plaque segmentation method based on improved U-Net network
CN113421250A (en) * 2021-07-05 2021-09-21 北京理工大学 Intelligent fundus disease diagnosis method based on lesion-free image training
CN113447997A (en) * 2021-05-21 2021-09-28 中国石油大学(北京) Reservoir fracture identification method, identification device and identification system
CN113470107A (en) * 2021-06-04 2021-10-01 广州医科大学附属第一医院 Bronchial centerline extraction method, bronchial centerline extraction system, and storage medium storing bronchial centerline extraction program
CN113538471A (en) * 2021-06-30 2021-10-22 上海联影医疗科技股份有限公司 Method and device for dividing patch, computer equipment and storage medium
CN114240844A (en) * 2021-11-23 2022-03-25 电子科技大学 Unsupervised key point positioning and target detection method in medical image
CN114332381A (en) * 2022-01-05 2022-04-12 北京理工大学 Aorta CT image key point detection method and system based on three-dimensional reconstruction
WO2022247168A1 (en) * 2021-05-24 2022-12-01 山东省人工智能研究院 Positional convolutional attention network-based vascular plaque ct image segmentation method
CN115588012A (en) * 2022-12-13 2023-01-10 四川大学 Pelvic artery blood vessel segmentation method, system, storage medium and terminal

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018213841A1 (en) * 2017-05-19 2018-11-22 Google Llc Multi-task multi-modal machine learning model
CN109903292A (en) * 2019-01-24 2019-06-18 西安交通大学 A kind of three-dimensional image segmentation method and system based on full convolutional neural networks
CN110222759A (en) * 2019-06-03 2019-09-10 中国医科大学附属第一医院 A kind of Plaque Vulnerability in Coronary Artery automatic recognition system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018213841A1 (en) * 2017-05-19 2018-11-22 Google Llc Multi-task multi-modal machine learning model
CN109903292A (en) * 2019-01-24 2019-06-18 西安交通大学 A kind of three-dimensional image segmentation method and system based on full convolutional neural networks
CN110222759A (en) * 2019-06-03 2019-09-10 中国医科大学附属第一医院 A kind of Plaque Vulnerability in Coronary Artery automatic recognition system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
刘敏;方志军;高永彬;: "基于深度学习的冠状动脉CTA三维分割" *

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112184748B (en) * 2020-09-30 2023-09-15 陕西科技大学 Deformable context coding network model and method for segmenting liver and liver tumor
CN112184748A (en) * 2020-09-30 2021-01-05 陕西科技大学 Deformable context coding network model and segmentation method for liver and liver tumor
CN112446877B (en) * 2020-12-14 2022-11-11 清华大学 Multi-branch tubular structure segmentation and marking method in three-dimensional image
CN112446877A (en) * 2020-12-14 2021-03-05 清华大学 Multi-branch tubular structure segmentation and marking method in three-dimensional image
CN112785591A (en) * 2021-03-05 2021-05-11 杭州健培科技有限公司 Method and device for detecting and segmenting costal fracture in CT image
CN113447997A (en) * 2021-05-21 2021-09-28 中国石油大学(北京) Reservoir fracture identification method, identification device and identification system
WO2022247168A1 (en) * 2021-05-24 2022-12-01 山东省人工智能研究院 Positional convolutional attention network-based vascular plaque ct image segmentation method
CN113192062A (en) * 2021-05-25 2021-07-30 湖北工业大学 Arterial plaque ultrasonic image self-supervision segmentation method based on image restoration
CN113313714B (en) * 2021-05-25 2023-10-27 西安交通大学第二附属医院 Coronary OCT (optical coherence tomography) image lesion plaque segmentation method based on improved U-Net network
CN113313714A (en) * 2021-05-25 2021-08-27 西安交通大学第二附属医院 Coronary artery OCT image lesion plaque segmentation method based on improved U-Net network
CN113470107A (en) * 2021-06-04 2021-10-01 广州医科大学附属第一医院 Bronchial centerline extraction method, bronchial centerline extraction system, and storage medium storing bronchial centerline extraction program
CN113470107B (en) * 2021-06-04 2023-07-14 广州医科大学附属第一医院 Bronchial centerline extraction method, system and storage medium thereof
CN113256621A (en) * 2021-06-25 2021-08-13 腾讯科技（深圳）有限公司 Image processing method, image processing device, computer equipment and storage medium
CN113538471A (en) * 2021-06-30 2021-10-22 上海联影医疗科技股份有限公司 Method and device for dividing patch, computer equipment and storage medium
CN113538471B (en) * 2021-06-30 2023-09-22 上海联影医疗科技股份有限公司 Plaque segmentation method, plaque segmentation device, computer equipment and storage medium
CN113421250A (en) * 2021-07-05 2021-09-21 北京理工大学 Intelligent fundus disease diagnosis method based on lesion-free image training
CN114240844A (en) * 2021-11-23 2022-03-25 电子科技大学 Unsupervised key point positioning and target detection method in medical image
CN114332381A (en) * 2022-01-05 2022-04-12 北京理工大学 Aorta CT image key point detection method and system based on three-dimensional reconstruction
CN115588012A (en) * 2022-12-13 2023-01-10 四川大学 Pelvic artery blood vessel segmentation method, system, storage medium and terminal

Also Published As

Publication number Publication date
CN111476757B (en) 2023-05-23

Similar Documents

Publication Publication Date Title
CN111476757B (en) Coronary plaque data detection method, system, storage medium and terminal
KR102491988B1 (en) Methods and systems for using quantitative imaging
CN111709953B (en) Output method and device in lung lobe segment segmentation of CT (computed tomography) image
US10453191B2 (en) Automated intravascular plaque classification
CN110197713B (en) Medical image processing method, device, equipment and medium
CN110400298B (en) Method, device, equipment and medium for detecting heart clinical index
US20220284583A1 (en) Computerised tomography image processing
CN112991346B (en) Training method and training system for learning network for medical image analysis
US20220215541A1 (en) Method, device and computer-readable medium for automatically classifying coronary lesion according to cad-rads classification by a deep neural network
CN111612756B (en) Coronary artery specificity calcification detection method and device
CN112785551A (en) Coronary artery segmentation method based on deep learning
Wei et al. Computerized detection of noncalcified plaques in coronary CT angiography: evaluation of topological soft gradient prescreening method and luminal analysis
CN111724365B (en) Interventional instrument detection method, system and device for endovascular aneurysm repair operation
CN110956610A (en) Method and system for predicting thrombolytic curative effect of lower limb deep venous thrombosis based on sparse representation
CN114511581B (en) Multi-task multi-resolution collaborative esophageal cancer lesion segmentation method and device
CN114998292A (en) Cardiovascular calcified plaque detection system based on residual double attention mechanism
US20220215542A1 (en) Method, device and computer-readable medium for automatically detecting hemodynamically significant coronary stenosis
US20230394654A1 (en) Method and system for assessing functionally significant vessel obstruction based on machine learning
CN116416221A (en) Ultrasonic image analysis method
CN115984300A (en) Pulmonary embolism identification method based on flat scan CT
CN113223704B (en) Auxiliary diagnosis method for computed tomography aortic aneurysm based on deep learning
CN115294023A (en) Liver tumor automatic segmentation method and device
CN113674291A (en) Full-type aortic dissection real-false lumen image segmentation method and system
CN114519722A (en) Carotid artery extraction method based on convolutional neural network
CN114022405A (en) Intravascular ultrasound image processing method based on deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant