CN114141339B - Pathological image classification method, device, equipment and storage medium for membranous nephropathy - Google Patents
Pathological image classification method, device, equipment and storage medium for membranous nephropathy Download PDFInfo
- Publication number
- CN114141339B CN114141339B CN202210103536.0A CN202210103536A CN114141339B CN 114141339 B CN114141339 B CN 114141339B CN 202210103536 A CN202210103536 A CN 202210103536A CN 114141339 B CN114141339 B CN 114141339B
- Authority
- CN
- China
- Prior art keywords
- classification
- glomerular
- membranous nephropathy
- image
- pathological
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000001575 pathological effect Effects 0.000 title claims abstract description 101
- 206010018372 Glomerulonephritis membranous Diseases 0.000 title claims abstract description 68
- 201000008350 membranous glomerulonephritis Diseases 0.000 title claims abstract description 68
- 231100000855 membranous nephropathy Toxicity 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000011218 segmentation Effects 0.000 claims abstract description 91
- 238000003062 neural network model Methods 0.000 claims abstract description 44
- 230000001434 glomerular Effects 0.000 claims abstract description 35
- 238000011862 kidney biopsy Methods 0.000 claims abstract description 35
- 210000000585 glomerular basement membrane Anatomy 0.000 claims abstract description 34
- 230000002093 peripheral effect Effects 0.000 claims abstract description 10
- 238000012549 training Methods 0.000 claims description 53
- 230000006870 function Effects 0.000 claims description 40
- 238000012360 testing method Methods 0.000 claims description 34
- 230000004927 fusion Effects 0.000 claims description 32
- 238000005070 sampling Methods 0.000 claims description 20
- 238000012795 verification Methods 0.000 claims description 17
- 238000010586 diagram Methods 0.000 claims description 14
- 238000011176 pooling Methods 0.000 claims description 12
- 238000010200 validation analysis Methods 0.000 claims description 6
- 238000002372 labelling Methods 0.000 claims description 5
- 230000007170 pathology Effects 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 abstract description 4
- 201000010099 disease Diseases 0.000 abstract description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 abstract description 2
- 210000002469 basement membrane Anatomy 0.000 description 6
- 239000012133 immunoprecipitate Substances 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000013145 classification model Methods 0.000 description 3
- 230000008021 deposition Effects 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000000638 solvent extraction Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 210000004027 cell Anatomy 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 206010029164 Nephrotic syndrome Diseases 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 208000020832 chronic kidney disease Diseases 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 208000028208 end stage renal disease Diseases 0.000 description 1
- 201000000523 end stage renal failure Diseases 0.000 description 1
- 210000002919 epithelial cell Anatomy 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010166 immunofluorescence Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000012421 spiking Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Epidemiology (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a pathological image classification method, a device, equipment and a storage medium for membranous nephropathy, wherein the method comprises the following steps: acquiring a glomerular pathological image of a renal biopsy; inputting the glomerular pathological image of the renal biopsy into a segmentation module of a pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex thereof; and inputting the segmentation mask image into a classification module of the MTLSC-Net neural network model to obtain a classification result of the pathological image of the membranous nephropathy. According to the pathological image classification method provided by the embodiment of the application, based on the multitask deep learning neural network model, the characteristics related to the classification of the pathological images of the membranous nephropathy are obtained through segmentation, the pathological images of the membranous nephropathy are rapidly and accurately classified, a pathologist is assisted in classifying the disease stage of the membranous nephropathy, and the working efficiency of the pathologist is effectively improved.
Description
Technical Field
The present invention relates to the field of image classification technologies, and in particular, to a method, an apparatus, a device, and a storage medium for classifying pathological images of membranous nephropathy.
Background
Membranous nephropathy is the second most common disease in nephrotic syndrome in china, with a markedly increasing trend in incidence and nearly half of these patients may develop end-stage renal failure. The membranous nephropathy needs to be diagnosed and staged through pathological examination such as light microscope, electron microscope, immunofluorescence and the like by renal biopsy, and the pathological feature of the membranous nephropathy is immune complex deposition under epithelial cells of a glomerular layer, so that diffuse glomerular basement membrane is thickened unevenly and spiked. The pathological stage of membranous nephropathy is usually divided into five stages, but stages I, II, III and IV have more important significance. When the membranous nephropathy is staged, a pathologist needs to carefully observe various stained renal biopsy pathological sections under an electron microscope or a light microscope, and confirm all glomeruli and related pathological features on the same section one by one, so that the work of pathological physician reading is heavy.
At present, various medical image data are increased in magnitude order, medical pathological images are complex, tissues and cells to be detected are difficult to distinguish, and great challenges are brought to the traditional medical image pattern recognition technology. Therefore, how to accurately classify pathological images of membranous nephropathy according to different properties of glomerular basement membrane and immune complexes thereof is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the application provides a pathological image classification method, a pathological image classification device, pathological image classification equipment and a storage medium for membranous nephropathy. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In a first aspect, an embodiment of the present application provides a method for classifying pathological images of membranous nephropathy, including:
acquiring a glomerular pathological image of a renal biopsy;
inputting the glomerular pathological image of the renal biopsy into a segmentation module of a pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex thereof;
and inputting the segmentation mask image into a classification module of the MTLSC-Net neural network model to obtain a classification result of the pathological image of the membranous nephropathy.
In an optional embodiment, before inputting the glomerular pathological image of the kidney biopsy into the pre-trained MTLSC-Net neural network model, the method further comprises:
constructing a training set, a testing set and a verifying set;
and training and testing the MTLSC-Net neural network model according to the training set, the testing set and the verification set.
In an optional embodiment, constructing a training set, a test set, and a validation set comprises:
acquiring a plurality of glomerular pathological images of kidney biopsy;
marking the pixel value of a glomerular basement membrane and immune complexes around the glomerular basement membrane in a renal biopsy glomerular pathological image as 1, marking the pixel values of the rest parts as 0 to obtain a marked segmentation mask image data set, and dividing the segmentation mask image data set into a first training set, a first test set and a first verification set;
labeling the category of a glomerular basement membrane in the segmentation mask image dataset to obtain a labeled classified image dataset, and dividing the classified image dataset into a second training set, a second testing set and a second verification set.
In an optional embodiment, training the MTLSC-Net neural network model from the training set, the test set, and the validation set comprises:
training a partitioning module of the MTLSC-Net neural network model according to the first training set, the first testing set and the first verifying set and the binary cross entropy loss function;
the binary cross entropy loss function is shown below:
wherein,
is the value of the true tag(s),
is a predicted value;
training a classification module of the MTLSC-Net neural network model according to a second training set, a second testing set and a second verifying set and a multi-classification cross entropy loss function;
the multi-class cross entropy loss function is as follows:
wherein,
in order to be a true tag value,
in order to predict the value of the target,
are classified.
In an optional embodiment, the method further comprises training the MTLSC-Net neural network model according to a total loss function;
the total loss function is as follows:
wherein,
representing model parameters.
In an optional embodiment, the segmentation module of the MTLSC-Net neural network model comprises:
the coding unit consists of a residual error network and a pooling layer and is used for down-sampling the renal biopsy glomerular pathological image to obtain a plurality of feature maps with different scales;
the decoding unit consists of an upsampling layer and a Sigmoid function, is used for upsampling the feature map, adopts the Sigmoid function to predict, marks the predicted value larger than a preset threshold value as 1, and marks the predicted value smaller than the preset threshold value as 0 to obtain a segmentation mask;
and the transverse connection unit is used for merging the feature map after the up-sampling in the decoding unit and the feature map with the same scale in the coding unit and fusing the high-layer semantic information and the low-layer position information.
In an optional embodiment, the classification module of the MTLSC-Net neural network model comprises:
the feature fusion unit is used for merging the feature maps output by the segmentation modules to obtain a merged multi-scale fusion feature map;
the convolution pooling unit is used for selectively learning the features of the multi-scale fusion feature map according to the segmentation mask and downsampling the multi-scale fusion feature map;
and the full-connection unit is used for inputting the multi-scale fusion characteristic diagram after down sampling into a full-connection layer to obtain a classification result of the pathological image of the membranous nephropathy.
In a second aspect, an embodiment of the present application provides a pathological image classification device for membranous nephropathy, including:
the acquisition module is used for acquiring a glomerular pathological image of the renal biopsy;
the segmentation module is used for inputting the glomerular pathological image of the renal biopsy into the segmentation module of the pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex of the glomerular basement membrane;
and the classification module is used for inputting the segmentation mask image into the classification module of the MTLSC-Net neural network model to obtain the classification result of the pathological image of the membranous nephropathy.
In a third aspect, an embodiment of the present application provides a pathological image classification device for membranous nephropathy, including a processor and a memory storing program instructions, where the processor is configured to execute the pathological image classification method for membranous nephropathy provided in the above embodiment when executing the program instructions.
In a fourth aspect, the present application provides a computer-readable medium, on which computer-readable instructions are stored, where the computer-readable instructions are executed by a processor to implement the method for classifying pathological images of membranous nephropathy provided in the foregoing embodiment.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
the embodiment of the application provides a pathological image classification method of membranous nephropathy based on deep learning, segmentation and labeling are carried out based on pathological features of membranous nephropathy, a multi-scale feature fusion mode is adopted, high-level semantic information and low-level position information are combined, and accuracy of a basement membrane of a glomerulus and a peripheral immune complex deposition segmentation mask of the glomerulus is improved. Residual connection is adopted in the segmentation module to replace a convolutional neural network, so that the capability of extracting features of the model is improved, and the network learning process is simplified. Secondly, the classification module of the application adopts a feature sharing mode, directly utilizes the multi-scale features of the segmentation module, and combines the segmentation mask to classify the extracted features. Because similar cells in pathological sections are numerous and the pathological background is complex, the classification module can concentrate on pathological features related to membranous nephropathy in the training process by combining with a segmentation mask, and the classification effect is better. Furthermore, the MTLSC-Net model provided by the application adopts a loss function of multi-task learning to train the segmentation module and the classification module simultaneously through the multi-task learning technology. The two tasks share one model, shared information is mutually supplemented, and the accuracy of image classification is improved. The pathological images of membranous nephropathy can be rapidly and accurately classified, and pathologists can rapidly perform follow-up diagnosis according to classification results, so that the working efficiency of the pathologists can be effectively improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart illustrating a pathological image classification method of membranous nephropathy according to an exemplary embodiment;
FIG. 2 is a schematic diagram of an MTLSC-Net model according to an exemplary embodiment;
FIG. 3 is a diagram illustrating classification results of a membranous nephropathy pathology image, according to an exemplary embodiment;
FIG. 4 is a schematic illustration of a glomerular basement membrane and its surrounding immune complexes shown according to an exemplary embodiment;
FIG. 5 is a schematic diagram illustrating a segmentation mask in accordance with an illustrative embodiment;
fig. 6 is a schematic structural view showing a pathological image classification device of membranous nephropathy according to an exemplary embodiment;
fig. 7 is a schematic structural view showing a pathological image classification apparatus for membranous nephropathy according to an exemplary embodiment;
FIG. 8 is a schematic diagram illustrating a computer storage medium in accordance with an exemplary embodiment.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The embodiment of the application provides a pathological image Classification model of membranous nephropathy (MTLSC-Net) based on deep Learning, which is based on a residual error structure and a U-shaped framework, divides an area with significant changes of membranous nephropathy, classifies division masks with changed features, fuses Multi-scale features extracted by a division module and then applies the Multi-scale features to a Classification module, and inputs the division masks to the Classification module. The segmentation module and the classification module adopt a feature sharing mode, so that the calculation time is greatly reduced, and the effective utilization degree of the multi-task learning model on the features is improved. The classification is carried out on the basis of segmentation, so that the classification result of the pathological image of the membranous nephropathy is more accurate, the film-watching efficiency of a doctor is effectively improved, and the doctor is assisted to carry out follow-up diagnosis according to the classification result.
The pathological image classification method for membranous nephropathy provided in the embodiment of the present application will be described in detail below with reference to fig. 1. Referring to fig. 1, the method specifically includes the following steps.
S101, acquiring a glomerular pathological image of the kidney biopsy.
In one possible implementation manner, an image of the kidney biopsy is obtained, and then a glomerular image in the kidney biopsy image is extracted to obtain a glomerular pathological image of the kidney biopsy.
The pretrained MTLSC-Net neural network model in the embodiment of the application can continuously segment the segmented glomerular pathological image of the renal biopsy to obtain an image of a glomerular basement membrane and a peripheral immune complex thereof, and then classify according to the characteristics of the glomerular basement membrane and the peripheral immune complex thereof.
S102, inputting the glomerular pathological image of the renal biopsy into a segmentation module of a pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex of the glomerular basement membrane.
In a possible implementation manner, before performing step S102, constructing a training set, a testing set, and a validation set, and training the MTLSC-Net neural network model according to the training set, the testing set, and the validation set.
Specifically, a training set, a test set and a verification set of a segmentation module in the MTLSC-Net neural network model are constructed firstly. A plurality of glomerular pathological images of renal biopsy are obtained, based on the pathological features of membranous nephropathy, only the glomerular basement membrane and the surrounding immune complexes in the glomerular images need to be segmented and labeled, fig. 4 is a schematic diagram of the glomerular basement membrane and the surrounding immune complexes according to an exemplary embodiment, and the segmentation and labeling of the glomerular basement membrane and the surrounding immune complexes in the glomerular images can be referred to in fig. 4.
In a possible implementation manner, in the pathological section, the pixel value of the glomerular basement membrane and the immune complex around the glomerular basement membrane is labeled as 1, and the pixel value of the rest part of the glomerular basement membrane is labeled as 0, so as to obtain the labeled binary segmentation mask image data set. Fig. 5 is a diagram illustrating a segmentation mask, according to an example embodiment.
Further, the segmentation mask image data set is randomly divided into a first training set, a first test set and a first verification set in a proportion of 80%, 10% and 10%. The embodiment of the application does not specifically limit the division ratio, and can be set according to actual requirements.
Further, a training set, a testing set and a verification set of the classification module in the MTLSC-Net neural network model are constructed.
Specifically, the classification of the glomerular basement membrane in the segmentation mask image is labeled to obtain a labeled classified image dataset.
In one possible implementation, the classification can be 4 types according to the condition of the basement membrane of the glomerulus and the immune complex around the glomerulus, and fig. 3 is a schematic diagram illustrating the classification result of a pathological image of membranous nephropathy according to an exemplary embodiment, as shown in fig. 3, from left to right, the first type image, the second type image, the third type image and the fourth type image are respectively.
Wherein, the surface of the basement membrane of the first image has few immunoprecipitates, irregular shapes, sparse distribution and no nail-shaped protrusions. Immunoprecipitates increased on the basement membrane of the second type of image, spiking and partitioning of the immunoprecipitates occurred. The basement membrane of the third image was clearly thickened, surrounding the immunoprecipitates. The fourth type of image partially disappeared the immunoprecipitates, leaving a worm-eaten basement membrane.
Further, the classified image data set is randomly divided into a second training set, a second testing set and a second verification set according to the proportion of 80%, 10% and 10%. The embodiment of the application does not specifically limit the division ratio, and can be set according to actual requirements.
Then, the MTLSC-Net neural network model is trained by utilizing the constructed training set, the testing set and the verification set. The MTLSC-Net neural network model comprises a segmentation module and a classification module, and the segmentation module is trained firstly.
The segmentation module is designed by utilizing residual connection and a U-shaped framework, a plurality of feature maps with different scales are obtained after up-sampling for a plurality of times, the multi-scale feature maps are fused, and the high-level semantic advantage and the low-level detail advantage of the model are combined to accurately segment the pathological change area related to the membranous nephropathy.
Specifically, the segmentation module is mainly divided into an encoder for feature extraction, a decoder for feature map upsampling, and a horizontal line connection for feature fusion, and can refer to the upper half of the dotted line in fig. 2.
As shown in fig. 2, the segmentation module includes an encoding unit, which is composed of a residual network and a pooling layer, and is configured to perform downsampling on a renal biopsy glomerular pathology image to obtain a plurality of feature maps of different scales. Specifically, the encoding unit is mainly used for extracting a feature map of an image, and after each feature map is extracted, the feature map passes through a pooling layer to obtain a feature scale map, and finally, the feature scale map is subjected to four times of downsampling to obtain five feature maps m1, m2, m3, m4 and m5 with different scales.
The device also comprises a decoding unit which consists of an up-sampling layer and a Sigmoid function and is used for up-sampling the characteristic diagram and predicting by adopting the Sigmoid function to obtain the segmentation mask. Specifically, the decoding unit part mainly comprises an up-sampling layer, a small-scale feature map is up-sampled to a large-scale feature map, a final output layer of the decoding unit adopts a Sigmoid function to predict, a predicted value larger than a threshold is marked as 1, a predicted value smaller than the threshold is marked as 0, and the decoding unit finally outputs a segmentation mask S for subsequent auxiliary classification model selection features.
The device also comprises a transverse connection unit which is used for merging the feature map after the up-sampling in the decoding unit and the feature map with the same scale in the coding unit and fusing the high-level semantic information and the low-level position information.
When the segmentation module is trained, the segmentation module is trained by using a first training set, a binary cross entropy is used as a loss function, and the loss function of the binary cross entropy is as follows:
wherein,
in order to be a true tag value,
is a predicted value.
The classification module is then trained. The classification module performs up-sampling on the multi-scale features of the segmentation module, then combines the multi-scale features with the segmentation mask, and classifies pathological sections of membranous nephropathy through a convolutional neural network and a full-link layer. The classification module can be seen in the lower half of the dashed line in fig. 2.
As shown in fig. 2, the classification module includes a feature fusion unit, configured to merge the feature maps output by the segmentation module to obtain a merged multi-scale fusion feature map. Specifically, multi-scale feature map fusion is performed on the feature maps M3, M4, and M5 of the top three layers output by the segmentation module, and M4 and M5 may be upsampled to the same scale as M3 and then merged to obtain a merged multi-scale fusion feature map M.
The system also comprises a convolution pooling unit which is used for selectively learning the characteristics of the multi-scale fusion characteristic graph according to the segmentation mask and downsampling the multi-scale fusion characteristic graph. Specifically, feature selective learning is performed on the multi-scale fusion feature map M by combining with a segmentation mask S output by a segmentation module, and downsampling is performed on the multi-scale fusion feature map M by adopting a mode of repeating a convolution neural network and a pooling layer twice.
The full-connection unit is used for inputting the multi-scale fusion characteristic diagram after down-sampling into a full-connection layer to obtain a classification result of the pathological image of the membranous nephropathy. Specifically, the multi-scale fusion feature map M obtained through the final down-sampling is input into a full connection layer, the full connection layer is classified, 4 neurons of the output layer are obtained, the classification result of the glomerulopathological image is obtained, and the activation function adopted by the final output layer is a Softmax function.
When training the classification module, training by using a second training set, and adopting a multi-classification cross entropy loss function, wherein the multi-classification cross entropy loss function is as follows:
wherein,
in order to be a true tag value,
in order to predict the value of the target,
are classified.
Further, model optimization and testing are performed. The model training aims at minimizing the total loss function, and the performance of the model is evaluated by combining the first verification set and the second verification set to obtain the parameters corresponding to the optimal model.
The total loss function is as follows:
wherein,
representing model parameters.
And further, testing the obtained optimal model by using the first test set and the second test set, and evaluating the performance and classification capability of the optimal model. And obtaining the well-trained MTLSC-Net neural network model.
After the trained MTLSC-Net neural network model is obtained, the glomerular biopsy glomerular pathological image is input into a segmentation module of the trained MTLSC-Net neural network model, and a segmentation mask image of a glomerular basement membrane and a peripheral immune complex of the glomerular basement membrane is obtained.
S103, inputting the segmentation mask image into a classification module of the MTLSC-Net neural network model to obtain a classification result of the pathological image of the membranous nephropathy.
Furthermore, the binary segmentation mask image output by the segmentation module is input into the classification module, so that the classification result of the membranous nephropathy pathological image can be obtained.
Because the pathological features of membranous nephropathy are mainly reflected in glomerular basement membrane change and immune complex deposition, and the background in the pathological section of renal biopsy is complex and numerous in interference, the multitask deep learning classification model provided by the application firstly partitions the region with significant change of membranous nephropathy and then classifies the partition mask with change features. And fusing the multi-scale features extracted by the segmentation module, applying the fused multi-scale features to the classification module, and inputting the segmentation mask code to the classification module. The segmentation module and the classification module adopt a feature sharing mode, so that the calculation time is greatly reduced, and the effective utilization degree of the multi-task learning model to the features is improved. And classification is performed on the basis of segmentation, so that the image classification result is more accurate and reliable. The pathological images of membranous nephropathy can be rapidly and accurately classified, and pathologists can rapidly perform follow-up diagnosis according to classification results, so that the working efficiency of the pathologists can be effectively improved.
An embodiment of the present application further provides a pathological image classification device for membranous nephropathy, which is configured to perform the pathological image classification method for membranous nephropathy according to the above embodiment, as shown in fig. 6, and the device includes:
an obtaining module 601, configured to obtain a glomerular pathological image of a renal biopsy;
a segmentation module 602, configured to input the glomerular pathological image of the renal biopsy into a segmentation module of a pre-trained MTLSC-Net neural network model, to obtain a segmentation mask image of a glomerular basement membrane and a surrounding immune complex thereof;
and the classification module 603 is used for inputting the segmentation mask image into a classification module of the MTLSC-Net neural network model to obtain a classification result of the pathological image of the membranous nephropathy.
The system also comprises a model training module used for constructing a training set, a test set and a verification set; and training the MTLSC-Net neural network model according to the training set, the testing set and the verification set.
The model training module is used for acquiring a plurality of kidney biopsy glomerulus pathological images;
marking the pixel value of a glomerular basement membrane and immune complexes around the glomerular basement membrane in a renal biopsy glomerular pathological image as 1, marking the pixel values of the rest parts as 0 to obtain a marked segmentation mask image data set, and dividing the segmentation mask image data set into a first training set, a first test set and a first verification set;
labeling the category of a glomerular basement membrane in the segmentation mask image dataset to obtain a labeled classified image dataset, and dividing the classified image dataset into a second training set, a second testing set and a second verification set.
The model training module is used for training a partitioning module of the MTLSC-Net neural network model according to a first training set, a first testing set and a first verifying set and a binary cross entropy loss function;
the binary cross entropy loss function is shown below:
wherein,
in order to be a true tag value,
is a predicted value;
training a classification module of the MTLSC-Net neural network model according to a second training set, a second testing set and a second verifying set and a multi-classification cross entropy loss function;
the multi-class cross entropy loss function is as follows:
wherein,
in order to be a true tag value,
in order to predict the value of the target,
are classified.
In an optional embodiment, the method further comprises training the MTLSC-Net neural network model according to a total loss function;
the total loss function is as follows:
wherein,
representing model parameters.
In an optional embodiment, the segmentation module of the MTLSC-Net neural network model comprises:
the coding unit consists of a residual error network and a pooling layer and is used for down-sampling the renal biopsy glomerular pathological image to obtain a plurality of feature maps with different scales;
the decoding unit consists of an upsampling layer and a Sigmoid function, is used for upsampling the feature map, adopts the Sigmoid function to predict, marks the predicted value larger than a preset threshold value as 1, and marks the predicted value smaller than the preset threshold value as 0 to obtain a segmentation mask;
and the transverse connection unit is used for merging the feature map after the up-sampling in the decoding unit and the feature map with the same scale in the coding unit and fusing the high-layer semantic information and the low-layer position information.
In an optional embodiment, the classification module of the MTLSC-Net neural network model comprises:
the feature fusion unit is used for merging the feature maps output by the segmentation modules to obtain a merged multi-scale fusion feature map;
the convolution pooling unit is used for selectively learning the features of the multi-scale fusion feature map according to the segmentation mask and downsampling the multi-scale fusion feature map;
and the full-connection unit is used for inputting the multi-scale fusion characteristic diagram after down sampling into a full-connection layer to obtain a classification result of the pathological image of the membranous nephropathy.
It should be noted that, when the pathological image classification method for membranous nephropathy is performed, the above-mentioned division of each functional module is only used for example, and in practical applications, the above-mentioned function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the above-mentioned functions. In addition, the pathological image classification device for membranous nephropathy and the pathological image classification method for membranous nephropathy provided in the above embodiments belong to the same concept, and details of implementation processes are shown in the method embodiments, which are not described herein again.
The embodiment of the present application further provides an electronic device corresponding to the pathological image classification method for membranous nephropathy provided in the foregoing embodiment, so as to execute the pathological image classification method for membranous nephropathy.
Referring to fig. 7, a schematic diagram of an electronic device provided in some embodiments of the present application is shown. As shown in fig. 7, the electronic apparatus includes: the processor 700, the memory 701, the bus 702 and the communication interface 703, wherein the processor 700, the communication interface 703 and the memory 701 are connected through the bus 702; the memory 701 stores a computer program that can be executed on the processor 700, and the processor 700 executes the computer program to execute the pathological image classification method for membranous nephropathy provided in any of the foregoing embodiments of the present application.
The Memory 701 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The processor 700 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 700. The Processor 700 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701, and completes the steps of the method in combination with the hardware thereof.
The electronic device provided by the embodiment of the application and the pathological image classification method of membranous nephropathy provided by the embodiment of the application have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Referring to fig. 8, the computer readable storage medium is an optical disc 800, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer readable storage medium performs the method for classifying pathological images of membranous nephropathy according to any of the embodiments described above.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the pathological image classification method of membranous nephropathy provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented by the application program stored in the computer-readable storage medium.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (8)
1. A method for classifying pathological images of membranous nephropathy, comprising:
acquiring a glomerular pathological image of a renal biopsy;
inputting the glomerular pathological image of the renal biopsy into a segmentation module of a pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex thereof; the segmentation module includes: the coding unit consists of a residual error network and a pooling layer and is used for down-sampling the renal biopsy glomerular pathological image to obtain a plurality of feature maps with different scales; the decoding unit consists of an upsampling layer and a Sigmoid function, is used for upsampling the feature map, adopts the Sigmoid function to predict, marks the predicted value larger than a preset threshold value as 1, and marks the predicted value smaller than the preset threshold value as 0 to obtain a segmentation mask; the transverse connection unit is used for merging the feature map after the up-sampling in the decoding unit and the feature map with the same scale in the coding unit and fusing the high-level semantic information and the low-level position information;
inputting the segmentation mask image into a classification module of the MTLSC-Net neural network model to obtain a classification result of the pathological image of membranous nephropathy, wherein the classification module comprises: the feature fusion unit is used for merging the feature maps output by the segmentation modules to obtain a merged multi-scale fusion feature map; the convolution pooling unit is used for selectively learning the features of the multi-scale fusion feature map according to the segmentation mask and downsampling the multi-scale fusion feature map; and the full-connection unit is used for inputting the multi-scale fusion characteristic diagram after down sampling into a full-connection layer to obtain a classification result of the pathological image of the membranous nephropathy.
2. The method of claim 1, wherein prior to inputting the renal biopsy glomerular pathology image into a pre-trained MTLSC-Net neural network model, further comprising:
constructing a training set, a testing set and a verifying set;
and training the MTLSC-Net neural network model according to the training set, the test set and the verification set.
3. The method of claim 2, wherein constructing a training set, a test set, and a validation set comprises:
acquiring a plurality of glomerular pathological images of kidney biopsy;
marking the pixel value of a glomerular basement membrane and immune complexes around the glomerular basement membrane in the renal biopsy glomerular pathological image as 1, marking the pixel values of the rest parts as 0 to obtain a marked segmentation mask image data set, and dividing the segmentation mask image data set into a first training set, a first test set and a first verification set;
labeling the category of a glomerular basement membrane in the segmentation mask image data set to obtain a labeled classified image data set, and dividing the classified image data set into a second training set, a second testing set and a second verification set.
4. The method of claim 3, wherein training the MTLSC-Net neural network model from the training set, the test set, and the validation set comprises:
training a segmentation module of the MTLSC-Net neural network model according to the first training set, the first testing set and the first verifying set and a binary cross entropy loss function;
the binary cross entropy loss function is as follows:
wherein,
in order to be a true tag value,
is a predicted value;
training a classification module of the MTLSC-Net neural network model according to the second training set, the second testing set and the second verifying set and the multi-classification cross entropy loss function;
the multi-class cross-entropy loss function is as follows:
wherein,
in order to be a true tag value,
in order to predict the value of the target,
are classified.
5. The method of claim 4, further comprising training the MTLSC-Net neural network model according to a total loss function;
the total loss function is as follows:
wherein,
representing model parameters.
6. A pathological image classification device for membranous nephropathy, comprising:
the acquisition module is used for acquiring a glomerular pathological image of the renal biopsy;
the segmentation module is used for inputting the glomerular pathological image of the renal biopsy into the segmentation module of the pre-trained MTLSC-Net neural network model to obtain a segmentation mask image of a glomerular basement membrane and a peripheral immune complex of the glomerular basement membrane; the segmentation module includes: the coding unit consists of a residual error network and a pooling layer and is used for down-sampling the renal biopsy glomerular pathological image to obtain a plurality of feature maps with different scales; the decoding unit consists of an upsampling layer and a Sigmoid function, is used for upsampling the feature map, adopts the Sigmoid function to predict, marks the predicted value larger than a preset threshold value as 1, and marks the predicted value smaller than the preset threshold value as 0 to obtain a segmentation mask; the transverse connection unit is used for merging the feature map after the up-sampling in the decoding unit and the feature map with the same scale in the coding unit and fusing the high-level semantic information and the low-level position information;
the classification module is used for inputting the segmentation mask image into the classification module of the MTLSC-Net neural network model to obtain the classification result of the membranous nephropathy pathological image, and the classification module comprises: the feature fusion unit is used for merging the feature maps output by the segmentation modules to obtain a merged multi-scale fusion feature map; the convolution pooling unit is used for selectively learning the features of the multi-scale fusion feature map according to the segmentation mask and downsampling the multi-scale fusion feature map; and the full-connection unit is used for inputting the multi-scale fusion characteristic diagram after down sampling into a full-connection layer to obtain a classification result of the pathological image of the membranous nephropathy.
7. A pathological image classification device of membranous nephropathy, comprising a processor and a memory storing program instructions, the processor being configured to execute the pathological image classification method of membranous nephropathy according to any one of claims 1 to 5 when executing the program instructions.
8. A computer readable medium having computer readable instructions stored thereon, which are executed by a processor to implement the method for classifying pathological images of membranous nephropathy according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210103536.0A CN114141339B (en) | 2022-01-26 | 2022-01-26 | Pathological image classification method, device, equipment and storage medium for membranous nephropathy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210103536.0A CN114141339B (en) | 2022-01-26 | 2022-01-26 | Pathological image classification method, device, equipment and storage medium for membranous nephropathy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114141339A CN114141339A (en) | 2022-03-04 |
| CN114141339B true CN114141339B (en) | 2022-08-05 |
Family
ID=80382198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210103536.0A Active CN114141339B (en) | 2022-01-26 | 2022-01-26 | Pathological image classification method, device, equipment and storage medium for membranous nephropathy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114141339B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115187819B (en) * | 2022-08-23 | 2023-05-16 | 北京医准智能科技有限公司 | Training method and device for image classification model, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10482603B1 (en) * | 2019-06-25 | 2019-11-19 | Artificial Intelligence, Ltd. | Medical image segmentation using an integrated edge guidance module and object segmentation network |
| CN111798416A (en) * | 2019-06-20 | 2020-10-20 | 太原理工大学 | Intelligent glomerulus detection method and system based on pathological image and deep learning |
| CN112633416A (en) * | 2021-01-16 | 2021-04-09 | 北京工业大学 | Brain CT image classification method fusing multi-scale superpixels |
| CN112785617A (en) * | 2021-02-23 | 2021-05-11 | 青岛科技大学 | Automatic segmentation method for residual UNet rectal cancer tumor magnetic resonance image |
| CN113160226A (en) * | 2021-05-24 | 2021-07-23 | 苏州大学 | Two-way guide network-based classification segmentation method and system for AMD lesion OCT image |
| CN113763340A (en) * | 2021-08-25 | 2021-12-07 | 中国人民解放军空军军医大学 | Automatic grading method based on multitask deep learning ankylosing spondylitis |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110599448B (en) * | 2019-07-31 | 2022-03-15 | 浙江工业大学 | Migratory learning lung lesion tissue detection system based on MaskScoring R-CNN network |
| CN110675406A (en) * | 2019-09-16 | 2020-01-10 | 南京信息工程大学 | CT image kidney segmentation algorithm based on residual double-attention depth network |
| CN112926612A (en) * | 2019-12-06 | 2021-06-08 | 中移(成都)信息通信科技有限公司 | Pathological image classification model training method, pathological image classification method and device |
-
2022
- 2022-01-26 CN CN202210103536.0A patent/CN114141339B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111798416A (en) * | 2019-06-20 | 2020-10-20 | 太原理工大学 | Intelligent glomerulus detection method and system based on pathological image and deep learning |
| US10482603B1 (en) * | 2019-06-25 | 2019-11-19 | Artificial Intelligence, Ltd. | Medical image segmentation using an integrated edge guidance module and object segmentation network |
| CN112633416A (en) * | 2021-01-16 | 2021-04-09 | 北京工业大学 | Brain CT image classification method fusing multi-scale superpixels |
| CN112785617A (en) * | 2021-02-23 | 2021-05-11 | 青岛科技大学 | Automatic segmentation method for residual UNet rectal cancer tumor magnetic resonance image |
| CN113160226A (en) * | 2021-05-24 | 2021-07-23 | 苏州大学 | Two-way guide network-based classification segmentation method and system for AMD lesion OCT image |
| CN113763340A (en) * | 2021-08-25 | 2021-12-07 | 中国人民解放军空军军医大学 | Automatic grading method based on multitask deep learning ankylosing spondylitis |
Non-Patent Citations (3)
| Title |
|---|
| A Mask R-CNN Model with Improved Region Proposal Network for Medical Ultrasound Image;Liu, J et al.;《14th International Conference on Intelligent Computing (ICIC)》;20180818;第26-33页 * |
| 基于深度学习的细胞及组织病理辅助诊断系统设计与实现;许高源;《万方学位论文数据库》;20220121;全文 * |
| 大气NO2长期暴露与中国成人慢性肾脏病的关联：基于中国慢性肾脏病流行病学调查;梁琛瑜等;《环境与职业医学》;20210630;第38卷(第6期);第566-572页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114141339A (en) | 2022-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jain et al. | Synthetic data augmentation for surface defect detection and classification using deep learning | |
| CN111612008B (en) | Image segmentation method based on convolution network | |
| CN110674866A (en) | Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network | |
| CN108345827B (en) | Method, system and neural network for identifying document direction | |
| CN115223063B (en) | Deep learning-based unmanned aerial vehicle remote sensing wheat new variety lodging area extraction method and system | |
| CN114897779A (en) | Cervical cytology image abnormal area positioning method and device based on fusion attention | |
| CN114266794B (en) | Pathological section image cancer region segmentation system based on full convolution neural network | |
| CN110930378B (en) | Emphysema image processing method and system based on low data demand | |
| CN114758137B (en) | Ultrasonic image segmentation method and device and computer readable storage medium | |
| CN111401421A (en) | Image category determination method based on deep learning, electronic device, and medium | |
| CN115565071A (en) | Hyperspectral image transform network training and classifying method | |
| Luo et al. | CAT-EDNet: Cross-attention transformer-based encoder–decoder network for salient defect detection of strip steel surface | |
| CN114141339B (en) | Pathological image classification method, device, equipment and storage medium for membranous nephropathy | |
| Chaudhary et al. | Satellite imagery analysis for road segmentation using U-Net architecture | |
| CN111210398A (en) | White blood cell recognition system based on multi-scale pooling | |
| CN112597997A (en) | Region-of-interest determining method, image content identifying method and device | |
| CN117274355A (en) | Drainage pipeline flow intelligent measurement method based on acceleration guidance area convolutional neural network and parallel multi-scale unified network | |
| Guo et al. | Udtiri: An open-source road pothole detection benchmark suite | |
| Dixit et al. | Automatic building extraction from high-resolution satellite images using deep learning techniques | |
| CN111340139A (en) | Method and device for judging complexity of image content | |
| CN116189130A (en) | Lane line segmentation method and device based on image annotation model | |
| CN113989671A (en) | Remote sensing scene classification method and system based on semantic perception and dynamic graph convolution | |
| CN113377985A (en) | Pyramid network-based traditional Chinese medicine image classification and retrieval method | |
| Benkhoui et al. | Effective pavement crack delineation using a cascaded dilation module and fully convolutional networks | |
| Bäuerle et al. | Training de-confusion: an interactive, network-supported visual analysis system for resolving errors in image classification training data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |