CN112991295B - Lymph node metastasis image analysis system, method and equipment based on deep learning - Google Patents

Lymph node metastasis image analysis system, method and equipment based on deep learning Download PDF

Info

Publication number
CN112991295B
CN112991295B CN202110272069.XA CN202110272069A CN112991295B CN 112991295 B CN112991295 B CN 112991295B CN 202110272069 A CN202110272069 A CN 202110272069A CN 112991295 B CN112991295 B CN 112991295B
Authority
CN
China
Prior art keywords
image
lymph node
node metastasis
deep learning
clinical information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110272069.XA
Other languages
Chinese (zh)
Other versions
CN112991295A (en
Inventor
田捷
董迪
李海林
胡振华
王思雯
胡朝恩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Automation of Chinese Academy of Science
Original Assignee
Institute of Automation of Chinese Academy of Science
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Automation of Chinese Academy of Science filed Critical Institute of Automation of Chinese Academy of Science
Priority to CN202110272069.XA priority Critical patent/CN112991295B/en
Publication of CN112991295A publication Critical patent/CN112991295A/en
Application granted granted Critical
Publication of CN112991295B publication Critical patent/CN112991295B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Multimedia (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

The invention belongs to the field of image analysis, particularly relates to a lymph node metastasis image analysis system, method and equipment based on deep learning, and aims to solve the problem that the prior art cannot well carry out non-invasive prediction on whether lymph node metastasis occurs or not aiming at a non-small cell cancer image. The invention comprises the following steps: the method comprises the steps of obtaining a CT image to be analyzed containing a focus microenvironment, obtaining the same tested imaging signs and clinical information as the CT image, respectively extracting one-dimensional CT image characteristics and one-dimensional clinical information characteristics, performing characteristic enhancement and normalization processing, fusing through a full-connection layer to generate fusion characteristic vectors, and classifying the fusion characteristic vectors to obtain an analysis result. The method realizes classification of the non-small cell lung cancer lymph node metastasis image data, has better robustness and generalization capability compared with the traditional vector model based on predefined image characteristics, and effectively improves the prognosis effect of patients.

Description

Lymph node metastasis image analysis system, method and equipment based on deep learning
Technical Field
The invention belongs to the field of image analysis, and particularly relates to a lymph node metastasis image analysis system, method and device based on deep learning.
Background
Lung cancer is one of the most rapidly growing malignant tumors, and the incidence rate of lung cancer varies greatly according to sex, age, race, geographical location and the like, and the main subtypes of lung cancer include squamous cell carcinoma, large cell carcinoma, lung adenocarcinoma and the like.
For the patients with early confirmed non-small cell lung cancer, if accurate preoperative classification can be obtained, the postoperative life is long and the life quality is high under normal conditions. Particularly for lymph node metastasis of early non-small cell lung cancer, effective characteristics of the lymph node metastasis are difficult to find out through naked eyes by means of imaging methods such as CT at present, so that the invention provides a method for integrating image information and clinical case history information of CT images through a deep learning technology so as to complete classification of lymph node metastasis image data.
Patent (cn201110356082. X) proposes a method for judging lymph node metastasis of gastric cancer based on predefined image characteristics. According to the method, a traditional vector model is constructed by segmenting a focus and extracting predefined textural features and gray features, and the traditional model based on the predefined image features is insufficient in generalization capability and low in model robustness. The deep learning model is generally superior to the traditional vector model based on the predefined image characteristics in generalization performance and robustness. The lymph node metastasis image data classification model based on the deep learning technology provided by the invention passes verification on an external data set in the early stage, the judgment accuracy and sensitivity of the lymph node metastasis image data classification model reach higher levels, and the generalization capability of the lymph node metastasis image data classification model is verified.
The background information of the CT image of the patient with the non-small cell lung cancer is complex, and the focus boundary is fuzzy and difficult to accurately position. In addition, a large amount of quantitative clinical indexes are still not fully utilized clinically. The traditional image analysis method relies on the accurate delineation of a tumor focus area, is difficult to aim at the characteristics, trains and predicts the focus image containing the focus microenvironment at the CT slice level, and integrates at the patient level. The traditional deep learning model can not enhance and fuse the clinical indexes and the imaging signs of the patient with the CT image data in the characteristic level in the model training process.
The method combines deep learning and medical big data, provides an effective lymph node metastasis analysis and classification system for related researchers, can perform characteristic learning on focus and microenvironment information of the focus aiming at the non-small cell lung cancer, and performs characteristic fusion of three input data, namely a non-small cell lung cancer CT image, clinical indexes and imaging signs at a characteristic level, thereby completing non-invasive prediction on whether lymph node metastasis occurs and providing specific lymph node metastasis probability.
Disclosure of Invention
In order to solve the above problems in the prior art, that is, the prior art cannot well perform non-invasive prediction on whether lymph node metastasis occurs or not for a non-small cell cancer image, the present invention provides a lymph node metastasis image analysis system based on deep learning by extracting a lesion and a microenvironment region thereof, integrating CT image data, patient clinical indicators, and imaging symptoms at a characteristic level, performing enhancement and fusion on the lesion and the microenvironment region, performing unified training, and outputting an integrated model at a patient level, the system including: the device comprises a CT image acquisition unit, a clinical information acquisition unit and a classification result output unit;
the CT image acquisition unit is configured to acquire a CT image to be analyzed containing a focus microenvironment; the CT image to be analyzed comprises n i Labeling a CT image slice to be analyzed with a focus;
the clinical information acquisition unit is configured to acquire the same imaging signs and clinical information of the CT image to be tested;
the classification result output unit is configured to obtain an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the classification result output unit specifically includes: a CT image feature extraction subunit, a clinical information feature extraction subunit, a feature fusion subunit and an analysis result integration subunit;
the CT image feature extraction subunit is configured to obtain one-dimensional CT image features through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning based on the CT image slice to be analyzed; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
the clinical information feature extraction subunit is configured to perform feature amplification through a branch 2 of the trained deep learning-based lymph node metastasis image analysis model to obtain one-dimensional clinical information features; the branch 2 comprises a feature amplification module and a full connection layer;
the feature fusion subunit is configured to perform feature splicing through a full connection Layer based on one-dimensional CT image features and one-dimensional clinical information features, perform Normalization processing through Layer Normalization, and perform feature fusion through the full connection Layer to generate fusion feature vectors;
and the analysis result integration subunit is configured to classify the fusion feature vectors to obtain analysis results, and integrate the analysis results of the same test subject by any one of a simple voting method, a highest confidence method or an average method to generate integrated analysis results.
In some preferred embodiments, the assay result is integrated into a subunit:
the simple voting method comprises the following steps:
Figure BDA0002974660640000041
the highest confidence method is as follows:
Figure BDA0002974660640000042
the averaging method comprises the following steps:
Figure BDA0002974660640000043
wherein, P i Shows the results of the analysis of the i-th patient, 0 TableNegative, 1 positive; n is i Representing the total number of slices of the ith patient; s i，j Represents the probability that the ith patient's j Zhang Qiepian is positive;
Figure BDA0002974660640000044
sgn denotes a step function; argmax represents the maximum function.
In some preferred embodiments, the system further includes a transition probability analysis unit configured to, after obtaining the integration analysis result, obtain a transition probability of the subject by a transition probability calculation method corresponding to an integration method in the analysis result integration subunit;
if the result integration is performed in the analysis result integration subunit by adopting a simple voting method, the transition probability calculation method is as follows:
Figure BDA0002974660640000045
wherein the content of the first and second substances,
Figure BDA0002974660640000046
s i，j represents the probability of positivity of the jth section of the ith patient, p i Expressing the lymph node metastasis probability of the ith patient, and the card expressing the card function;
if the result integration is performed by adopting a highest confidence method in the analysis result integration subunit, the transition probability calculation method is as follows:
Figure BDA0002974660640000051
wherein n is i Representing the total number of patient slices;
if the result integration is performed in the analysis result integration subunit by adopting an averaging method, the transition probability calculation method is as follows:
Figure BDA0002974660640000052
in some preferred embodiments, the CT image feature extraction subunit specifically functions to include:
and adjusting the CT image slices according to the preset window width and window position based on the CT image slices, extracting a rectangular region circumscribed by a focus microenvironment on each slice according to the preset multiple of the focus area, inputting the trained branch 1 of the lymph node metastasis image analysis model based on the deep learning, and obtaining the one-dimensional CT image characteristics.
In some preferred embodiments, the deep learning-based lymph node metastasis image analysis model is trained by a method comprising:
step B100, executing the functions of the CT image acquisition unit and the clinical information acquisition unit to acquire a CT image training set, an iconography symptom training set and a clinical information training set;
step B200, executing the CT image feature extraction subunit function to obtain one-dimensional CT image training set features;
step B300, gaussian noise is added into the clinical information training set and the imaging sign training set, and the function of the clinical information feature extraction subunit is executed to generate a one-dimensional clinical information feature training set;
step B400, executing the functions of the feature fusion subunit and the analysis result integration subunit to generate an integration analysis result, and calculating a loss function;
and B500, repeating the steps B100-B400, and adjusting model parameters through a random gradient algorithm to obtain the trained lymph node metastasis image analysis model based on deep learning.
In some preferred embodiments, in each training round, different random sampling weights are set for each sample according to the class ratio, so that the number of samples in different classes in each training is basically consistent.
In some preferred embodiments, the present invention may classify the fused feature vector by a softmax function.
In another aspect of the present invention, a lymph node metastasis image analysis method based on deep learning is provided, the method includes:
step S100, acquiring a CT image to be analyzed containing a focus microenvironment; the CT image to be analyzed comprises n i Labeling a CT image slice to be analyzed with a focus;
step S200, acquiring the same tested imaging signs and clinical information as the CT image;
step S300, acquiring an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the specific steps of step S300 are:
step S310, based on the CT image slice to be analyzed, acquiring one-dimensional CT image characteristics through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
step S320, performing feature amplification through a branch 2 of the trained lymph node metastasis image analysis model based on deep learning to obtain one-dimensional clinical information features; the branch 2 comprises a feature amplification module and a full connection layer;
step S330, based on the one-dimensional CT image characteristics and the one-dimensional clinical information characteristics, performing characteristic splicing through a full connection Layer, performing Normalization processing through Layer Normalization, and performing characteristic fusion through the full connection Layer to generate a fusion characteristic vector;
and step S340, classifying the fusion feature vectors to obtain analysis results, and integrating the results of the same tested analysis results through any one of a simple voting method, a highest confidence method or an average method to generate integrated analysis results.
In a third aspect of the present invention, an electronic device is provided, including: at least one processor; and a memory communicatively coupled to at least one of the processors; wherein the memory stores instructions executable by the processor for execution by the processor to implement the deep learning-based lymph node metastasis image analysis method described above.
In a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions for being executed by the computer to implement the above-mentioned lymph node metastasis image analysis method based on deep learning.
The invention has the beneficial effects that:
(1) The lymph node metastasis image analysis system based on deep learning realizes classification of non-small cell lung cancer lymph node metastasis image data by combining CT images, imaging signs and clinical information for analysis, has better robustness and generalization capability compared with the existing vector model based on predefined image characteristics, and effectively improves the prognosis effect of patients.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1 is a block diagram of a lymph node metastasis image analysis system based on deep learning according to an embodiment of the present invention;
FIG. 2 is a diagram of deep learning-based implementation of the present invention a network structure block diagram of a lymph node metastasis image analysis model;
FIG. 3 is a ROC curve of a lymph node metastasis image analysis model based on deep learning on a training set and corresponding AUC values;
FIG. 4 is a ROC curve and its corresponding AUC values on an external validation set of a lymph node metastasis image analysis model based on deep learning;
FIG. 5 is a diagram illustrating rectangular region extraction according to an embodiment of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention provides a lymph node metastasis image analysis system based on deep learning, which comprises: the device comprises a CT image acquisition unit, a clinical information acquisition unit and a classification result output unit;
the CT image acquisition unit is configured to acquire a CT image to be analyzed containing a focus microenvironment;
the clinical information acquisition unit is configured to acquire the same imaging signs and clinical information of the CT image to be tested;
the classification result output unit is configured to obtain an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information.
According to the lymph node metastasis image analysis system based on deep learning, the classification of the lymph node metastasis image data of the non-small cell lung cancer is realized by combining CT images, imaging signs and clinical information for analysis, and compared with the existing vector model based on predefined image features, the lymph node metastasis image analysis system based on deep learning has better robustness and generalization capability, and the prognosis effect of a patient is effectively improved.
The background information of the CT image of the patient with the non-small cell lung cancer is complex, and the focus boundary is fuzzy and difficult to accurately position. In addition, a large amount of quantitative clinical indexes are still not fully utilized clinically. The traditional image analysis method relies on the accurate delineation of a tumor focus area, is difficult to aim at the characteristics, trains and predicts the focus image containing the focus microenvironment at the CT slice level, and integrates at the patient level. The traditional deep learning model can not enhance and fuse the clinical indexes and imaging signs of the patient with the CT image data in the characteristic level in the model training process;
the method combines deep learning and medical big data, provides an effective lymph node metastasis analysis and classification system for related researchers, can perform characteristic learning on focus and microenvironment information of the focus aiming at the non-small cell lung cancer, and performs characteristic fusion of three input data, namely a non-small cell lung cancer CT image, clinical indexes and imaging signs at a characteristic level, thereby completing non-invasive prediction on whether lymph node metastasis occurs and providing specific lymph node metastasis probability.
In order to more clearly describe the lymph node metastasis image analysis system based on deep learning of the present invention, the functional units in the embodiment of the present invention are described in detail below with reference to fig. 1.
The lymph node metastasis image analysis system based on deep learning according to the first embodiment of the present invention includes a CT image acquisition unit, a clinical information acquisition unit, and a classification result output unit, and each functional unit is described in detail as follows:
the CT image acquisition unit is configured to acquire a CT image to be analyzed containing a focus microenvironment;
the clinical information acquisition unit is configured to acquire the same imaging signs and clinical information of the CT image to be tested;
the classification result output unit is configured to obtain an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the classification result output unit specifically includes: the CT image feature extraction subunit, the clinical information feature extraction subunit, the feature fusion subunit and the analysis result integration subunit are connected in series;
the CT image feature extraction subunit is configured to obtain one-dimensional CT image features through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning based on the CT image slice to be analyzed; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
in this embodiment, the CT image feature extraction subunit specifically has functions of:
as shown in the attached drawings2. N based on the CT image i And (4) slicing the focus area, and manually segmenting the focus area slice by slice to obtain the manual marking of the focus on each CT slice. Adjusting the CT image slices according to a preset window width and window level based on the CT image slices, extracting a rectangular region circumscribed by a focus microenvironment on each slice according to a preset multiple of a focus area, inputting a branch 1 of a trained deep learning-based lymph node metastasis image analysis model, and acquiring one-dimensional CT image characteristics. And taking each rectangular area as an independent input of the deep learning model in the model training process.
In this embodiment, preoperative CT image data of 1020 cases of patients with non-small cell lung cancer in IA stage are collected, and the data are manually sketched layer by an imaging physician, so as to delineate a three-dimensional lesion region of non-small cell lung cancer (i.e., sketched on slices with all lesions visible).
The CT images are adjusted by using 2048 and 0 window width window levels, respectively, and each layer is expanded by 3 times the lesion area for lesion microenvironment information to generate a rectangular region, as shown in fig. 5, which is a rectangular region frame of a single CT slice. The gray value of the rectangular area is adjusted to be between 0 and 255 and is stored as a gray image. With 64 × 64 pixels as a threshold, the picture with the undersized size is deleted, and finally the generated rectangular region picture is used as a final region of interest (ROI), and is used as an input of the model. In this example, a total of 13073 ROI pictures were obtained.
Dividing all ROI pictures into a training set, a verification set and a test set, wherein the training set is used for model training, the verification set is used for determining model hyper-parameters, and the test set is only used for model performance verification. ROI pictures belonging to the same patient only appear in the same set.
The clinical information characteristic extraction subunit is configured to perform characteristic amplification through a branch 2 of the trained deep learning-based lymph node metastasis image analysis model to obtain one-dimensional clinical information characteristics; the branch 2 comprises a feature amplification module and a full connection layer; in this embodiment, the purpose of the augmentation is to prevent the loss of information after fusion of low-dimensional clinical indicators and imaging signs with image features.
The feature fusion subunit is configured to perform feature splicing through a full connection Layer based on one-dimensional CT image features and one-dimensional clinical information features, perform Normalization processing through Layer Normalization, and perform feature fusion through the full connection Layer to generate a fusion feature vector, as shown in fig. 2;
and the analysis result integration subunit is configured to classify the fusion feature vectors to obtain analysis results, and integrate the analysis results of the same test subject by any one of a simple voting method, a highest confidence method or an average method to generate integrated analysis results.
In the present embodiment, the fused feature vectors are classified by a softmax function.
In this embodiment, the analysis result is integrated into a subunit:
the simple voting method comprises the following steps:
Figure BDA0002974660640000121
the highest confidence method is as follows:
Figure BDA0002974660640000122
the averaging method comprises the following steps:
Figure BDA0002974660640000123
wherein, P i Indicates the analysis result of the ith patient, 0 indicates negative, and 1 indicates positive; n is i Representing the total number of slices for the ith patient; s i，j Represents the probability that the ith patient's j Zhang Qiepian is positive;
Figure BDA0002974660640000124
sgn denotes a step function; argmax represents the maximum function.
In the present embodiment, after obtaining the integration analysis result, the transition probability of the test is obtained by the transition probability calculation method corresponding to the integration method in the analysis result integration subunit;
if the result integration is performed in the analysis result integration subunit by adopting a simple voting method, the transition probability calculation method is as follows:
Figure BDA0002974660640000125
wherein the content of the first and second substances,
Figure BDA0002974660640000126
s i，j represents the probability of positivity of the jth section of the ith patient, p i Expressing the lymph node metastasis probability of the ith patient, and the card expressing the card function;
if the result integration is performed by adopting a highest confidence method in the analysis result integration subunit, the transition probability calculation method is as follows:
Figure BDA0002974660640000131
wherein n is i Representing the total number of patient slices;
if the result integration is performed in the analysis result integration subunit by adopting an averaging method, the transition probability calculation method is as follows:
Figure BDA0002974660640000132
in this embodiment, the training method of the lymph node metastasis image analysis model based on deep learning includes:
step B100, executing the functions of the CT image acquisition unit and the clinical information acquisition unit to acquire a CT image training set, an iconography symptom training set and a clinical information training set;
step B200, executing the CT image feature extraction subunit function to obtain one-dimensional CT image training set features;
step B300, gaussian noise is added into the clinical information training set and the imaging sign training set, and the function of the clinical information feature extraction subunit is executed to generate a one-dimensional clinical information feature set; the robustness of the model can be effectively improved by adding Gaussian noise;
step B400, executing the functions of the feature fusion subunit and the analysis result integration subunit to generate an integration analysis result, and calculating a loss function;
and B500, repeating the steps B100-B400, and adjusting model parameters through a random gradient algorithm to obtain the trained lymph node metastasis image analysis model based on deep learning.
In this embodiment, a class random sampling method is further included, and in each training round, different random sampling weights are set for each sample according to class comparison, so that the number of samples of different classes in each training is substantially the same. Specifically, taking the example that the number ratio of samples with lymph node metastasis and samples without lymph node metastasis is 1:3, the weight ratio of randomly sampling samples with lymph node metastasis and samples without lymph node metastasis is 3:1, so that the balance of the number of the two types of samples in each round is ensured, the sensitivity and specificity of the model are further balanced, and the bias of model results is prevented.
Since the network input is the CT image slice of the patient, it is necessary to integrate the classification result of the same patient slice level into the patient level, and in this embodiment, it is converted into the classification result of the patient level by using a simple voting method, and at the same time, the lymph node transition probability of the patient level is output, and the algorithm is shown in the formula of the "summary of the invention" section.
In this embodiment, the method further includes verifying the system, verifying and evaluating the classification performance of the model by using an ROC curve (receiver operating characteristic curve), and after the model is constructed, drawing the ROC curve through the output of the internal verification set to determine the accuracy and robustness of the model. Specifically, the invention adopts four specific evaluation indexes to quantify the performance of the model, namely: area under the ROC curve (area under the ROC curve), accuracy (accuracy), sensitivity (sensitivity), specificity (specificity). The ROC curve of the model during validation is shown in fig. 3. If the accuracy and robustness of the model on the internal verification set are low, the hyper-parameters of the model training, such as learning rate (learning rate), batch size (batch size) and the like, are adjusted for retraining.
In the present embodiment, the result of the slice (ROI) level is as follows:
the accuracy, sensitivity and specificity in the training set are respectively as follows: 0.9102,0.9486,0.8351; the internal verification is concentrated to 0.8197,0.8346,0.8273; the external test is focused on 0.7823,0.9102,0.7706.
In patient-level validation, AUC and sensitivity in training set and validation set reached 0.9577 and 0.9728, with ROC curves as shown in fig. 3 of the following graph; AUC and sensitivity in the external test set reach 0.9056 and 0.9394, and ROC curves are shown in FIG. 4.
The lymph node metastasis image analysis method based on deep learning according to the second embodiment of the present invention includes steps S100 to S300, which are detailed as follows:
step S100, acquiring a CT image to be analyzed containing a focus microenvironment;
step S200, acquiring the same tested imaging signs and clinical information as the CT image;
step S300, acquiring an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the specific steps of step S300 are:
step S310, based on the CT image slice to be analyzed, acquiring one-dimensional CT image characteristics through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
step S320, performing feature amplification through a branch 2 of the trained lymph node metastasis image analysis model based on deep learning to obtain one-dimensional clinical information features; the branch 2 comprises a feature amplification module and a full connection layer;
step S330, based on the one-dimensional CT image characteristics and the one-dimensional clinical information characteristics, performing characteristic splicing through a full connection Layer, performing Normalization processing through Layer Normalization, and performing characteristic fusion through the full connection Layer to generate a fusion characteristic vector;
and step S340, classifying the fusion characteristics to obtain an analysis result, and integrating the results of the same tested analysis result by any one of a simple voting method, a highest confidence method or an average method to generate an integrated analysis result.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process and related description of the system described above may refer to the corresponding process in the foregoing method embodiments, and will not be described herein again.
It should be noted that, the lymph node metastasis image analysis system based on deep learning provided in the above embodiment is only exemplified by the division of the above functional modules, and in practical applications, the above functions may be allocated to different functional modules according to needs, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the above embodiment may be combined into one module, or may be further split into a plurality of sub-modules, so as to complete all or part of the above described functions. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not to be construed as unduly limiting the present invention.
An electronic device according to a third embodiment of the present invention is characterized by including: at least one processor; and a memory communicatively coupled to at least one of the processors; wherein the memory stores instructions executable by the processor for execution by the processor to implement the deep learning-based lymph node metastasis image analysis method described above. .
A computer-readable storage medium according to a fourth embodiment of the present invention is characterized in that the computer-readable storage medium stores computer instructions for being executed by the computer to implement the lymph node metastasis image analysis method based on deep learning of claim 8.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage device and the processing device described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The terms "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing or implying a particular order or sequence.
The terms "comprises," "comprising," or any other similar term are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.

Claims (8)

1. A lymph node metastasis image analysis system based on deep learning, the system comprising: the device comprises a CT image acquisition unit, a clinical information acquisition unit and a classification result output unit;
the CT image acquisition unit is configured to acquire a CT image to be analyzed containing a focus microenvironment; the CT image to be analyzed comprises n i Labeling a CT image slice to be analyzed with a focus;
the clinical information acquisition unit is configured to acquire the same imaging signs and clinical information of the CT image to be tested;
the classification result output unit is configured to obtain an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the classification result output unit specifically includes: the CT image feature extraction subunit, the clinical information feature extraction subunit, the feature fusion subunit and the analysis result integration subunit are connected in series;
the CT image feature extraction subunit is configured to obtain one-dimensional CT image features through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning based on the CT image slice to be analyzed; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
the clinical information feature extraction subunit is configured to perform feature amplification through a branch 2 of the trained deep learning-based lymph node metastasis image analysis model to obtain one-dimensional clinical information features; the branch 2 comprises a feature amplification module and a full connection layer;
the feature fusion subunit is configured to perform feature splicing through a full connection Layer based on one-dimensional CT image features and one-dimensional clinical information features, perform Normalization processing through Layer Normalization, and perform feature fusion through the full connection Layer to generate fusion feature vectors;
the analysis result integration subunit is configured to classify the fusion feature vectors to obtain analysis results, and integrate the analysis results of the same test subject by any one of a simple voting method, a highest confidence method or an average method to generate integrated analysis results;
the analysis result is integrated into a subunit:
the simple voting method comprises the following steps:
Figure FDA0003967333430000021
the highest confidence method is as follows:
Figure FDA0003967333430000022
the averaging method comprises the following steps:
Figure FDA0003967333430000023
wherein, P i Indicates the analysis result of the ith patient, 0 indicates negative, and 1 indicates positive; n is a radical of an alkyl radical i Representing the total number of slices for the ith patient; s i，j Represents the probability that the ith patient's j Zhang Qiepian is positive;
Figure FDA0003967333430000024
sgn denotes a step function; argmax represents the function of maximum;
the system also comprises a transition probability analysis unit which is configured to obtain the transition probability of the tested object through a transition probability calculation method corresponding to the integration method in the integration subunit of the analysis result after obtaining the integration analysis result;
if the result integration is performed in the analysis result integration subunit by adopting a simple voting method, the transition probability calculation method is as follows:
Figure FDA0003967333430000025
wherein the content of the first and second substances,
Figure FDA0003967333430000026
s i，j represents the probability of positivity of the jth section of the ith patient, p i Expressing the lymph node metastasis probability of the ith patient, and the card expressing the card function;
if the result integration is performed by adopting a highest confidence method in the analysis result integration subunit, the transition probability calculation method is as follows:
Figure FDA0003967333430000031
wherein n is i Representing the total number of patient slices;
if the result integration is performed in the analysis result integration subunit by adopting an averaging method, the transition probability calculation method is as follows:
Figure FDA0003967333430000032
2. the lymph node metastasis image analysis system based on deep learning of claim 1, wherein the CT image feature extraction subunit specifically functions as:
and adjusting the CT image slices at a preset window width and window level based on the CT image slices, extracting a rectangular region externally connected with a focus microenvironment on each slice according to a preset multiple of the focus area, and inputting a branch 1 of a trained lymph node metastasis image analysis model based on deep learning to obtain one-dimensional CT image characteristics.
3. The deep learning-based lymph node metastasis image analysis system according to claim 1, wherein the deep learning-based lymph node metastasis image analysis model is trained by a method comprising:
step B100, executing the functions of the CT image acquisition unit and the clinical information acquisition unit to acquire a CT image training set, an iconography symptom training set and a clinical information training set;
step B200, executing the CT image feature extraction subunit function to obtain one-dimensional CT image training set features;
step B300, gaussian noise is added into the clinical information training set and the imaging sign training set, and the function of the clinical information feature extraction subunit is executed to generate a one-dimensional clinical information feature training set;
step B400, executing the functions of the feature fusion subunit and the analysis result integration subunit to generate an integration analysis result, and calculating a loss function;
and B500, repeating the steps B100-B400, and adjusting model parameters through a random gradient algorithm to obtain the trained lymph node metastasis image analysis model based on deep learning.
4. The lymph node metastasis image analysis system based on deep learning of claim 3, wherein in each training round, different random sampling weights are set for each sample according to class ratio, so that the number of samples of different classes in each training is basically the same.
5. The deep learning-based lymph node metastasis image analysis system according to claim 1, wherein the fusion feature vectors are classified by a softmax function.
6. A lymph node metastasis image analysis method based on deep learning is characterized by comprising the following steps:
step S100, acquiring a CT image to be analyzed containing a focus microenvironment;
step S200, acquiring the same tested imaging signs and clinical information as the CT image;
step S300, acquiring an integrated classification result through a trained lymph node metastasis image analysis model based on deep learning based on the CT image, the imaging symptom and the clinical information;
the specific steps of step S300 are:
step S310, based on the CT image slice to be analyzed, acquiring one-dimensional CT image characteristics through a branch 1 of a trained lymph node metastasis image analysis model based on deep learning; the branch 1 comprises a bottleeck bottleneck layer, a residual module and a batch-norm batch normalization module;
step S320, performing feature amplification through a branch 2 of the trained lymph node metastasis image analysis model based on deep learning to obtain one-dimensional clinical information features; the branch 2 comprises a feature amplification module and a full connection layer;
step S330, based on the one-dimensional CT image characteristics and the one-dimensional clinical information characteristics, performing characteristic splicing through a full connection Layer, performing Normalization processing through Layer Normalization, and performing characteristic fusion through the full connection Layer to generate a fusion characteristic vector;
step S340, classifying based on the fusion characteristic vector to obtain an analysis result, and integrating the results of the same tested analysis result by any one of a simple voting method, a highest confidence method or an average method to generate an integrated analysis result;
in the step S340:
the simple voting method comprises the following steps:
Figure FDA0003967333430000051
the highest confidence method is as follows:
Figure FDA0003967333430000052
the averaging method comprises the following steps:
Figure FDA0003967333430000053
wherein, P i Indicates the analysis result of the ith patient, 0 indicates negative, and 1 indicates positive; n is i Representing the total number of slices of the ith patient; s i，j Represents the probability that the j Zhang Qiepian of the ith patient is positive;
Figure FDA0003967333430000054
sgn denotes a step function; argmax represents the function of maximum;
a step S350 configured to acquire a transition probability of the subject by a transition probability calculation method corresponding to the integration method in the step S340 after acquiring the integration analysis result;
if the result integration is performed by using a simple voting method in step S340, the transition probability calculation method is:
Figure FDA0003967333430000061
wherein the content of the first and second substances,
Figure FDA0003967333430000062
s i，j represents the probability of positivity of the jth section of the ith patient, p i Expressing the lymph node metastasis probability of the ith patient, and the card expressing the card function;
if the highest confidence method is adopted for result integration in step S340, the transition probability calculation method is:
Figure FDA0003967333430000063
wherein n is i Representing the total number of patient slices;
if the result integration is performed by using the averaging method in step S340, the transition probability calculation method is:
Figure FDA0003967333430000064
7. an electronic device, comprising: at least one processor; and a memory communicatively coupled to at least one of the processors; wherein the memory stores instructions executable by the processor for execution by the processor to implement the deep learning based lymph node metastasis image analysis method of claim 6.
8. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions for execution by the computer to implement the deep learning-based lymph node metastasis image analysis method according to claim 6.
CN202110272069.XA 2021-03-12 2021-03-12 Lymph node metastasis image analysis system, method and equipment based on deep learning Active CN112991295B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110272069.XA CN112991295B (en) 2021-03-12 2021-03-12 Lymph node metastasis image analysis system, method and equipment based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110272069.XA CN112991295B (en) 2021-03-12 2021-03-12 Lymph node metastasis image analysis system, method and equipment based on deep learning

Publications (2)

Publication Number Publication Date
CN112991295A CN112991295A (en) 2021-06-18
CN112991295B true CN112991295B (en) 2023-04-07

Family

ID=76335388

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110272069.XA Active CN112991295B (en) 2021-03-12 2021-03-12 Lymph node metastasis image analysis system, method and equipment based on deep learning

Country Status (1)

Country Link
CN (1) CN112991295B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113850753B (en) * 2021-08-17 2023-09-01 苏州鸿熙融合智能医疗科技有限公司 Medical image information computing method, device, edge computing equipment and storage medium
CN113724876A (en) * 2021-09-10 2021-11-30 南昌大学第二附属医院 Intra-stroke hospital complication prediction model based on multi-mode fusion and DFS-LLE algorithm
CN114187467B (en) * 2021-11-11 2023-04-28 电子科技大学 Method and device for classifying benign and malignant lung nodules based on CNN model
CN116129200A (en) * 2023-04-17 2023-05-16 厦门大学 Bronchoscope image benign and malignant focus classification device based on deep learning
CN117689985A (en) * 2023-12-08 2024-03-12 首都医科大学附属北京妇产医院 Uterine smooth muscle tumor recognition system and method based on ensemble voting mechanism

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111915596A (en) * 2020-08-07 2020-11-10 杭州深睿博联科技有限公司 Method and device for predicting benign and malignant pulmonary nodules

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102436551B (en) * 2011-11-10 2014-09-17 西安电子科技大学 Computer aided gastric cancer diagnosis method based on target tracking
US10441225B2 (en) * 2018-02-21 2019-10-15 Case Western Reserve University Predicting disease recurrence following trimodality therapy in non-small cell lung cancer using computed tomography derived radiomic features and clinico-pathologic features
CN110246109B (en) * 2019-05-15 2022-03-18 清华大学 Analysis system, method, device and medium fusing CT image and personalized information
CN110910371A (en) * 2019-11-22 2020-03-24 北京理工大学 Liver tumor automatic classification method and device based on physiological indexes and image fusion
CN111242174B (en) * 2019-12-31 2022-09-23 浙江大学 Liver cancer image feature extraction and pathological classification method based on imaging omics
CN112184658B (en) * 2020-09-24 2023-11-24 上海健康医学院 Method, medium and electronic equipment for prognosis survival prediction of non-small cell lung cancer

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111915596A (en) * 2020-08-07 2020-11-10 杭州深睿博联科技有限公司 Method and device for predicting benign and malignant pulmonary nodules

Also Published As

Publication number Publication date
CN112991295A (en) 2021-06-18

Similar Documents

Publication Publication Date Title
CN112991295B (en) Lymph node metastasis image analysis system, method and equipment based on deep learning
US11842556B2 (en) Image analysis method, apparatus, program, and learned deep learning algorithm
US20240078670A1 (en) Systems for cell shape estimation
CN105027165B (en) Tissue object-based machine learning system for automated scoring of digital whole slides
Zerhouni et al. Wide residual networks for mitosis detection
EP3721373A1 (en) Deep-learning systems and methods for joint cell and region classification in biological images
EP3721406A1 (en) Method of computing tumor spatial and inter-marker heterogeneity
JP2020502534A (en) Computer scoring based on primary staining and immunohistochemical imaging
Guo et al. Deep learning for assessing image focus for automated cervical cancer screening
Tania et al. Intelligent image-based colourimetric tests using machine learning framework for lateral flow assays
Zewdie et al. Classification of breast cancer types, sub-types and grade from histopathological images using deep learning technique
JP2023512560A (en) A federated learning system to train machine learning algorithms and maintain patient privacy
CN112700461B (en) System for pulmonary nodule detection and characterization class identification
CN113261012B (en) Method, device and system for processing image
Ström et al. Pathologist-level grading of prostate biopsies with artificial intelligence
CN112990214A (en) Medical image feature recognition prediction model
CN114445356A (en) Multi-resolution-based full-field pathological section image tumor rapid positioning method
Paeng et al. A unified framework for tumor proliferation score prediction in breast histopathology
CN113420793A (en) Improved convolutional neural network ResNeSt 50-based gastric ring cell carcinoma classification method
KR20200136004A (en) Method for detecting cells with at least one malformation in a cell sample
Tikkanen et al. Training based cell detection from bright-field microscope images
Sun et al. Detection of breast tumour tissue regions in histopathological images using convolutional neural networks
CN113222928B (en) Urine cytology artificial intelligence urothelial cancer identification system
Mishra et al. Assessing robustness of deep learning methods in dermatological workflow
CN117529750A (en) Digital synthesis of histological staining using multiple immunofluorescence imaging

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant