CN108537773B - Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases - Google Patents

Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases Download PDF

Info

Publication number
CN108537773B
CN108537773B CN201810141703.4A CN201810141703A CN108537773B CN 108537773 B CN108537773 B CN 108537773B CN 201810141703 A CN201810141703 A CN 201810141703A CN 108537773 B CN108537773 B CN 108537773B
Authority
CN
China
Prior art keywords
classification
image
fusion
pancreatic
basic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810141703.4A
Other languages
Chinese (zh)
Other versions
CN108537773A (en
Inventor
杨晓冬
程超
左长京
张玉全
刘兆邦
孙高峰
潘桂霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Institute of Biomedical Engineering and Technology of CAS
Original Assignee
Suzhou Institute of Biomedical Engineering and Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Institute of Biomedical Engineering and Technology of CAS filed Critical Suzhou Institute of Biomedical Engineering and Technology of CAS
Priority to CN201810141703.4A priority Critical patent/CN108537773B/en
Publication of CN108537773A publication Critical patent/CN108537773A/en
Application granted granted Critical
Publication of CN108537773B publication Critical patent/CN108537773B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Abstract

The invention discloses a method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases, which comprises the steps of reading and normalizing pancreatic medical image data to obtain a normalized image; denoising, registering and image fusing are carried out on the normalized image to obtain a multi-modal fused image; selecting an interested region from the image with clear pancreatic structure display, mapping the interested region to other images, and storing the interested region as a natural image format which can be identified by a subsequent classification network; extracting, classifying and fusing the characteristics of the multi-mode fusion image according to the selected region of interest, and establishing a basic classification network model aiming at the fused characteristics; identifying the classification result of each basic classification network to obtain a final classification identification result; the invention has strong universality, is suitable for clinical practice, and can also be used for scientific research in the fields of pancreatic cancer and pancreatitis.

Description

Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases
Technical Field
The invention relates to the technical field of intelligent auxiliary diagnosis, in particular to a method for intelligently and auxiliarily identifying pancreatic cancer and pancreatic inflammatory diseases.
Background
Pancreatic Cancer (PC) is a common malignant tumor of the digestive system, and among the malignant tumors in China, the incidence rate is 7 th, the mortality rate is 6 th, and the 3-year survival rate is less than 5%. Early symptoms of pancreatic cancer are often not obvious, and the early symptoms of pancreatic cancer are often advanced when abdominal pain, jaundice and obvious weight loss appear. In the diagnosis of pancreatic cancer, since the clinical manifestations are very similar to those of other pancreatic inflammatory diseases, such as Chronic Pancreatitis (CP), all have the manifestations of abdominal pain, dyspepsia, anorexia, nausea, vomiting, weight loss and obstructive jaundice, and the conventional imaging slice is more overlapped with other pancreatic inflammatory diseases, it is difficult to clearly diagnose pancreatic cancer before surgery, especially to accurately identify pancreatic cancer and pancreatic inflammatory diseases.
As is well known, the imaging examination plays a key role in diagnosing pancreatic lesions, but the imaging examination can only provide the most intuitive images, and the information identification is limited by the direct display effect of the examination and the self level and experience of imaging physicians, and often because of the resolving power of human eyes and human negligence, more information contained in the imaging pictures cannot be fully utilized, such as super-visualization underlying features having the distinguishing power for lesion tissues, the doctors must skip reading in the traditional reading mode. Therefore, there is a need for a high-level auxiliary technique (CAD technique) for processing multi-modal images by integrating various examination information to improve the detection rate of lesions such as tumors, calcification, inflammation, and fibrosis. The technology can identify the diagnosis information which can not be identified by human eyes, and the diagnosis information can be used as the second eyes of doctors, so that the accuracy of pancreatic cancer diagnosis is improved, and the diagnosis technology plays an increasingly important role in the pancreatic cancer diagnosis process.
Imaging examinations include multi-row computed tomography, magnetic resonance, ultrasound, Endoscopic Ultrasound (EUS), PET, etc., but these examination techniques have limitations such as: CT has poor sensitivity to small tumors and isopycnic lesions with diameters <2cm, and has disadvantages in differential diagnosis of pancreatic cancer and chronic pancreatitis, because phenomena such as the appearance of calcific foci, the expansion of pancreatic ducts, the appearance of local tumors, double duct signs, the blockage of pancreatic ducts, the infiltration of peripheral fat, and the obstruction of peripheral veins of pancreas can appear in both diseases; patients with metal foreign bodies such as vascular stents in the body cannot be examined by MRI, and the diagnostic value of MRI on pancreatic lesions is controversial; ultrasound is not good enough for retroperitoneal images of patients with more intestinal gas and obesity; EUS belongs to an invasive imaging device, causes discomfort to the patient, and performs unsatisfactorily in the differentiation of chronic pancreatitis from pancreatic cancer, particularly in patients with pancreatic cancer accompanied by chronic pancreatitis, where 22-36% of chronic pancreatitis is reported to be misdiagnosed as pancreatic cancer; PET is essentially a functional phenomenon, reflecting specific metabolic processes, but inflammatory foci, especially autoimmune chronic pancreatitis, also show high uptake of 18F FDG, similar to pancreatic cancer.
As described above, any of the above-mentioned inspection techniques cannot accurately determine pancreatic diseases, and therefore, the imaging omics-based intelligent assisted pancreatic cancer and pancreatitis identification system and method have high application value in clinical research. The invention aims to deeply excavate the super-visual bottom image information of medical images of various modes, and realize the classification and identification of pancreatic cancer and pancreatic inflammatory diseases through medical images according to the bottom image characteristics with the capability of distinguishing focuses.
At present, the intelligent auxiliary diagnosis of pancreatic diseases by using image processing technology at home and abroad mainly focuses on the following aspects:
in 2001, Norton I D et al proposed an autonomous learning artificial neural network to analyze EUS images and differentiate between malignancies and pancreatitis. In 2008, Das a and the like use image analysis software to perform texture analysis on pancreatic EUS images, and establish a neural network-based pancreatic cancer prediction model through Principal Component Analysis (PCA) dimension reduction. In 2013, Zhu M and the like extract texture features from a pancreatic EUS image region of interest by using an image processing technology, and then better combine the features by using the distance between a class algorithm and a sequential forward search algorithm (SFS), so that a Support Vector Machine (SVM) prediction model is established. Similar algorithms are proposed when Chua's philosophy equals 2008, class interval primary feature selection is used, a sequential forward search algorithm is used for further feature optimization, thereafter, Chua's philosophy and the like improve texture feature extraction, multiple fractal dimension features based on M-band wavelet transformation are selected, and a classification model established based on the method is superior to the previously proposed method in terms of running time and classification accuracy. The Wujijun Shuoshi combines the computer diagnosis result of fuzzy classification with the radioactive particle implantation therapy to expand the whole classification system, so that the pancreatic cancer and the non-cancer can be identified, and the pancreatic cancer and the pancreatitis can be further identified. In 2015, Zhu J et al introduced a new lesion descriptor, local ternary pattern variance, in order to improve the performance of classification models.
In 2016, Hanania et al used a gray level co-occurrence matrix to classify the malignancy of intraductal papillary mucinous tumors. In 2016, Chakraborty and the like predict the survival of pancreatic ductal adenocarcinoma patients who use new adjuvant chemotherapy by texture analysis based on enhanced CT images, and the patients extract 169 standard texture features including gray level co-occurrence matrix, run matrix, local binary pattern, fractal dimension, first-order statistical feature and the like from a focus area, and establish a prediction model based on a naive Bayes classification model. In 2017, Gazit and the like classify papillary mucinous tumors and pancreatic cystic tumors in ducts on the basis of enhanced CT images, manually design a new characteristic representing solid components in cysts, and establish a classification model based on an Ada-boost classification model by combining 255 standard texture characteristics.
In 1993, Du-Yih Tsai et al proposed a method for detecting subtle abnormalities based on CT pancreatic images. The method is a simple cascaded filter detection method, the first step introduces the square of logarithmic operation of gray levels to improve the edge of low gray levels, then the gray levels are transferred to a deleted fuzzy area, and the last step uses logarithmic operation to enhance the outline of details. 2013, a quantum genetic algorithm optimized support vector machine classification method for pancreatic cancer detection is provided based on a pancreas CT image by the Master Zhao, and the like.
The research shows that the existing intelligent auxiliary pancreatic disease identification system has the following defects: (1) the pancreas or focus area needs to be finely divided, so that doctors need to have deep professional background and rich clinical experience, time and labor are consumed, and a division error is difficult to avoid; (2) the characteristic extraction is carried out by adopting a manual design mode, the extracted characteristic representation and generalization capability are poor, and researchers are required to carry out deep research on the field of the problem to be solved so as to design the characteristic with better adaptability; (3) the above studies are all directed to single-modality images, and performance improvement possibly brought by other modality images is neglected.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an intelligent auxiliary identification method for pancreatic cancer and pancreatic inflammatory diseases based on imaging omics and deep learning.
The invention provides a method for intelligently and auxiliarily identifying pancreatic cancer and pancreatic inflammatory diseases, which comprises the following steps:
1) reading and normalizing the pancreatic medical image data to obtain a normalized image;
2) denoising, registering and image fusing are carried out on the normalized image to obtain a multi-modal fused image;
3) selecting an interested region from the image with clear pancreatic structure display, mapping the interested region to other images, and storing the interested region into a natural image format which can be identified by a subsequent classification network;
4) extracting, classifying and fusing the characteristics of the multi-modal image or the fused image according to the selected region of interest, and establishing a basic classification network model aiming at the fused characteristics;
5) and identifying the classification result of each basic classification network to obtain a final classification identification result.
Preferably, the pancreatic medical image data in step 1) is derived from a PACS system and a medical imaging device.
Preferably, the image fusion in step 2) adopts a pixel-level image fusion technique, which includes a spatial domain algorithm and a transform domain algorithm.
Preferably, the region of interest in step 3) is a rectangle containing all pancreatic tissue at the lesion, and the natural image format is.
Preferably, the specific steps of feature extraction, classification, fusion and basic classification network model establishment described in step 4) are as follows:
1) constructing a special depth pyramid convolutional neural network for the multi-modal fusion image, wherein the network has a structure that a series of pyramid pooling layers are used before a full connection layer, so that the input image is allowed to be in any size;
2) inputting data into a special depth pyramid convolutional neural network for the multi-mode fusion image, extracting features output by a full connection layer, and generating a feature map;
3) fusing the above features based on bilinear fusion function, that is, summing after performing outer product operation on corresponding position elements of two feature graphs to obtain a fusion feature graph, wherein the channel number of the fusion feature graph is the square of the channel number of the original feature graph and is expressed as
Figure GDA0003516667120000041
Wherein, ybilRepresenting a fused feature map, xaAnd xbRepresents a characteristic diagram, xa、xb∈RH×W×DH, W, D respectively indicate the length, width and number of channels of the feature map,
Figure GDA0003516667120000042
4) performing dimension reduction processing on the fusion feature map by adopting a convolution fusion function to obtain a dimension-reduced fusion feature map, namely performing convolution operation on a fusion result of the bilinear fusion function and a filter f, and introducing an offset value bias to realize dimension reduction, wherein the dimension reduction is represented as
yconv＝ybil*f+bias；
Wherein, yconvFor the convolution fusion function, f ∈ R1×1×2D×D，bias∈RD；
5) And training a classification model according to the dimension-reduced fusion characteristic diagram, namely establishing a basic classification network model, wherein the used classification method is to combine weak classification models to form a strong classification model or train a support vector machine of a kernel function method.
Preferably, the identification step described in step 5) is as follows:
1) training each established basic classification network model through training data, and calculating a classification error rate;
2) calculating coefficients of each basic classification network model according to the classification error rate;
3) unifying the class labels of the basic classification network models, solving the prediction probability of each basic classification network model on each class label of the example to be detected, removing the deviation points, then carrying out weighted voting on the residual prediction probability, and obtaining the final classification identification result.
Preferably, the method for calculating the classification error rate includes: let M basic classification network models be recorded as CmM {1,2, …, M }, and the training data set T { (y)1，x1)，(y2，x2)，…， (yN，xN) Therein of
Figure GDA0003516667120000051
yiE, Y { -1, +1}, and calculating the classification error rate e of the mth classification model on the training data setmOf the formula
Figure GDA0003516667120000052
The coefficients of the basic classification network models areαmThe calculation formula is
Figure GDA0003516667120000053
Preferably, the class labels of the basic classification network models are unified in { -1,1}, a unified function Am(x) In order to realize the purpose,
Figure GDA0003516667120000054
the prediction probability PmThe calculation method of (1) is that, wherein Label is a category Label,
Pm(Label＝1)＝(Am(x)+1)/2
Pm(Label＝-1)＝1-(Am(x)+1)/2。
preferably, the classification identification result is obtained by classifying the prediction probabilities PL of the M basic classification network modelsmAs indicated by the general representation of the,
Figure GDA0003516667120000055
predicting probability P of any basic classification network modelmmaxAs indicated by the general representation of the,
Pmmax＝max[Pm(Label＝1),Pm(Label＝-1)]，
calculating PLm+PmmaxSorting the results, removing the basic classification network models corresponding to the maximum value and the minimum value, and constructing a linear combination f (x) for the rest M-2 basic network classification models to realize weighted voting so as to obtain a final classification identification result C (x) sign (f (x));
wherein the linear combination
Figure GDA0003516667120000061
The invention and the beneficial effects thereof are explained as follows: (1) the intelligent auxiliary identification method can select the region of interest in a form of manually drawing a rectangle. Because the pancreas is different from other human body parts such as the lung, the mammary gland and the like, the change of the whole structure of the pancreas can be frequently caused when a certain region of the pancreas is diseased, for example, the head of the pancreas is enlarged when the head of the pancreas cancer causes the atrophy of the tail of the pancreas, so that the invention is different from other diseases, the focus region can be finely divided only by using a CAD system, an interested region is selected by a radiologist with abundant experience from a single-mode or fusion image which shows the clear pancreas structure, the interested region is a rectangle containing all pancreas tissues including the focus, the structure of the interested region is simpler than that of artificial fine division, and simultaneously, the uncertainty caused by the immature pancreas automatic division technology is avoided;
(2) in the invention, the final identification result is characterized by three aspects: features of the multi-modal images, features of the multi-modal fused images, and top-level fused features of the multi-modal images. The respective characteristics of the multi-modal images can provide different human body characteristics acquired by different imaging devices; the multi-modal fusion image features can fuse images of different modalities into one image, so that the features of different modalities can be trained together in the stages of feature extraction and classification; the top layer fusion features of the multi-modal images are that top layer features extracted from images in different modes are fused together and then input into a classification model, namely a strong classification model is trained by utilizing the top layer features of the images in different modes, so that the top layer features of the images in different modes are better utilized;
(3) the invention provides a formula for obtaining a final identification result by fusing classification results, wherein the formula removes classification outlier results, then carries out weighted voting on the rest classification results to obtain the final identification result, and the weight of each classification network result gives consideration to the classification error rate of the classification network result in a training stage and the certainty factor of example classification;
(4) in the invention, the pyramid pooling is introduced into the feature extraction network, so that the input images do not need to be unified to the same size, and can be input into the classification network in any size form, thereby avoiding the loss of useful information and the introduction of redundant information;
(5) the invention has strong universality, can carry out combined analysis on medical images of various modes under the selection of doctors, can only carry out analysis on the medical image of a certain mode, is suitable for clinical practice, and can also be used for scientific research in the fields of pancreatic cancer and pancreatic inflammatory diseases.
Drawings
FIG. 1 is a flow chart of an intelligent assisted authentication method according to the present invention;
the dotted line process is an optional process, namely, the process can be performed only when two or more modal images are acquired, otherwise, the process can be performed only by a solid line process, namely, classification and identification are performed on a single mode image;
FIG. 2 is an example of a PET/CT image fusion modality;
FIG. 3 is an example of a deep pyramid pooling convolutional neural network construction;
wherein, DCNN represents a deep convolutional neural network.
Detailed Description
The intelligent auxiliary identification method for pancreatic cancer medical images is described in detail below with reference to the accompanying drawings so that those skilled in the art can implement the method according to the description.
Example 1
The invention discloses a method for intelligently and auxiliarily identifying pancreatic cancer and pancreatic inflammatory diseases, which comprises the following specific steps:
1) reading a multi-mode image, and performing gray level normalization operation;
2) image preprocessing, namely performing operations such as denoising and registration on the normalized image obtained in the step 1), so as to obtain a multi-modal image with improved quality and uniform sampling intervals, and further performing image fusion;
3) according to the multi-modal image and the fused image obtained in the step 2), a radiologist with abundant experience draws a rectangle in a single-mode or fused image with clear pancreas structure display to include the region of interest, namely, a rectangular target area is selected, the region of interest is mapped to other modal images, and the region of interest is stored into a natural image format which can be identified by a subsequent classification network, such as png, bmp and the like;
4) constructing a depth pyramid pooling convolutional neural network, extracting multi-mode and fusion image characteristics and classifying according to the region of interest obtained in the step 3); meanwhile, feature fusion is carried out by utilizing the multi-mode features extracted by the network, and a classification model is established according to the fused features;
5) and 4) identifying the classification results of the basic classification networks in the step 4), removing outlier classification results by combining the classification error rate of the basic classification networks in the training stage and the performance of the basic classification networks in specific examples, and carrying out weighted voting on the rest classification results to obtain the final classification identification results and the certainty factor thereof.
The method comprises the following steps that 1), image data are obtained and normalized; sources of the image data include, but are not limited to, PACS systems and medical imaging devices, including, but not limited to, CT scans, PET/CT scans, SPECT scans, MRI, ultrasound, X-rays, angiograms, fluorograms, micrographs, or combinations thereof; the acquired data is normalized, including but not limited to, segmentation and compression of the gray scale range of the medical image, to enhance the useful details in the image.
Step 2) obtaining the denoised and registered images of different modes, and carrying out image fusion, wherein the method comprises the following steps:
s2-1, denoising the normalized image obtained in the first step, wherein denoising methods include, but are not limited to, mean filtering, median filtering, adaptive median filtering, frequency domain filtering and the like, and combinations of the filtering methods;
s2-2, registering images, namely registering images with low spatial resolution to images with high spatial resolution to obtain uniform sampling intervals, for example, for PET/CT scanning, matching the PET images to the CT images by adopting simple scaling transformation, wherein the registration method comprises but is not limited to feature-based and mutual information-based correlation registration technologies;
and S2-3, fusing the registered images, wherein the adopted pixel level image fusion technology comprises but is not limited to a spatial domain algorithm mainly based on a logic filtering method, a gray-scale weighted average method and a contrast modulation method and a transform domain algorithm mainly based on a pyramid decomposition fusion method and a wavelet transform method, and the number of channels can be adjusted according to the actual requirements of the input mode number, the requirement of a deep convolution neural network on an input layer and the like.
Step 3) is that a doctor selects an interested region from a single-mode or fusion image with clear pancreas structure display, and the interested region is reflected on images of other modalities at the same time, and the method comprises the following steps:
s3-1, selecting the images with the clearest pancreas structure by the experienced radiologist from the images with various modes and the fused images;
s3-2, extracting a region of interest from the image selected in the step 3-1 by the radiologist, wherein the region of interest is a rectangle covering all pancreatic tissues containing the focus, the structure of the region of interest is simpler than that of manual fine segmentation, and the region of interest is reflected on images of other modalities, wherein the boundary of the pancreas and the boundary of the region of interest closest to the boundary are 5-10 pixels;
s3-3, saving the interested areas in the various modes and the fusion image into natural image formats such as png and bmp which can be identified by a subsequent classification network.
Step 4) constructing a special depth pyramid pooling convolutional neural network classification model of each mode and fusion image, wherein each network has the characteristic of arbitrary input, and simultaneously fusing the multi-mode features extracted by the classification model to construct a classification model for the fusion features, and the method specifically comprises the following steps:
s4-1, constructing respective special depth pyramid convolution neural network classification models for each modal image and each fused image, wherein a network structure of each classification model uses a series of pyramid pooling layers before a full connection layer, so that the input image can be in any size, in addition, the network also comprises a short connect structure, a network-in-network and other structures, so as to accelerate the training speed and improve the performance, the optimization algorithm adopts methods such as random gradient descent, Adm, Nadam, Adradag, Adadelta and RMSprop, the classification layers use softmax or linear SVM as an activation function, and the depth and the width of the network are adjusted by using experimental methods such as grid search and the like, so that each classification model can achieve the highest accuracy, namely the network can maximally extract the characteristics with the distinguishing capability in the image;
s4-2, inputting the data into the trained special depth pyramid convolutional neural network for each modal image again, and extracting the output characteristics of the full connection layer at the previous layer (the second last layer) of the classification layer;
s4-3, firstly, the above-mentioned modal image features are fused based on the bilinear fusion function. Bilinear fusion is to sum up the position elements corresponding to 2 characteristic graphs after performing outer product operation, and the number of channels of the fused characteristic graphs is the square of the number of channels of the original characteristic graphs and is expressed as
Figure GDA0003516667120000091
Wherein x isaAnd xbFeature maps, y, representing images of different modalitiesbilRepresenting a fused spatial feature map;
xa、xb∈RH×W×Dh, W, D respectively indicate the length, width and number of channels of the feature map,
Figure GDA0003516667120000092
Figure GDA0003516667120000093
s4-4, the problem that the dimension of the characteristic graph obtained in the steps is too high, and the invention further adopts a convolution fusion function yconv＝fconv(xa，xb) Performing dimension reduction on the fusion characteristic diagram, performing convolution operation on the fusion result of the bilinear fusion function and the filter f, and introducing a bias value bias to realize dimension reduction, which is expressed as
yconv＝ybil*f+bias
Wherein f ∈ R1×1×2D×D，bias∈RD. Thus, the multi-modal image top-level features are better fused together;
s4-5, training a classification model according to the fusion characteristics obtained in the S4-4, and training the classification model to reach the highest accuracy; alternative classification methods include, but are not limited to, training a strong classification model formed by a combination of weak classification models, such as an Adaboost classification model, a random forest classification model, etc., or training a support vector machine of a kernel function method, etc.
And step 5) identifying the classification results of the basic classification models in the step 4), removing outlier classification results by combining the classification error rate of the basic classification models in the training stage and the performance of the basic classification models in specific examples, and carrying out weighted voting on the rest classification results. The method comprises the following steps:
s5-1, calculating the classification error rate of the classification result of each basic classification model on the test data in the training stage, setting M basic classification models, and recording them as CmWhere M is {1,2, …, M }, and the training data set T is { (y)1，x1)，(x2，y2)，…，(xN,yN) Therein of
Figure GDA0003516667120000101
yiE, Y { -1, +1}, and calculating the classification error rate e of the mth classification model on the training data setm，
Figure GDA0003516667120000102
S5-2, calculating coefficients alpha of M classification models obtained based on classification error ratem，
Figure GDA0003516667120000103
And S5-3, unifying the class labels of different classification models, obtaining the prediction probability of each classification model on each class label of the test case, removing two judgment deviation points, and calculating the weighted sum of the judgment probabilities of the residual classification models to obtain the final diagnosis opinion and the certainty factor.
In some examples of implementation, it is possible to,after the softmax prediction is completed, unifying the class labels of the softmax to { -1,1}, and calculating the prediction probability on each class label; for softmax, the output value itself is the prediction probability; for SVM, Adaboost and other classification models, and for the example x, the specific method for calculating the prediction probability is to unify the class labels of all basic classification network models into { -1,1}, and unify the function Am(x) In order to realize the purpose,
Figure GDA0003516667120000104
prediction probability PmThe calculation method of (1) is that, wherein Label is a category Label,
Pm(Label＝1)＝(Am(x)+1)/2
Pm(Label＝-1)＝1-(Am(x)+1)/2。
the classification identification result is obtained by calculating the prediction probability PL of M basic classification network modelsmAs indicated by the general representation of the,
Figure GDA0003516667120000111
predicting probability P of any basic classification network modelmmaxAs indicated by the general representation of the,
Pmmax＝max[Pm(Label＝1),Pm(Label＝-1)]，
calculating PLm+PmmaxThe results are sorted, the basic classification network models corresponding to the maximum value and the minimum value are removed, linear combinations are constructed for the remaining M-2 basic classification models,
Figure GDA0003516667120000112
obtaining a final classification discrimination result C (x) sign (f (x)), wherein the linear combination f (x) realizes the weighted voting of the M-2 basic classification models, Cm(x) The preceding coefficients represent the mth classification model Cm(x) Here, the sign of the sum of all coefficients being 1, f (x) determines the class of instance x, and the absolute value of f (x) indicates the accuracy of the classification.
Example 2
The image in FIG. 2 is provided by Changhai Hospital, Shanghai, and is exemplified by PET/CT to illustrate the image fusion process. Firstly, a registered PET image a) and a registered CT image b) are fused into a pseudo-image c), and then the pseudo-image c) is changed into a gray-scale image d) through gray-scale conversion, so that the information of the two modes is fused together for subsequent processing, and the pseudo-image can also be directly subjected to subsequent processing.
Example 3
An example of constructing a depth pyramid pooling convolutional neural network, although the specific network structure of the depth pyramid pooling convolutional neural network of each modal image is different, the depth pyramid pooling convolutional neural network includes the following 4 parts:
1) the input of images with any size can be processed by decentralization, standardization, ZCA whitening and the like, so that the convergence is easier during training, and the training process is accelerated;
2) constructing a Deep Convolutional Neural Network (DCNN), wherein the neural network comprises a convolutional layer, a pooling layer, a BN layer, a short connection and the like, and the neural network has the optimal feature extraction capability by adjusting and optimizing the network depth, the network width, an optimization algorithm, an activation function, the learning rate and the like;
3) the pyramid pooling layer is introduced, so that feature maps of different sizes generated by the convolutional neural network in the step 2) can be unified to a full connection layer of the same size, and therefore, the network can process input images of different sizes;
4) and a classification layer for classifying the features collected from the input image, thereby converting the disease diagnosis problem into a feature classification problem, wherein the activation function includes but is not limited to softmax and linear SVM.
While embodiments of the invention have been disclosed above, it is not limited to the applications set forth in the description and the embodiments, which are fully applicable in various fields adapted to the invention, and it will be readily apparent to those skilled in the art that additional modifications may be made, and the invention is not limited to the specific details and examples set forth herein, without departing from the general concept defined by the claims and their equivalents.

Claims (7)

1. A method for intelligent auxiliary identification of pancreatic cancer and pancreatic inflammatory diseases is characterized by comprising the following steps:
1) reading and normalizing the pancreatic medical image data to obtain a normalized image;
2) denoising, registering and image fusing the normalized image to obtain a multi-modal fused image;
3) selecting an interested region from the image with clear pancreatic structure display, mapping the interested region to other images, and storing the interested region as a natural image format which can be identified by a subsequent classification network;
4) extracting, classifying and fusing the characteristics of the multi-mode fusion image according to the selected region of interest, and establishing a basic classification network model aiming at the fused characteristics;
5) identifying the classification result of each basic classification network to obtain a final classification identification result;
the specific steps of feature extraction, classification, fusion and basic classification network model establishment in the step 4) are as follows:
a: constructing a special depth pyramid convolutional neural network for the multi-modal fusion image, wherein the network has a structure that a series of pyramid pooling layers are used before a full connection layer, so that the input image is allowed to be in any size;
b: inputting data into a special depth pyramid convolutional neural network for the multi-mode fusion image, extracting features output by a full connection layer, and generating a feature map;
c: fusing the above features based on bilinear fusion function, that is, summing after performing outer product operation on corresponding position elements of two feature graphs to obtain a fusion feature graph, wherein the channel number of the fusion feature graph is the square of the channel number of the original feature graph and is expressed as
Figure FDA0003516667110000011
Wherein, ybilRepresents a fused feature map, xaAnd xbRepresents a characteristic diagram, xa、xb∈RH×W×DH, W, D respectively indicate the length, width and number of channels of the feature map,
Figure FDA0003516667110000015
a. b represents the different images of the image to be displayed,
Figure FDA0003516667110000012
refers to a feature value at position (i, j) in different channels in the feature map,
Figure FDA0003516667110000013
b refers to the characteristic value at the position (i, j) in different channels in the characteristic diagram;
Figure FDA0003516667110000014
refers to the range to which the calculated value belongs, wherein D2Representing the number of channels as the original square;
d: performing dimension reduction processing on the fusion feature map by adopting a convolution fusion function to obtain a dimension-reduced fusion feature map, namely performing convolution operation on a fusion result of the bilinear fusion function and a filter f, and introducing an offset value bias to realize dimension reduction, wherein the dimension reduction is represented as
yconv＝ybil*f+bias，
Wherein, yconvFor the convolution fusion function, f represents a two-dimensional convolution, f belongs to R1×1×2D×DWherein, 1 × 1 × 2D × D represents the number, length, width and number of channels of the feature map respectively; bias represents the offset in the convolution operation, bias ∈ RD；
E: training a classification model according to the dimension-reduced fusion characteristic diagram, namely establishing a basic classification network model, wherein the classification method is to combine weak classification models to form a strong classification model or train a support vector machine of a kernel function method;
the identification step in step 5) is as follows:
a: training each established basic classification network model through training data, and calculating a classification error rate;
b: calculating coefficients of each basic classification network model according to the classification error rate;
c: unifying the class labels of the basic classification network models, solving the prediction probability of each basic classification network model on each class label of the example to be detected, removing the deviation points, then carrying out weighted voting on the residual prediction probability, and obtaining the final classification identification result.
2. The method of claim 1, wherein the pancreatic medical image data in step 1) is derived from a PACS system and a medical imaging device.
3. The method according to claim 1, wherein the image fusion in step 2) adopts a pixel level image fusion technique, including a spatial domain algorithm and a transform domain algorithm.
4. The method according to claim 1, wherein the region of interest in step 3) is a rectangle containing all pancreatic tissue at the lesion, and the native image format is.
5. The method of claim 1, wherein the classification error rate is calculated by: let M basic classification network models be recorded as CmM {1,2, …, M }, and the training data set T { (y)1，x1)，(y2，x2)，…，(yN，xN) Therein of
Figure FDA0003516667110000021
yiE, Y { -1, +1}, and calculating the classification error rate e of the mth classification model on the training data setmIs of the formula
Figure FDA0003516667110000022
The coefficient of each basic classification network model is alphamThe calculation formula is
Figure FDA0003516667110000023
6. The method of claim 5, wherein the class labels of the basic classification network models are unified in { -1,1}, a unified function Am(x) In order to realize the purpose,
Figure FDA0003516667110000031
prediction probability PmThe calculation method of (a) is that,
Pm(Label＝1)＝(Am(x)+1)/2
Pm(Label＝-1)＝1-(Am(x)+1)/2，
wherein, Label is a category Label.
7. The method of claim 6, wherein the classification discrimination result is obtained by determining the predicted probability PL of M basic classification network modelsmAs indicated by the general representation of the,
Figure FDA0003516667110000032
predicting probability P of any basic classification network modelmmaxAs indicated by the general representation of the,
Pmmax＝max[Pm(Label＝1),Pm(Label＝-1)]，
calculating PLm+PmmaxSorting the results, removing the basic classification network models corresponding to the maximum value and the minimum value, and constructing a linear combination f (x) for the rest M-2 basic network classification models to realize weighted voting so as to obtain a final classification identification result C (x) sign (f (x));
wherein the combination is linear
Figure FDA0003516667110000033
CN201810141703.4A 2018-02-11 2018-02-11 Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases Active CN108537773B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810141703.4A CN108537773B (en) 2018-02-11 2018-02-11 Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810141703.4A CN108537773B (en) 2018-02-11 2018-02-11 Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases

Publications (2)

Publication Number Publication Date
CN108537773A CN108537773A (en) 2018-09-14
CN108537773B true CN108537773B (en) 2022-06-17

Family

ID=63485999

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810141703.4A Active CN108537773B (en) 2018-02-11 2018-02-11 Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases

Country Status (1)

Country Link
CN (1) CN108537773B (en)

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110909755B (en) * 2018-09-17 2023-05-30 阿里巴巴集团控股有限公司 Object feature processing method and device
CN109559296B (en) * 2018-10-08 2020-08-25 广州市大智网络科技有限公司 Medical image registration method and system based on full convolution neural network and mutual information
CN109544512B (en) * 2018-10-26 2020-09-18 浙江大学 Multi-mode-based embryo pregnancy result prediction device
CN109544517A (en) * 2018-11-06 2019-03-29 中山大学附属第医院 Method and system are analysed in multi-modal ultrasound group credit based on deep learning
CN109273084B (en) * 2018-11-06 2021-06-22 中山大学附属第一医院 Method and system based on multi-mode ultrasound omics feature modeling
EP3660741B1 (en) * 2018-11-29 2022-05-04 Koninklijke Philips N.V. Feature identification in medical imaging
CN109815965B (en) * 2019-02-13 2021-07-06 腾讯科技（深圳）有限公司 Image filtering method and device and storage medium
CN109998599A (en) * 2019-03-07 2019-07-12 华中科技大学 A kind of light based on AI technology/sound double-mode imaging fundus oculi disease diagnostic system
CN109949288A (en) * 2019-03-15 2019-06-28 上海联影智能医疗科技有限公司 Tumor type determines system, method and storage medium
CN110188788A (en) * 2019-04-15 2019-08-30 浙江工业大学 The classification method of cystic Tumor of Pancreas CT image based on radiation group feature
CN110349662B (en) * 2019-05-23 2023-01-13 复旦大学 Cross-image set outlier sample discovery method and system for filtering lung mass misdetection results
CN110619639A (en) * 2019-08-26 2019-12-27 苏州同调医学科技有限公司 Method for segmenting radiotherapy image by combining deep neural network and probability map model
CN110909672A (en) * 2019-11-21 2020-03-24 江苏德劭信息科技有限公司 Smoking action recognition method based on double-current convolutional neural network and SVM
CN111667486B (en) * 2020-04-29 2023-11-17 杭州深睿博联科技有限公司 Multi-modal fusion pancreas segmentation method and system based on deep learning
CN111798410A (en) * 2020-06-01 2020-10-20 深圳市第二人民医院(深圳市转化医学研究院) Cancer cell pathological grading method, device, equipment and medium based on deep learning model
CN111680687B (en) * 2020-06-09 2022-05-10 江西理工大学 Depth fusion classification method applied to mammary X-ray image anomaly identification
CN111833332A (en) * 2020-07-15 2020-10-27 中国医学科学院肿瘤医院深圳医院 Generation method and identification method of energy spectrum CT identification model of bone metastasis tumor and bone island
CN112070809B (en) * 2020-07-22 2024-01-26 中国科学院苏州生物医学工程技术研究所 Pancreatic cancer accurate diagnosis system based on PET/CT double-time imaging
CN112419306B (en) * 2020-12-11 2024-03-15 长春工业大学 NAS-FPN-based lung nodule detection method
CN112951426B (en) * 2021-03-15 2023-02-28 山东大学齐鲁医院 Construction method and evaluation system of pancreatic duct adenoma inflammatory infiltration degree judgment model
CN113066110A (en) * 2021-05-06 2021-07-02 北京爱康宜诚医疗器材有限公司 Method and device for selecting marking points in pelvis registration
CN113449770B (en) * 2021-05-18 2024-02-13 科大讯飞股份有限公司 Image detection method, electronic device and storage device
CN115240854B (en) * 2022-07-29 2023-10-03 中国医学科学院北京协和医院 Pancreatitis prognosis data processing method and system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101976436A (en) * 2010-10-14 2011-02-16 西北工业大学 Pixel-level multi-focus image fusion method based on correction of differential image
CN105956532A (en) * 2016-04-25 2016-09-21 大连理工大学 Traffic scene classification method based on multi-scale convolution neural network
CN106682435A (en) * 2016-12-31 2017-05-17 西安百利信息科技有限公司 System and method for automatically detecting lesions in medical image through multi-model fusion
CN107291822A (en) * 2017-05-24 2017-10-24 北京邮电大学 The problem of based on deep learning disaggregated model training method, sorting technique and device
CN107403201A (en) * 2017-08-11 2017-11-28 强深智能医疗科技（昆山）有限公司 Tumour radiotherapy target area and jeopardize that organ is intelligent, automation delineation method
CN107492097A (en) * 2017-08-07 2017-12-19 北京深睿博联科技有限责任公司 A kind of method and device for identifying MRI image area-of-interest

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10402697B2 (en) * 2016-08-01 2019-09-03 Nvidia Corporation Fusing multilayer and multimodal deep neural networks for video classification

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101976436A (en) * 2010-10-14 2011-02-16 西北工业大学 Pixel-level multi-focus image fusion method based on correction of differential image
CN105956532A (en) * 2016-04-25 2016-09-21 大连理工大学 Traffic scene classification method based on multi-scale convolution neural network
CN106682435A (en) * 2016-12-31 2017-05-17 西安百利信息科技有限公司 System and method for automatically detecting lesions in medical image through multi-model fusion
CN107291822A (en) * 2017-05-24 2017-10-24 北京邮电大学 The problem of based on deep learning disaggregated model training method, sorting technique and device
CN107492097A (en) * 2017-08-07 2017-12-19 北京深睿博联科技有限责任公司 A kind of method and device for identifying MRI image area-of-interest
CN107403201A (en) * 2017-08-11 2017-11-28 强深智能医疗科技（昆山）有限公司 Tumour radiotherapy target area and jeopardize that organ is intelligent, automation delineation method

Also Published As

Publication number Publication date
CN108537773A (en) 2018-09-14

Similar Documents

Publication Publication Date Title
CN108537773B (en) Method for intelligently assisting in identifying pancreatic cancer and pancreatic inflammatory diseases
Li et al. Attention dense-u-net for automatic breast mass segmentation in digital mammogram
CN110310281B (en) Mask-RCNN deep learning-based pulmonary nodule detection and segmentation method in virtual medical treatment
US11101033B2 (en) Medical image aided diagnosis method and system combining image recognition and report editing
Sharif et al. A comprehensive review on multi-organs tumor detection based on machine learning
Yousef et al. A holistic overview of deep learning approach in medical imaging
Li et al. A 3D deep supervised densely network for small organs of human temporal bone segmentation in CT images
Chan et al. Texture-map-based branch-collaborative network for oral cancer detection
Naik et al. Lung nodule classification on computed tomography images using deep learning
JP2009502230A (en) Detection of wounds in medical images
US7359538B2 (en) Detection and analysis of lesions in contact with a structural boundary
EP2208183B1 (en) Computer-aided detection (cad) of a disease
Fan et al. Lung nodule detection based on 3D convolutional neural networks
Mittapalli et al. Multiscale CNN with compound fusions for false positive reduction in lung nodule detection
CN111798424B (en) Medical image-based nodule detection method and device and electronic equipment
US20150065868A1 (en) System, method, and computer accessible medium for volumetric texture analysis for computer aided detection and diagnosis of polyps
Wang et al. Whole mammographic mass segmentation using attention mechanism and multiscale pooling adversarial network
Sahli et al. U-Net: A valuable encoder-decoder architecture for liver tumors segmentation in CT images
Guo et al. Coarse-to-fine airway segmentation using multi information fusion network and CNN-based region growing
Atiyah et al. Brain MRI Images Segmentation Based on U-Net Architecture
Jadhav et al. 3D virtual pancreatography
Almutairi et al. An efficient USE-Net deep learning model for cancer detection
Paliwal et al. A Comprehensive Analysis of Identifying Lung Cancer via Different Machine Learning Approach
CN115132275B (en) Method for predicting EGFR gene mutation state based on end-to-end three-dimensional convolutional neural network
CN114757894A (en) Bone tumor focus analysis system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant