CN113160120A - Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning - Google Patents
Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning Download PDFInfo
- Publication number
- CN113160120A CN113160120A CN202110162875.1A CN202110162875A CN113160120A CN 113160120 A CN113160120 A CN 113160120A CN 202110162875 A CN202110162875 A CN 202110162875A CN 113160120 A CN113160120 A CN 113160120A
- Authority
- CN
- China
- Prior art keywords
- liver
- blood vessel
- image
- segmentation
- vessel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000004204 blood vessel Anatomy 0.000 title claims abstract description 196
- 210000004185 liver Anatomy 0.000 title claims abstract description 175
- 230000011218 segmentation Effects 0.000 title claims abstract description 120
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000004927 fusion Effects 0.000 title claims abstract description 39
- 238000013135 deep learning Methods 0.000 title claims abstract description 38
- 230000010365 information processing Effects 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 18
- 238000004364 calculation method Methods 0.000 claims abstract description 17
- 230000002093 peripheral effect Effects 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 238000004891 communication Methods 0.000 claims abstract description 4
- 238000012549 training Methods 0.000 claims description 26
- 230000000694 effects Effects 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 8
- 238000005259 measurement Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 238000013136 deep learning model Methods 0.000 abstract description 2
- 238000010801 machine learning Methods 0.000 abstract description 2
- 230000002440 hepatic effect Effects 0.000 description 9
- 230000006872 improvement Effects 0.000 description 5
- 230000007547 defect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 208000014018 liver neoplasm Diseases 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000002591 computed tomography Methods 0.000 description 3
- 201000007270 liver cancer Diseases 0.000 description 3
- 206010028980 Neoplasm Diseases 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003902 lesion Effects 0.000 description 2
- 208000019423 liver disease Diseases 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 206010061998 Hepatic lesion Diseases 0.000 description 1
- 206010019695 Hepatic neoplasm Diseases 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 239000013043 chemical agent Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000002526 effect on cardiovascular system Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 230000002685 pulmonary effect Effects 0.000 description 1
- 238000002271 resection Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 238000002054 transplantation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration by the use of local operators
- G06T5/30—Erosion or dilatation, e.g. thinning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30056—Liver; Hepatic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30101—Blood vessel; Artery; Vein; Vascular
Abstract
The invention belongs to the technical field of medical image processing, and discloses a liver blood vessel segmentation method and a system based on multi-mode fusion and deep learning, wherein the method comprises the following steps: the information and image acquisition module is used for acquiring basic information and CT images of an examiner; the image information processing module is used for preprocessing the image to be subjected to liver blood vessel segmentation by the information processing server; the liver blood vessel segmentation module is used for transmitting the processed communication to the blood vessel segmentation algorithm server for calculation by the information processing server, and transmitting the result back to the information processing server by the algorithm server; and the blood vessel three-dimensional reconstruction module is used for reconstructing the blood vessel image segmented by the model into a three-dimensional liver blood vessel tree model. According to the invention, the liver blood vessel and the peripheral local information are obtained by means of a classical machine learning algorithm, and the liver blood vessel is accurately segmented by combining a deep learning model, so that finer and fuzzy liver blood vessel branches can be identified, and the accuracy and fineness of the liver blood vessel segmentation are effectively improved.
Description
Technical Field
The invention belongs to the technical field of medical image processing, and particularly relates to a liver blood vessel segmentation method and system based on multi-mode fusion and deep learning.
Background
At present, products and applications combining artificial intelligence and medical treatment are no longer few, such as intelligent pulmonary nodule detection, cardiovascular examination, intelligent diagnosis of various tumor diseases and the like, which show good effects, and the progress of the medical technical field is greatly promoted. Medical CT images are images formed by physical and chemical means, and each organ or lesion of the human body shows a difference in density due to a difference in ability to absorb physical light or an influence of a chemical agent, and a doctor can visually recognize the internal structure of the human body by the CT images. However, in some special environments, professional doctors still cannot efficiently and accurately distinguish fine organs and pathological changes. Computer vision and medical images are always closely related, and a target object in the medical images can be segmented through a neural network model, so that a target entity is displayed more clearly. In particular, the U-Net neural network model changes the common convolutional neural network into a U-shaped structure to bridge the left and right corresponding network layers, the depth of the model is the number of the layers of the whole network, and better class characteristics can be learned.
It is estimated by research that over one fifth of the population in china suffers from liver disease. According to the report of 2018 international cancer research center, the incidence rate of liver cancer in China is the third and the second of the mortality rate of male cancer, and the number of liver cancer patients in China is the most worldwide based on population scale estimation. At present, the clinical treatment method of liver cancer mainly comprises liver tumor resection, liver transplantation, radio frequency/microwave ablation based on enhanced CT image guidance and the like, and the key for the success of the operations is to accurately master the blood vessel edge, the blood vessel branch and the relation between the blood vessel and the focus of the liver. The image slices generated by CT examination of a single patient are about 200-500, and manual segmentation of hepatic vessels in CT scanning slices by an image expert is time-consuming, labor-consuming and easy to make mistakes. Therefore, the automatic segmentation and three-dimensional model reconstruction of the liver blood vessel are realized by combining the artificial intelligence technology and the computer vision, which is one of the most effective solutions at present.
At present, the research on the liver blood vessel segmentation algorithm is less, and no mature technology or product publication exists in China. The main reasons are as follows: firstly, the difference between different patients and different liver disease development is relatively large, and some traditional algorithms cannot be unified; secondly, the liver and the liver blood vessels have little difference in density, strong homogeneity and are not easy to distinguish; thirdly, liver blood vessels are tiny, fuzzy, difficult to label and lack of a large amount of effective data sets. In the prior art, a classic blood vessel segmentation algorithm based on gray level or gradient, such as three-dimensional region growth, fuzzy clustering and the like, cannot effectively solve the problem of low-contrast liver blood vessel segmentation; the general deep learning network model has limited training data set size and is easily influenced by the imbalance of positive and negative samples of blood vessels and liver.
Through the above analysis, the problems and defects of the prior art are as follows: in the prior art, a classic blood vessel segmentation algorithm based on gray scale or gradient cannot effectively solve liver blood vessel segmentation with low contrast; in addition, the scale of a training data set of a general deep learning network model is limited, and the general deep learning network model is easily influenced by the uneven balance of blood vessels and liver positive and negative samples.
The difficulty in solving the above problems and defects is:
in the aspect of algorithm optimization, firstly, most hepatic vessels are fine and have limited resolution, the contrast between the hepatic vessels and a hepatic region is low, the interference of a hepatic lesion region is strong, so that the characteristics of the hepatic vessels are difficult to capture, and great challenges are provided for the data processing of the hepatic vessels and the recognition capability of a model; secondly, the ratio of the positive sample to the negative sample of the liver blood vessel and the liver area is extremely unbalanced, and the negative samples need to be greatly reduced by an effective means; thirdly, the difference of liver images of different patients is very large, the data diversity needs to be greatly increased, and the model accuracy is improved. Three different improvements were proposed based on this study: firstly, pre-segmenting liver blood vessels by using a multi-mode fusion method, and filtering noise interference in non-blood vessel regions; expanding the pre-segmented liver blood vessel region, and taking out the expanded non-blood vessel region as a negative sample; thirdly, continuously increasing and correcting data to participate in model training, and continuously expanding the scale of the model training by using a system mutually fed back with an algorithm.
The significance of solving the problems and the defects is as follows: the invention improves the blood vessel segmentation efficiency and precision, and reduces the workload and subjective errors of medical staff; meanwhile, the invention carries out three-dimensional modeling based on the liver blood vessel which is accurately segmented, completely and three-dimensionally displays the morphology and tiny branches of the liver blood vessel, and assists the diagnosis of difficult lesions and the decision of liver surgery schemes.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a liver vessel segmentation method and system based on multi-mode fusion and deep learning.
The invention is realized in such a way, and the liver blood vessel segmentation method based on multi-mode fusion and deep learning comprises the following steps:
step one, the information processing server acquires basic information of a patient and a CT image sequence in a DICOM format from a security port. The strategy can acquire original data of a patient from a system without barriers while guaranteeing data security, and provides data guarantee for a series of improvement measures subsequently implemented by the algorithm.
And step two, cascading a liver segmentation module to obtain a corresponding liver mask, automatically adjusting the window width and the window level, converting the window width and the window level into a PNG (portable network generator) format gray image, and taking out a liver part according to the mask. The strategy can effectively eliminate the interference of non-liver regions in the CT image of the patient, and converts the DICOM format into the PNG format to prepare for further pre-segmentation of the hepatic blood vessels.
Step three, extracting gradient characteristics of the blood vessel texture through a Hessian matrix; obtaining local information of the blood vessel through a clustering algorithm; and carrying out fusion pre-segmentation on the two images through a model to obtain a local image of the blood vessel. The step is the core of the invention, and aims to obtain the real blood vessel core image information, eliminate the interference of non-blood vessel areas and provide a fundamental basis for further fine segmentation of blood vessels.
And fourthly, removing noise in the pre-segmented blood vessel by adopting opening operation, expanding the blood vessel area by using an expansion algorithm, and carrying out and operation on the blood vessel area and the original liver image to construct a new liver image. The step can effectively remove the noise of the blood vessel area in the step three, and meanwhile, the problem of data unbalance is effectively solved.
And fifthly, transmitting the processed image to a U-Net blood vessel segmentation algorithm server for calculation by the information processing server through the gRPC, and transmitting the result back to the information processing server by the algorithm server. The step completes the high-speed transmission of data and the high-speed operation in an algorithm server in the method, and realizes the high-efficiency liver blood vessel segmentation capability of the model.
And step six, the information processing server transmits the examiner information and the blood vessel segmentation result into a liver blood vessel three-dimensional reconstruction model, and the three-dimensional reconstruction model returns a three-dimensional reconstruction result in a VTK format. The step is to carry out three-dimensional reconstruction on the liver blood vessel finely segmented by the method so as to store and manage the segmentation result of the liver blood vessel of the patient and display the segmentation result, thereby providing conditions for subsequent related application and research.
Further, in the second step, the process of automatically adjusting the window width and converting the window level into the PNG format grayscale image is as follows: the DICOM image window width and window level are adjusted in a self-adaptive mode, so that the liver blood vessel has the optimal developing effect, and a liver blood vessel image enhancement method at a pixel level is constructed.
Further, in the second step, the specific process of taking out the liver part according to the mask is as follows: and the cascade liver segmentation module acquires the corresponding liver mask and segments the liver region in the image sequence.
Further, in the third step, the specific process of obtaining the local image of the blood vessel by fusing and pre-dividing the two through the model is as follows:
fusing the liver blood vessel characteristics by adopting two-channel input, inputting the gray value of a liver image by a channel I, performing normalization processing, and further acquiring a blood vessel core region by a clustering algorithm;
inputting blood vessel texture information of the liver image after Hessian matrix processing by a channel II;
calculation of eigenvalues λ by Hessian matrix1And λ2And constructing a blood vessel measurement function according to the characteristic values:
further, weighting and fusing the two channel results through the model to obtain a blood vessel core area image; removing noise data through open operation;
and expanding the blood vessel image after the noise is removed by adopting an expansion algorithm, and pre-segmenting the blood vessel image and the peripheral area in the original liver image by taking the blood vessel image as a mask to form a new data set.
Further, in the fifth step, the information processing server transmits the processed image to the U-Net vessel segmentation algorithm server for calculation, and the deep learning network based on the U-Net architecture of the vessel segmentation method improves the liver vessel recognition efficiency by improving the module structure, the loss function, the training strategy and the like.
Further, in the sixth step, the information processing server transmits the examiner information and the blood vessel segmentation result to a liver blood vessel three-dimensional reconstruction model for liver blood vessel segmentation, specifically:
and (3) adopting a U-Net model training set, drawing and correcting labels of the training set by a medical image professional technician, and repeating the iterative training model until the result performance meets the preset target.
Further, the U-Net model is a U-Net architecture deep learning network model, the training set label is composed of 0, 1 and 2, 0 marks the picture background, 1 marks the liver blood vessel, and 2 marks the peripheral liver region of the blood vessel.
Further, the liver vessel segmentation adopts preprocessed liver data in an information processing server, the processed image is input into a trained U-Net model through a gPC, and a liver vessel refined segmentation result is output.
Another object of the present invention is to provide a liver vessel segmentation system based on multi-modal fusion and deep learning, which implements the liver vessel segmentation method based on multi-modal fusion and deep learning, the liver vessel segmentation system based on multi-modal fusion and deep learning comprising:
the information and image acquisition module is used for acquiring basic information and CT images of an examiner;
the image information processing module is used for preprocessing the image to be subjected to liver blood vessel segmentation by the information processing server;
the liver blood vessel segmentation module is used for transmitting the processed communication to the blood vessel segmentation algorithm server for calculation by the information processing server, and transmitting the result back to the information processing server by the algorithm server;
the blood vessel three-dimensional reconstruction module is used for reconstructing the blood vessel image segmented by the model into a three-dimensional liver blood vessel tree model;
and the information storage module is used for storing the segmented blood vessel data, the three-dimensional blood vessel model and the basic information of the CT examiner in a specific database and a file storage area.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention proposes three different improvements: firstly, local information of the liver blood vessels is extracted by using a dual-channel image fusion algorithm, the liver blood vessels are pre-segmented, a large area of liver regions without blood vessels are filtered, and the network can learn the blood vessel characteristics more easily; secondly, expanding the surrounding area of the blood vessel by adopting an expansion algorithm to serve as a negative sample, so that the positive and negative samples of the training set are balanced; thirdly, the diversity of the data samples is greatly increased, the scale of the data samples is continuously expanded by a system which is fed back with the algorithm, the data set is continuously increased and corrected, and a neural network model is iteratively trained; the three different improvement methods are specifically as follows: (1) aiming at the problems of low contrast, fuzzy blood vessels, very small blood vessels, strong noise interference and the like of a liver blood vessel image, the invention provides an image fusion method based on two channels to pre-segment the liver blood vessels, extract the gray level and gradient characteristics of the liver blood vessels, filter out a large area of liver regions without blood vessels and enable a network to learn the characteristics of the liver blood vessels more easily. (2) Aiming at the problem of imbalance of positive and negative samples of a liver blood vessel image data set, the peripheral region of the blood vessel expanded by adopting a blood vessel pre-segmentation result is a negative sample, and the blood vessel marked manually and accurately is used as a positive sample training model, so that the model performance is effectively improved. (3) The invention aims at the problems that the manual three-dimensional reconstruction of the liver blood vessel has large workload and is easy to make mistakes, and the common automatic reconstruction model is difficult to reconstruct the thinner branch containing the blood vessel, and adopts a two-stage blood vessel segmentation method: extracting a plurality of blood vessel characteristics by means of a traditional algorithm, and pre-segmenting all regions containing liver blood vessels; and a high-performance AI model is used, a data set is continuously increased, and an expert continuously corrects the data training model to finely divide the small blood vessels, so that the problem that three-dimensional reconstruction of the fine branches of the hepatic blood vessels is difficult is solved.
According to the invention, the liver blood vessel and the peripheral local information are obtained by means of a classic machine learning algorithm, and the liver blood vessel is accurately segmented by combining a deep learning model, so that finer and fuzzy liver blood vessel branches can be identified, and the accuracy and fineness of the liver blood vessel segmentation are effectively improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a liver vessel segmentation method based on multi-modal fusion and deep learning according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a liver vessel segmentation system based on multi-modal fusion and deep learning according to an embodiment of the present invention.
In the figure: 1. an information and image acquisition module; 2. an image information processing module; 3. a liver blood vessel segmentation module; 4. a blood vessel three-dimensional reconstruction module; 5. and an information storage module.
Fig. 3 is a data processing flow chart in a liver vessel segmentation process based on multi-modal fusion and deep learning according to an embodiment of the present invention.
FIG. 4 is a schematic diagram illustrating an image processing module according to an embodiment of the present invention comparing image processing results with different algorithms;
in the figure: FIG. A is an original drawing; the processing result of the Hessian algorithm is shown in a graph (b); figure (c) results of the two-channel fusion algorithm; opening operation denoising processing results; panel (e) results of the vessel dilation process; the expansion result and the result of the original and the calculation are shown in (f).
Fig. 5 is a schematic diagram of a model algorithm implementation process provided in the embodiment of the present invention.
Fig. 6 is a comparison graph of the segmentation effect of the model algorithm provided by the embodiment of the invention, in terms of identification of small blood vessels and the integrity of the blood vessel segmentation, compared with the fully trained model segmentation effect of the U-Net algorithm alone. The first line is an original image, the second line only adopts a U-Net model segmentation effect, and the third line is a segmentation effect graph of the embodiment of the invention.
Fig. 7 is a comparison graph of the segmentation effect of the model algorithm provided by the embodiment of the invention, in terms of noise interference in liver blood vessel segmentation, compared with the segmentation effect of the model fully trained by only using the U-Net algorithm. The first line is an original image, the second line only adopts the U-Net model segmentation effect, and the third line is a segmentation effect graph of the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Aiming at the problems in the prior art, the invention provides a liver vessel segmentation method and a liver vessel segmentation system based on multi-mode fusion and deep learning, and the invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a liver vessel segmentation method based on multi-modal fusion and deep learning according to an embodiment of the present invention includes:
s101: the information processing server acquires basic information of the patient and a CT image sequence in a DICOM format from the security port.
S102: and the cascade liver segmentation module is used for acquiring a corresponding liver mask, automatically adjusting the window width and the window level, converting the window width and the window level into a PNG (portable network generator) format gray image, and taking out a liver part according to the mask.
S103: extracting gradient characteristics of the blood vessel texture through a Hessian matrix; obtaining local information of the blood vessel through a clustering algorithm; and carrying out fusion pre-segmentation on the two images through a model to obtain a local image of the blood vessel.
S104: noise in the pre-segmented blood vessel is removed by adopting opening operation, the blood vessel area is expanded by using an expansion algorithm, and the blood vessel area and an original liver image are subjected to AND operation to construct a new liver image.
S105: through the gRPC, the information processing server transmits the processed image to the U-Net blood vessel segmentation algorithm server for calculation, and the algorithm server transmits the result back to the information processing server.
S106: and the information processing server transmits the examiner information and the blood vessel segmentation result to the liver blood vessel three-dimensional reconstruction model, and the three-dimensional reconstruction model returns a three-dimensional reconstruction result in a VTK format.
A person skilled in the art of the liver vessel segmentation method based on multi-modal fusion and deep learning provided by the present invention may also use other steps to implement, and the liver vessel segmentation method based on multi-modal fusion and deep learning provided by the present invention shown in fig. 1 is only a specific example.
In S102 provided by the embodiment of the present invention, the specific process of automatically adjusting the window width and converting the window level into the PNG-format grayscale image is as follows: the DICOM image window width and window level are adjusted in a self-adaptive mode, so that the liver blood vessel has the best image display effect, and a liver blood vessel image enhancement method at a pixel level is constructed.
In S102 provided by the embodiment of the present invention, a specific process of extracting a liver part according to a mask is as follows: and the cascade liver segmentation module is used for acquiring the corresponding liver mask and segmenting the liver region in the image sequence.
In S1023 provided by the embodiment of the present invention, the specific process of obtaining the local blood vessel image by performing fusion pre-segmentation on the two through models is as follows:
fusing the liver blood vessel characteristics by adopting dual-channel input, inputting the gray value of a liver image by a channel I, and performing normalization processing (px/256), thereby obtaining a blood vessel core region by a clustering algorithm;
and inputting the blood vessel texture information of the liver image after Hessian matrix processing by the channel two. Calculation of eigenvalues λ by Hessian matrix1And λ2And constructing a blood vessel measurement function according to the characteristic values:
wherein R isbDescribing the deviation from a speckle-like structure, S is the Frobenius matrix norm, β and c are the controlled linear filter metrics RbAnd a threshold value of S sensitivity.
Weighting and fusing the two channel results through a model to obtain a blood vessel core area image;
removing noise data through open operation;
and expanding the blood vessel image after the noise is removed by adopting an expansion algorithm, and pre-segmenting the blood vessel image and the peripheral area in the original liver image by taking the blood vessel image as a mask to form a new data set.
In S105 provided by the embodiment of the present invention, in the process that the information processing server transmits the processed image to the U-Net vessel segmentation algorithm server for calculation, the deep learning network based on the U-Net architecture of the vessel segmentation method improves the liver vessel recognition efficiency by improving the module structure, the loss function, the training strategy, and the like.
In S106 provided by the embodiment of the present invention, the information processing server transmits the examiner information and the blood vessel segmentation result to the three-dimensional liver blood vessel reconstruction model for liver blood vessel segmentation, specifically:
and (3) adopting a U-Net model training set, drawing and correcting labels of the training set by a medical image professional technician, and repeating the iterative training model until the result performance meets the preset target.
A U-Net architecture deep learning network model is characterized in that a training set label is composed of 0, 1 and 2, wherein 0 marks a picture background, 1 marks a liver blood vessel, and 2 marks a liver region at the periphery of the blood vessel.
The liver vessel segmentation adopts preprocessing liver data in an information processing server, inputs the processed image into a trained U-Net model through gRPC, and outputs a refined segmentation result of the liver vessel.
As shown in fig. 2, a liver vessel segmentation system based on multi-modal fusion and deep learning according to an embodiment of the present invention includes:
and the information and image acquisition module 1 is used for acquiring basic information and CT images of the examiner.
And the image information processing module 2 is used for carrying out preprocessing operation on the image to be subjected to liver blood vessel segmentation by the information processing server.
And in the liver blood vessel segmentation module 3, the information processing server transmits the processed communication to the blood vessel segmentation algorithm server for calculation, and the algorithm server transmits the result back to the information processing server.
And the blood vessel three-dimensional reconstruction module 4 is used for reconstructing the blood vessel image segmented by the model into a three-dimensional liver blood vessel tree model.
And the information storage module 5 is used for storing the segmented blood vessel data, the three-dimensional blood vessel model and the basic information of the CT examiner in a specific database and a file storage area.
The technical solution of the present invention is further described with reference to the following specific examples.
The information processing server retrieves the basic information of the examiner and the CT image sequence in DICOM format from the medical information system (such as PACS, RIS and the like) through the security port; the method comprises the steps of automatically adjusting window width and window level, converting the window level into a PNG (portable network generator) format gray image, cascading a liver segmentation module, obtaining a liver mask, and taking out a liver part according to the mask. Pre-dividing the blood vessel by a multi-modal image fusion algorithm; the data input adopts two channels, wherein the channel 1 inputs the clustering result of the gray value of the liver image, and the channel 2 inputs the blood vessel gradient information of the liver image after Hessian matrix processing; and carrying out operations such as denoising, expansion and the like on the result to form a new image. Through the gRPC, the information processing server transmits the processed image to the algorithm server for calculation, so as to obtain a refined segmentation result of the liver blood vessel, and the algorithm server transmits the result back to the information processing server. The information processing server stores the processed examiner information and the blood vessel segmentation result, and calls a liver blood vessel three-dimensional reconstruction model, and the three-dimensional reconstruction model returns a three-dimensional reconstruction result in a VTK format.
The invention relates to a hepatic blood vessel AI segmentation model based on a self-improved algorithm and a continuous feedback growth system carrying an algorithm data set. The subject algorithm of the present invention is based on a team self-research algorithm, the core of which is shown in FIG. 3. Acquiring basic information of an inspector and a CT image sequence from a security port by an information processing server; obtaining local information of the liver blood vessel by using a multi-modal image fusion algorithm, and removing noise interference; then, the information processing server transmits the processed image to an algorithm server with a high-performance GPU (graphics processing unit) through the gRPC (graphics processing unit), the algorithm server performs calculation by using a trained U-Net-based deep neural network model, and a calculation result is transmitted back to the information processing server; the information processing server stores the processed examiner information and the blood vessel segmentation result on one hand, and calls a blood vessel three-dimensional reconstruction module to reconstruct the blood vessel image segmented by the model into a three-dimensional liver blood vessel tree model and return the three-dimensional reconstruction result in a VTK format on the other hand. Some data sets with poor prediction results are collected and are put into the data sets for training after being corrected by a professional labeling team.
As shown in fig. 4, a schematic diagram illustrating comparison of image processing results by different algorithms in a liver blood vessel image fusion process is shown. FIG. 4(a) is an original liver image; calculation of eigenvalues λ using Hessian matrices1And λ2Constructing a blood vessel measurement function according to the characteristic value,
the image processed by this method is shown in fig. 4 (b); in the clustering algorithm for realizing the blood vessel pre-segmentation, the gray level and gradient characteristics of the image are input, the specified clustering number K is 3, and one type of result with the largest average value of various gray levels in the clustering result is extracted and shown in fig. 4 (c); the result of denoising the pre-segmented blood vessel by using the opening operation is shown in fig. 4 (d); setting the nucleus size at 9 × 9, and expanding the blood vessel shown in fig. 4(d), and the result is shown in fig. 4 (e); as shown in fig. 4(f), the result of the and operation in fig. 4(a) and 4(e) is the input image for model training and prediction.
The implementation process of the liver blood vessel segmentation algorithm is mainly divided into 4 stages, namely: the implementation process of separating the liver, extracting the local information of the blood vessel, segmenting the blood vessel by a U-Net model and performing 3D modeling on the blood vessel of the liver is shown in figure 5.
The first embodiment is as follows: in the aspects of identifying tiny blood vessels and segmenting the integrity of the blood vessels, the same training data set (200 patients who do CT scanning, about 15000 pieces of image data) is adopted, and the method of the invention and the method which only adopts the U-Net model are fully trained. The segmentation effect pairs of the two methods on the liver blood vessels are shown in fig. 6, wherein the first line is an original image, the second line only adopts the segmentation effect of the U-Net model, and the third line is a segmentation effect graph according to the embodiment of the invention. The red square frame in the figure circles the defect of only adopting the U-Net model for segmentation, and shows that the embodiment of the invention has certain advantages in the aspects of tiny blood vessel identification and blood vessel segmentation integrity.
Example two: in terms of noise interference on liver blood vessel identification, the same training data set (200 patients who do CT scanning, about 15000 image data) is adopted, and the method of the invention and the method only adopting the U-Net model are fully trained. The segmentation effect pairs of the two methods on the liver blood vessels are shown in fig. 7, wherein the first line is an original image, the second line only adopts the segmentation effect of the U-Net model, and the third line is a segmentation effect graph according to the embodiment of the invention. The red square in the figure circles out the error segmentation of the 'false positive' caused by the noise interference only by adopting the U-Net model, and the embodiment of the invention has stronger anti-noise interference performance.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code provided on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any modification, equivalent replacement, and improvement made by those skilled in the art within the technical scope of the present invention disclosed in the present invention should be covered within the scope of the present invention.
Claims (10)
1. A liver vessel segmentation method based on multi-mode fusion and deep learning is characterized by comprising the following steps:
the information processing server acquires basic information of a patient and a CT image sequence in a DICOM format from the security port;
the cascade liver segmentation module is used for acquiring a corresponding liver mask, automatically adjusting the window width and window level and converting the window width and the window level into a PNG (portable network group) format gray image and taking out a liver part according to the mask;
extracting gradient characteristics of the blood vessel texture through a Hessian matrix; obtaining local information of the blood vessel through a clustering algorithm; fusing and pre-dividing the two images through a model to obtain a local image of the blood vessel;
noise in the pre-segmented blood vessel is removed by adopting opening operation, the blood vessel area is expanded by using an expansion algorithm, and the blood vessel area and an original liver image are subjected to AND operation to construct a new liver image;
through the gRPC, the information processing server transmits the processed image to the U-Net blood vessel segmentation algorithm server for calculation, and the algorithm server transmits the result back to the information processing server;
the information processing server transmits the examiner information and the blood vessel segmentation result into the liver blood vessel three-dimensional reconstruction model, and the three-dimensional reconstruction model returns a three-dimensional reconstruction result in a VTK format.
2. The liver vessel segmentation method based on multi-modal fusion and deep learning of claim 1, wherein the specific process of automatically adjusting the window width and converting the window level into the PNG format gray scale image is as follows: the DICOM image window width and window level are adjusted in a self-adaptive mode, so that the liver blood vessel has the best developing effect, and a liver blood vessel image enhancement method at a pixel level is constructed.
3. The liver vessel segmentation method based on multi-modal fusion and deep learning as claimed in claim 1, wherein the step of taking out liver part according to mask comprises the following specific steps: and the cascade liver segmentation module is used for acquiring the corresponding liver mask and segmenting the liver region in the image sequence.
4. The liver vessel segmentation method based on multi-modal fusion and deep learning as claimed in claim 1, wherein the specific process of pre-segmenting the two by model fusion to obtain the vessel local image is as follows: fusing the liver blood vessel characteristics by adopting two-channel input, inputting the gray value of a liver image by a channel I, performing normalization processing, and further acquiring a blood vessel core region by a clustering algorithm;
inputting blood vessel texture information of the liver image after Hessian matrix processing by a channel II;
calculation of eigenvalues λ by Hessian matrix1And λ2And constructing a blood vessel measurement function according to the characteristic values:
5. the liver vessel segmentation method based on multi-modal fusion and deep learning of claim 4 is characterized in that the weighted fusion of the two channel results through the model obtains the vessel core region image; noise data are removed through open operation, and a model is a feature fusion function in a blood vessel pre-segmentation algorithm; and expanding the blood vessel image after the noise is removed by adopting an expansion algorithm, and pre-segmenting the blood vessel image and the peripheral area in the original liver image by taking the blood vessel image as a mask to form a new data set.
6. The liver vessel segmentation method based on multi-modal fusion and deep learning of claim 1, wherein the information processing server transmits the processed image to a U-Net vessel segmentation algorithm server for computation, and the deep learning network based on the U-Net architecture of the vessel segmentation method improves the liver vessel recognition efficiency by improving the module structure, the loss function, the training strategy, and so on.
7. The liver vessel segmentation method based on multi-modal fusion and deep learning as claimed in claim 1, wherein the information processing server transmits examiner information and vessel segmentation results into a liver vessel three-dimensional reconstruction model for liver vessel segmentation, specifically:
and (3) adopting a U-Net model training set, drawing and correcting labels of the training set by a medical image professional technician, and repeatedly iterating the training set until the result performance meets a preset target.
8. The liver vessel segmentation method based on multi-modal fusion and deep learning of claim 7, wherein the U-Net model is a U-Net architecture deep learning network model, the training set label is composed of 0, 1 and 2, 0 marks the picture background, 1 marks the liver vessel, and 2 marks the liver region around the vessel.
9. The liver vessel segmentation method based on multi-modal fusion and deep learning of claim 7 is characterized in that the liver vessel segmentation adopts pre-processed liver data in an information processing server, processed images are input into a trained U-Net model through a gPC, and refined segmentation results of liver vessels are output.
10. A liver vessel segmentation system based on multi-modal fusion and deep learning for implementing the liver vessel segmentation method based on multi-modal fusion and deep learning according to any one of claims 1 to 9, wherein the liver vessel segmentation system based on multi-modal fusion and deep learning comprises:
the information and image acquisition module is used for acquiring basic information and CT images of an examiner;
the image information processing module is used for preprocessing the image to be subjected to liver blood vessel segmentation by the information processing server;
the liver blood vessel segmentation module is used for transmitting the processed communication to the blood vessel segmentation algorithm server for calculation by the information processing server, and transmitting the result back to the information processing server by the algorithm server;
the blood vessel three-dimensional reconstruction module is used for reconstructing the blood vessel image segmented by the model into a three-dimensional liver blood vessel tree model;
and the information storage module is used for storing the segmented blood vessel data, the three-dimensional blood vessel model and the basic information of the CT examiner in a specific database and a file storage area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110162875.1A CN113160120A (en) | 2021-02-05 | 2021-02-05 | Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110162875.1A CN113160120A (en) | 2021-02-05 | 2021-02-05 | Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113160120A true CN113160120A (en) | 2021-07-23 |
Family
ID=76883151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110162875.1A Pending CN113160120A (en) | 2021-02-05 | 2021-02-05 | Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113160120A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114359205A (en) * | 2021-12-29 | 2022-04-15 | 推想医疗科技股份有限公司 | Head and neck blood vessel analysis method and device, storage medium and electronic equipment |
| CN116863146A (en) * | 2023-06-09 | 2023-10-10 | 强联智创（北京）科技有限公司 | Method, apparatus and storage medium for extracting hemangio features |
| CN116862930A (en) * | 2023-09-04 | 2023-10-10 | 首都医科大学附属北京天坛医院 | Cerebral vessel segmentation method, device, equipment and storage medium suitable for multiple modes |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105825516A (en) * | 2016-03-25 | 2016-08-03 | 上海慧达医疗器械有限公司 | DICOM image blood flow analysis system |
| US20200211188A1 (en) * | 2018-12-28 | 2020-07-02 | Shanghai United Imaging Intelligence Co., Ltd. | Systems and methods for image processing |
| CN111429404A (en) * | 2020-03-04 | 2020-07-17 | 张峰 | Imaging system and method for detecting cardiovascular and cerebrovascular vessels |
| CN112184659A (en) * | 2020-09-24 | 2021-01-05 | 上海健康医学院 | Lung image processing method, device and equipment |
-
2021
- 2021-02-05 CN CN202110162875.1A patent/CN113160120A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105825516A (en) * | 2016-03-25 | 2016-08-03 | 上海慧达医疗器械有限公司 | DICOM image blood flow analysis system |
| US20200211188A1 (en) * | 2018-12-28 | 2020-07-02 | Shanghai United Imaging Intelligence Co., Ltd. | Systems and methods for image processing |
| CN111429404A (en) * | 2020-03-04 | 2020-07-17 | 张峰 | Imaging system and method for detecting cardiovascular and cerebrovascular vessels |
| CN112184659A (en) * | 2020-09-24 | 2021-01-05 | 上海健康医学院 | Lung image processing method, device and equipment |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114359205A (en) * | 2021-12-29 | 2022-04-15 | 推想医疗科技股份有限公司 | Head and neck blood vessel analysis method and device, storage medium and electronic equipment |
| CN114359205B (en) * | 2021-12-29 | 2022-11-01 | 推想医疗科技股份有限公司 | Head and neck blood vessel analysis method and device, storage medium and electronic equipment |
| CN116863146A (en) * | 2023-06-09 | 2023-10-10 | 强联智创（北京）科技有限公司 | Method, apparatus and storage medium for extracting hemangio features |
| CN116863146B (en) * | 2023-06-09 | 2024-03-08 | 强联智创（北京）科技有限公司 | Method, apparatus and storage medium for extracting hemangio features |
| CN116862930A (en) * | 2023-09-04 | 2023-10-10 | 首都医科大学附属北京天坛医院 | Cerebral vessel segmentation method, device, equipment and storage medium suitable for multiple modes |
| CN116862930B (en) * | 2023-09-04 | 2023-11-28 | 首都医科大学附属北京天坛医院 | Cerebral vessel segmentation method, device, equipment and storage medium suitable for multiple modes |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Automatic nodule detection for lung cancer in CT images: A review | |
| Bellver et al. | Detection-aided liver lesion segmentation using deep learning | |
| Selver et al. | Patient oriented and robust automatic liver segmentation for pre-evaluation of liver transplantation | |
| CN113160120A (en) | Liver blood vessel segmentation method and system based on multi-mode fusion and deep learning | |
| CN113420826B (en) | Liver focus image processing system and image processing method | |
| Fan et al. | Lung nodule detection based on 3D convolutional neural networks | |
| Maiora et al. | Abdominal CTA image analisys through active learning and decision random forests: Aplication to AAA segmentation | |
| Ye et al. | Medical image diagnosis of prostate tumor based on PSP-Net+ VGG16 deep learning network | |
| Biswas et al. | Chest X-ray enhancement to interpret pneumonia malformation based on fuzzy soft set and Dempster–Shafer theory of evidence | |
| Feng et al. | Deep learning for chest radiology: a review | |
| CN111292307A (en) | Digestive system gallstone recognition method and positioning method | |
| Osadebey et al. | Three-stage segmentation of lung region from CT images using deep neural networks | |
| CN115908297A (en) | Topology knowledge-based blood vessel segmentation modeling method in medical image | |
| Javidi et al. | Retinal image assessment using bi-level adaptive morphological component analysis | |
| Yang et al. | RADCU-Net: Residual attention and dual-supervision cascaded U-Net for retinal blood vessel segmentation | |
| Wang et al. | Deep learning based fetal middle cerebral artery segmentation in large-scale ultrasound images | |
| Kareem et al. | Skin lesions classification using deep learning techniques | |
| Dong et al. | Deep supervision adversarial learning network for retinal vessel segmentation | |
| Banerjee et al. | A CADe system for gliomas in brain MRI using convolutional neural networks | |
| Affane et al. | Robust deep 3-d architectures based on vascular patterns for liver vessel segmentation | |
| Wang et al. | Automated segmentation of breast arterial calcifications from digital mammography | |
| Wang et al. | Triplanar convolutional neural network for automatic liver and tumor image segmentation | |
| US20230410291A1 (en) | Method and system for segmenting and identifying at least one tubular structure in medical images | |
| Dickson et al. | A Dual Channel Multiscale Convolution U-Net Methodfor Liver Tumor Segmentation from Abdomen CT Images | |
| CN115222674A (en) | Detection device for intracranial aneurysm rupture risk based on multi-dimensional feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210723 |
|
| RJ01 | Rejection of invention patent application after publication |