CN112419271B - Image segmentation method, device and computer readable storage medium - Google Patents
Image segmentation method, device and computer readable storage medium Download PDFInfo
- Publication number
- CN112419271B CN112419271B CN202011325510.8A CN202011325510A CN112419271B CN 112419271 B CN112419271 B CN 112419271B CN 202011325510 A CN202011325510 A CN 202011325510A CN 112419271 B CN112419271 B CN 112419271B
- Authority
- CN
- China
- Prior art keywords
- image segmentation
- blood vessel
- image
- full
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003709 image segmentation Methods 0.000 title claims abstract description 120
- 238000000034 method Methods 0.000 title claims abstract description 86
- 210000004204 blood vessel Anatomy 0.000 claims abstract description 83
- 238000012549 training Methods 0.000 claims abstract description 69
- 238000013135 deep learning Methods 0.000 claims abstract description 38
- 230000011218 segmentation Effects 0.000 claims abstract description 31
- 238000013528 artificial neural network Methods 0.000 claims description 38
- 238000012360 testing method Methods 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 21
- 238000005070 sampling Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 230000003213 activating effect Effects 0.000 claims 4
- 238000013527 convolutional neural network Methods 0.000 abstract description 9
- 230000000875 corresponding effect Effects 0.000 description 18
- 238000010586 diagram Methods 0.000 description 14
- 238000004590 computer program Methods 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000007796 conventional method Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000002198 surface plasmon resonance spectroscopy Methods 0.000 description 2
- 238000002604 ultrasonography Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000001727 in vivo Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 230000002792 vascular Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10132—Ultrasound image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30101—Blood vessel; Artery; Vein; Vascular
Abstract
The invention discloses an image segmentation method, an image segmentation device and a computer readable storage medium, wherein a training sample set is constructed based on a target blood vessel image data set and a corresponding label set; training a mixed deep learning network comprising a full convolutional neural network and a U-net by adopting a training sample set to obtain an image segmentation model; and inputting the blood vessel image to be segmented into an image segmentation model for image segmentation. By implementing the invention, the mixed deep learning network is adopted to segment the blood vessel image, the integral characteristic of the image is emphasized, and the blood vessel segmentation precision and the robustness are effectively improved.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to an image segmentation method, apparatus, and computer readable storage medium.
Background
Photoacoustic imaging technology has the ability to recognize molecular specificity and to achieve lateral resolution at the cellular level under the light diffraction limit, and is widely used in vascular imaging. The blood vessel image carries basic medical information, and can provide effective guidance for professional diagnosis.
Vessel image segmentation is an important task in biomedical image analysis, and modern image processing technology makes a good contribution to vessel segmentation. At present, a threshold segmentation method, a region growing method, a maximum entropy method and a k-means clustering method are generally adopted for segmenting the blood vessel image in the prior art, and the problems of limited segmentation accuracy exist in the methods.
Disclosure of Invention
The main objective of the embodiments of the present invention is to provide an image segmentation method, an image segmentation device and a computer readable storage medium, which can at least solve the problem of limited segmentation accuracy during segmentation of blood vessel images in the related art.
To achieve the above object, a first aspect of an embodiment of the present invention provides an image segmentation method, including:
constructing a training sample set based on the target blood vessel image data set and the corresponding label set; the target blood vessel image data set comprises a plurality of blood vessel image samples, and the label set comprises classification labels corresponding to the blood vessel image samples;
training a preset mixed deep learning network by adopting the training sample set to obtain an image segmentation model; the mixed deep learning network comprises a first full-convolution neural network and a second full-convolution neural network, wherein the first full-convolution neural network executes two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, the convolution layer of the second full-convolution neural network is of a U-shaped structure, and the second full-convolution neural network executes four up-sampling operations with the step length of 2 and downsampling operations respectively in the deconvolution process;
and inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
To achieve the above object, a second aspect of an embodiment of the present invention provides an image segmentation apparatus, including:
the construction module is used for constructing a training sample set based on the target blood vessel image data set and the corresponding label set; the target blood vessel image data set comprises a plurality of blood vessel image samples, and the label set comprises classification labels corresponding to the blood vessel image samples;
the training module is used for training a preset mixed deep learning network by adopting the training sample set to obtain an image segmentation model; the mixed deep learning network comprises a first full-convolution neural network and a second full-convolution neural network, wherein the first full-convolution neural network executes two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, the convolution layer of the second full-convolution neural network is of a U-shaped structure, and the second full-convolution neural network executes four up-sampling operations with the step length of 2 and downsampling operations respectively in the deconvolution process;
the segmentation module is used for inputting the blood vessel image to be segmented into the image segmentation model to carry out image segmentation.
To achieve the above object, a third aspect of an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a communication bus;
the communication bus is used for realizing connection communication between the processor and the memory;
the processor is configured to execute one or more programs stored in the memory to implement the steps of any one of the image segmentation methods described above.
To achieve the above object, a fourth aspect of the embodiments of the present invention provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps of any one of the above-described image segmentation methods.
According to the image segmentation method, the image segmentation device and the computer-readable storage medium provided by the embodiment of the invention, a training sample set is constructed based on a target blood vessel image data set and a corresponding label set; training a mixed deep learning network comprising a full convolutional neural network and a U-net by adopting a training sample set to obtain an image segmentation model; and inputting the blood vessel image to be segmented into an image segmentation model for image segmentation. By implementing the invention, the mixed deep learning network is adopted to segment the blood vessel image, the integral characteristic of the image is emphasized, and the blood vessel segmentation precision and the robustness are effectively improved.
Additional features and corresponding effects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other drawings may be obtained from them without inventive effort for a person skilled in the art.
Fig. 1 is a basic flow diagram of an image segmentation method according to a first embodiment of the present invention;
fig. 2 is a schematic diagram of a network structure of an FCN according to a first embodiment of the present invention;
fig. 3 is a schematic diagram of a network structure of a U-net according to a first embodiment of the present invention;
fig. 4 is a schematic structural diagram of a hybrid deep learning network according to a first embodiment of the present invention;
FIG. 5 is a schematic diagram showing the result of the conventional image segmentation method according to the first embodiment of the present invention;
fig. 6 is an evaluation index box diagram of the deep learning method according to the first embodiment of the present invention;
fig. 7 is a visual schematic diagram of an image segmentation method based on a deep learning network according to a first embodiment of the present invention;
fig. 8 is a schematic structural diagram of an image segmentation apparatus according to a second embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more comprehensible, the technical solutions in the embodiments of the present invention will be clearly described in conjunction with the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
First embodiment:
first, the present embodiment describes a conventional image segmentation optimization algorithm, which mainly includes the following:
the threshold segmentation method is to select an appropriate threshold pixel intensity as a segmentation line. Thus, a clear classification between foreground and background can be observed. Two major drawbacks of the thresholding method are the high sensitivity of thresholding and the lack of morphological information considerations.
The Region Growing (RG) method is a process of aggregating pixels or sub-regions into a larger Region according to a predefined standard. The basic idea is to start with a set of seed points manually selected as an initial point. The seed point may be a single pixel or a small region. The first step is to combine adjacent pixels or regions with similar properties to form a new growth seed point. The next step is to repeat the above process until the region converges (no other seed points can be found). It can clearly be seen that a key problem with RG is that the choice of initial growth point cannot be determined empirically.
The maximum entropy method is used to describe the degree of uncertainty of information, and the essence of the maximum entropy principle is that the probability of occurrence of a certain event in the system satisfies all known constraints without making assumptions about any unknown information, in other words, regarding the unknown information as equal probabilities. In the maximum entropy image segmentation, the total entropy of the image under each segmentation threshold is calculated, the maximum entropy is found, and the segmentation threshold corresponding to the maximum entropy is used as a final threshold. Pixels in the image having a gray level greater than this threshold are classified as foreground, otherwise as background.
The K-means clustering method is an iterative algorithm and mainly comprises the following 4 steps: a) Randomly selecting a group of K-type initial centroids; b) Labeling each sample according to the distance between the sample and each clustering center; c) Calculating and updating a new centroid for each class; d) Repeating steps b) and c) until the center converges.
However, the above-mentioned conventional image segmentation optimization algorithms focus on local features of the image, and do not consider spatial information of the image, and are sub-optimal subdivision solutions.
In order to solve the technical problem that the segmentation accuracy is limited when the above-mentioned image segmentation optimization algorithm performs the segmentation of the blood vessel image in the related art, the present embodiment provides an image segmentation method, as shown in fig. 1, which is a basic flow diagram of the image segmentation method provided in the present embodiment, where the image segmentation method provided in the present embodiment includes the following steps:
step 101, constructing a training sample set based on the target blood vessel image data set and the corresponding label set.
Specifically, the target blood vessel image data set of the embodiment includes a plurality of blood vessel image samples, and the label set includes classification labels corresponding to the blood vessel image samples.
In this embodiment, an in vivo blood vessel image may be acquired from the ear of a webster mouse in switzerland using an OR-PAM system that employs a surface plasmon resonance sensor as an ultrasound detector. The maximum amplitude of each PA a-line is projected to the depth direction, and a Maximum Amplitude Projection (MAP) image is reconstructed. The lateral resolution of the system is around 4.5um, so that the blood vessel is visualized. The surface plasmon resonance sensor of this embodiment can respond to ultrasound and a wide bandwidth, and a determined depth resolution OR-PAM system of around 7.6um takes about 10 minutes to capture a 512 x 512 pixel blood vessel image.
In addition, all the label sets corresponding to the data set images can be obtained through manual labeling of graphic interface image annotation software Labelme developed by the Massachusetts institute of technology.
In an alternative implementation of the present embodiment, before constructing the training sample set based on the target blood vessel image dataset and the corresponding label set, the method further includes: obtaining effective blood vessel image samples with image quality meeting preset quality requirements from a limited number of blood vessel image samples; and carrying out data enhancement processing on the effective blood vessel image samples, and constructing a target blood vessel image data set with the number of samples meeting the preset number requirement.
In particular, the images obtained in an OR-PAM system are typically limited, and some images need to be discarded due to quality problems such as noise, break points OR discontinuities. Because the number of images provided by the PA system is insufficient, the embodiment can perform image sample expansion by adopting data enhancement methods such as cutting, overturning, mapping and the like on the obtained effective blood vessel image so as to avoid the problems of over fitting and low training precision in the subsequent training of the model. In addition, the present embodiment may crop the dataset image to 256×256 pixels to speed up the training process. And, can also choose some of them as the test set at random from the final dataset, the rest of pictures put into training set and verification set at random.
And 102, training a preset mixed deep learning network by adopting a training sample set to obtain an image segmentation model.
In particular, convolutional Neural Networks (CNNs) are a powerful visual model that can produce a hierarchy of features. The application of CNN in semantic segmentation has exceeded the most advanced level. While the former model of GoogleNet, VGG, alexNet, etc., shows better performance, none of the models is capable of end-to-end training because of the presence of a fully connected layer prior to network output and consistent tag size dimensions. In addition, the fully connected layer of the network expands the extracted features into one-dimensional vectors, discarding the spatial information of the feature map extracted from each map. The full convolution network (Full Convolutional Network, FCN) replaces the full connection layer with a convolution layer, avoiding pre-and post-processing of the image, thereby preserving spatial information.
The hybrid deep learning network of this embodiment includes a first fully convolutional neural network (FCN) and a second fully convolutional neural network (U-net), both networks having a convolutional kernel of size 3*3. The method comprises the steps that a first full convolution neural network performs two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, a convolution layer of a second full convolution neural network is of a U-shaped structure, and the second full convolution neural network performs four up-sampling operations and four down-sampling operations with the step length of 2 in the deconvolution process respectively.
As shown in fig. 2, which is a schematic diagram of the network structure of the FCN provided in this embodiment, except for the last layer of the network, a nonlinear correction unit ReLu is added to each convolution kernel, and there is no significant difference between up-sampling and deconvolution. Therefore, the network adopts an up-sampling method to reduce the number of training parameters, and simultaneously, two convolution operations and one dropout block are used to prevent over-fitting in the process of convolution to deconvolution conversion.
As shown in fig. 3, the network structure of the U-net provided in this embodiment is a model developed based on FCN, and has strong robustness, and has a wide application field in academia and industry. Although both networks are full convolution layers, subtle differences can be found in the connection layer, the U-net combines the low-level features of the encoding part of the network with the high-level features of the decoding part, which effectively avoids the loss of characteristics caused by the pooling layer in the network. In addition, the network replaces the additional layer with the connection layer, blends the low-layer features with the high-layer features, rather than simply adding corresponding pixels, thereby expanding the channel capacity.
As shown in fig. 4, which is a schematic structural diagram of the hybrid deep learning network provided in this embodiment, it should also be noted that, in the hybrid deep learning network Hy-Net based on FCN and U-Net in this embodiment, the results from FCN and U-Net are combined with a connection block (connection) and an activation block (sigmoid). The final probability map, i.e., the network output, is processed by the sigmoid function with the default threshold set to 0.5, indicating that map entries greater than 0.5 are classified as foreground and the remaining entries are considered background.
In an optional implementation manner of this embodiment, training a preset hybrid deep learning network by using a training sample set to obtain an image segmentation model includes: setting the initial learning rate to 0.0001, setting the minimum batch size to 2, and performing iterative training on a preset mixed deep learning network by adopting a training sample set according to a random gradient descent algorithm; and when the loss function value obtained by iterative training is converged to a preset function value, determining the network model obtained by the current iterative training as a trained image segmentation model.
Specifically, in this embodiment, the training process of the network is repeated for multiple iterations to optimize, and the output obtained by predicting each training of the neural network will be calculated as a Loss Function (Loss Function) with the classification label marked by the sample, where the Loss Function may be cross entropy Loss; and then updating the trainable parameters in the network by adopting a random gradient descent algorithm, adjusting parameters such as the weight of the neural network, and the like, reducing the loss function value of the next iteration, judging that the model convergence condition is met when the loss function value meets the preset standard, namely finishing the training process of the whole neural network model, otherwise, continuing to train the next iteration until the model convergence condition is met.
And step 103, inputting the blood vessel image to be segmented into an image segmentation model for image segmentation.
Specifically, the Hy-Net in the embodiment optimizes the results of the two models by combining the characteristic outputs of the FCN and the U-Net, and effectively avoids the uniqueness of the output of the single model. Compared with the conventional method, the image segmentation model of the embodiment fully considers the integral characteristics of the image due to the use of convolution kernels (feature descriptors) and the parameter sharing characteristics of the kernels, and has higher accuracy and robustness when segmenting the blood vessel image.
In an optional implementation manner of this embodiment, before inputting the blood vessel image to be segmented into the image segmentation model for image segmentation, the method further includes: inputting a preset test sample set into an image segmentation model to obtain a classification label of test output; and performing correlation calculation on the classification labels output by the test and the classification labels marked by the test sample set. Correspondingly, when the correlation is larger than a preset correlation threshold, determining that the image segmentation model is effective, and then executing the step of inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
Specifically, in this embodiment, after the image segmentation model is trained, the validity of the image segmentation model is verified by using the test sample set, that is, the test sample set is input into the trained image segmentation model, then the relevance of the output classification label and the original classification label in the test sample set is compared to determine the validity of the model, when the relevance between the test data and the original data is greater than a preset threshold, the trained image segmentation model is determined to be a valid and correct model, and then the image to be segmented can be segmented by using the valid image segmentation model; otherwise, the training of the trained image segmentation model is insufficient, and the trained image segmentation model needs to be further optimized to ensure the image segmentation accuracy in the actual use process.
It should be noted that the following four indexes are: dice Coefficient (DC), intersection (IoU), sensitivity (Sen), and accuracy (Acc) were applied to each test experiment of the present embodiment to quantify the performance of the experiment of the present embodiment on various segmentation methods.
In the present embodiment, DC, ioU, sen and Acc of four conventional non-deep learning methods are compared. The threshold segmentation method selects a pixel intensity threshold 100 as a threshold, and the segmentation precision reaches 97.40%. The remaining 3 indices produced were poor with average values of 70.98%, 56.09% and 61.64%, respectively.
In addition, the region growing method requires the selection of an initial seed point. Thus, the image pixels are ordered in ascending order of pixel curvature. The pixel with the smallest curvature is taken as an initial seed point. This selection method ensures that the algorithm starts from the region of the image that is the smoothest, reducing the number of segmentations. The threshold (maximum density distance between 8 pixels around the centroid) is set to 0.8. The evaluation scores for DC, ioU, sen and Acc based on the region growing method were 64.30%, 49.70%, 51.96% and 97.26%, respectively.
Maximum entropy method reaches 50.77%, 35.33%, 95.95% and 91.02% for DC, ioU, sen and Acc, respectively.
K-mean cluster segmentation was achieved in MATLAB using imseg function, yielding DC, ioU, sen and Acc as 75.21%, 60.93%, 70.92% and 97.59%, respectively.
A visual representation of the results of a conventional image segmentation method is shown in fig. 5 to better illustrate the major differences in segmentation of the various methods. The raw test image is shown in fig. 5 (a), represented by RGB channels, representing raw data captured by the PA imaging system. The image segmentation results of the manual labeling are shown in fig. 5 (b), and the image segmentation results of the threshold segmentation method, the region growing method, the maximum entropy method and the K-mean clustering method are shown in fig. 5 (c) to (f), respectively. As is clear from columns 3, 4, 5 in fig. 5, the conventional methods work well for bright images with sharp contour boundaries, but are less effective for images with unclear boundaries (columns 1, 2, 6). It can be seen that the four conventional segmentation methods described above lack robustness and generalization.
In addition, in the present embodiment, DC, ioU, sen and Acc of three deep learning methods (i.e., FCN, U-Net, hy-Net) are further compared. As shown in fig. 6, the evaluation index box diagram of the deep learning method provided in the present embodiment, where (a) to (d) in fig. 6 correspond to DC (which may also be expressed as dic in fig. 6), ioU, sen, and Acc, respectively. Wherein, the minimum values of the performances of the FCN pair DC, ioU, sen, acc are 60.31%, 43.17%, 53.23% and 92.82%, and the maximum values thereof are 84.07%, 72.52%, 87.43% and 99.71% respectively. The minimum values of U-net pair DC, ioU, sen, acc performance are 66.38%, 49.68%, 52.20% and 96.03%, respectively, and the maximum values are 96.77%, 93.75%, 98.29% and 99.94% respectively. The minimum performance values of Hy-Net on DC, ioU, sen, acc are 69.83%, 53.65%, 75.47%, 95.32%, respectively, with maximum values of 94.67%, 89.87%, 97.49%, 99.90%, respectively. The median dice for FCN, U-Net and Hy-Net were 66.32%, 83.79% and 85.13%, respectively, the median IoU for the three deep learning methods were 49.61%, 72.10% and 74.11%, respectively, the median Sen was 69.57%, 83.36% and 90.62%, and the median Acc was 96.38%, 98.11% and 98.18%, respectively.
In the deep learning method, the FCN has the worst performance, the U-Net times, and the Hy-Net has the best performance. Specifically, U-net performs 13.71%, 17.97%, 13.37% and 1.46% higher than FCN; hy-Net exhibits 15.34%, 20.05%, 18.62% and 1.55% higher performance than FCN; hy-Net exhibits 1.63%, 2.08%, 5.25% and 0.09% higher performance than U-Net.
Fig. 7 is a visual diagram of an image segmentation method based on a deep learning network, and fig. 7 (a) to (c) show the image segmentation results of FCN, U-Net and Hy-Net, respectively. The visualization results show that Hy-Net can be highly overlapped with the tag, either in large or small containers.
From the above quantification and visualization results, it can be seen that the performance of the Hy-Net is superior to that of FCN and U-Net, and has good stability and robustness. It is mainly shown that both FCN and U-net are segmented starved. This phenomenon can be further explained from the following two aspects: firstly, whether the super-parameters are adjusted differently, the iteration times are increased or the training set size is increased, the FCN and the U-net have the characteristic limitation of the model; secondly, the results of the two models are optimized by combining the characteristic output of the FCN and the U-Net by the Hy-Net, so that the uniqueness of the output of the single model is effectively avoided.
Furthermore, it should be noted that the uncertainty in the choice of the binarization threshold of the network is large, which may lead to under-segmentation or over-segmentation, and in this embodiment, a set of thresholds is tested, and excellent results are obtained according to the following configuration (FCN: 80, U-Net:100, hy-Net: 150).
In summary, according to the deep learning network (Hy-Net) for PA image vessel segmentation provided in the present embodiment, from the above evaluation results, the method of the present embodiment can obtain higher accuracy and robustness compared with the conventional method, and in addition, hy-Net is significantly better than the four evaluation indexes of FCN and U-Net.
According to the image segmentation method provided by the embodiment of the invention, a training sample set is constructed based on a target blood vessel image data set and a corresponding label set; training a mixed deep learning network comprising a full convolutional neural network and a U-net by adopting a training sample set to obtain an image segmentation model; and inputting the blood vessel image to be segmented into an image segmentation model for image segmentation. By implementing the invention, the mixed deep learning network is adopted to segment the blood vessel image, the integral characteristic of the image is emphasized, and the blood vessel segmentation precision and the robustness are effectively improved.
Second embodiment:
in order to solve the technical problem that the segmentation accuracy is limited when the image segmentation optimization algorithm in the related art performs the segmentation of the blood vessel image, the embodiment shows an image segmentation apparatus, specifically please refer to fig. 8, where the image segmentation apparatus of the embodiment includes:
a construction module 801, configured to construct a training sample set based on the target blood vessel image dataset and the corresponding label set; the target blood vessel image data set comprises a plurality of blood vessel image samples, and the label set comprises classification labels corresponding to the blood vessel image samples;
the training module 802 is configured to train a preset hybrid deep learning network by using a training sample set to obtain an image segmentation model; the mixed deep learning network comprises a first full-convolution neural network and a second full-convolution neural network, wherein the first full-convolution neural network executes two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, the convolution layer of the second full-convolution neural network is of a U-shaped structure, and the second full-convolution neural network respectively executes four up-sampling operations with the step length of 2 and one down-sampling operation in the deconvolution process;
the segmentation module 803 is configured to input the blood vessel image to be segmented into an image segmentation model for image segmentation.
In some implementations of the present embodiment, the building block 801 is further configured to: before a training sample set is constructed based on a target blood vessel image data set and a corresponding label set, acquiring effective blood vessel image samples with image quality meeting preset quality requirements from a limited number of blood vessel image samples; and carrying out data enhancement processing on the effective blood vessel image samples, and constructing a target blood vessel image data set with the number of samples meeting the preset number requirement.
In some implementations of this embodiment, training module 802 is specifically configured to: setting the initial learning rate to 0.0001, setting the minimum batch size to 2, and performing iterative training on a preset mixed deep learning network by adopting a training sample set according to a random gradient descent algorithm; and when the loss function value obtained by iterative training is converged to a preset function value, determining the network model obtained by the current iterative training as a trained image segmentation model.
In some implementations of the present embodiment, the image segmentation apparatus further includes: the testing module is used for inputting a preset testing sample set into the image segmentation model before inputting the blood vessel image to be segmented into the image segmentation model for image segmentation, so as to obtain a classification label of test output; performing correlation degree calculation on the classification labels output by the test and the classification labels marked by the test sample set; and when the correlation degree is larger than a preset correlation degree threshold value, determining that the image segmentation model is effective. Correspondingly, the splitting module 803 is specifically configured to: and when the image segmentation model is effective, inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
It should be noted that, the image segmentation method in the foregoing embodiment may be implemented based on the image segmentation apparatus provided in the present embodiment, and those skilled in the art can clearly understand that, for convenience and brevity of description, the specific working process of the image segmentation apparatus described in the present embodiment may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
By adopting the image segmentation device provided by the embodiment, a training sample set is constructed based on the target blood vessel image data set and the corresponding label set; training a mixed deep learning network comprising a full convolutional neural network and a U-net by adopting a training sample set to obtain an image segmentation model; and inputting the blood vessel image to be segmented into an image segmentation model for image segmentation. By implementing the invention, the mixed deep learning network is adopted to segment the blood vessel image, the integral characteristic of the image is emphasized, and the blood vessel segmentation precision and the robustness are effectively improved.
Third embodiment:
the present embodiment provides an electronic device, as shown in fig. 9, which includes a processor 901, a memory 902, and a communication bus 903, wherein: the communication bus 903 is used to enable connection communication between the processor 901 and the memory 902; the processor 901 is configured to execute one or more computer programs stored in the memory 902 to implement at least one step of the image segmentation method in the above-described embodiment.
The present embodiments also provide a computer-readable storage medium including volatile or nonvolatile, removable or non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media includes, but is not limited to, RAM (Random Access Memory ), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory, charged erasable programmable Read-Only Memory), flash Memory or other Memory technology, CD-ROM (Compact Disc Read-Only Memory), digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
The computer readable storage medium in this embodiment may be used to store one or more computer programs, where the stored one or more computer programs may be executed by a processor to implement at least one step of the method in the first embodiment.
The present embodiment also provides a computer program which can be distributed on a computer readable medium and executed by a computable device to implement at least one step of the method of the above embodiment; and in some cases at least one of the steps shown or described may be performed in a different order than that described in the above embodiments.
The present embodiment also provides a computer program product comprising computer readable means having stored thereon a computer program as shown above. The computer readable means in this embodiment may comprise a computer readable storage medium as shown above.
It will be apparent to one skilled in the art that all or some of the steps of the methods, systems, functional modules/units in the apparatus disclosed above may be implemented as software (which may be implemented in computer program code executable by a computing apparatus), firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit.
Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, computer program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and may include any information delivery media. Therefore, the present invention is not limited to any specific combination of hardware and software.
The foregoing is a further detailed description of embodiments of the invention in connection with the specific embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the invention, and these should be considered to be within the scope of the invention.
Claims (10)
1. An image segmentation method, comprising:
constructing a training sample set based on the target blood vessel image data set and the corresponding label set; the target blood vessel image data set comprises a plurality of blood vessel image samples, and the label set comprises classification labels corresponding to the blood vessel image samples;
training a preset mixed deep learning network by adopting the training sample set to obtain an image segmentation model; the mixed deep learning network comprises a first full-convolution neural network, a second full-convolution neural network, a connecting block and an activating block, wherein the first full-convolution neural network is an FCN, the second full-convolution neural network is a U-net, the outputs of the first full-convolution neural network and the second full-convolution neural network are used as the inputs of the connecting block, the inputs of the connecting block are used as the inputs of the activating block, the first full-convolution neural network executes two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, the convolution layer of the second full-convolution neural network is of a U-shaped structure, and the second full-convolution neural network executes four up-sampling operations with the step length of 2 and down-sampling operations in the deconvolution process respectively;
and inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
2. The image segmentation method as set forth in claim 1, further comprising, prior to constructing the training sample set based on the target vessel image dataset and the corresponding label set:
obtaining effective blood vessel image samples with image quality meeting preset quality requirements from a limited number of blood vessel image samples;
and carrying out data enhancement processing on the effective blood vessel image samples, and constructing the target blood vessel image data set with the number of samples meeting the preset number requirement.
3. The image segmentation method as set forth in claim 1, wherein the training the preset hybrid deep learning network with the training sample set to obtain the image segmentation model comprises:
setting the initial learning rate to 0.0001, setting the minimum batch size to 2, and adopting the training sample set to perform iterative training on a preset mixed deep learning network according to a random gradient descent algorithm;
and when the loss function value obtained by iterative training is converged to a preset function value, determining the network model obtained by the current iterative training as a trained image segmentation model.
4. An image segmentation method according to any one of claims 1 to 3, wherein before inputting the blood vessel image to be segmented into the image segmentation model for image segmentation, the method further comprises:
inputting a preset test sample set into the image segmentation model to obtain a classification label of test output;
performing correlation degree calculation on the classification labels output by the test and the classification labels marked by the test sample set;
and when the correlation is greater than a preset correlation threshold, determining that the image segmentation model is effective, and then executing the step of inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
5. An image dividing apparatus, comprising:
the construction module is used for constructing a training sample set based on the target blood vessel image data set and the corresponding label set; the target blood vessel image data set comprises a plurality of blood vessel image samples, and the label set comprises classification labels corresponding to the blood vessel image samples;
the training module is used for training a preset mixed deep learning network by adopting the training sample set to obtain an image segmentation model; the mixed deep learning network comprises a first full-convolution neural network, a second full-convolution neural network, a connecting block and an activating block, wherein the first full-convolution neural network is an FCN, the second full-convolution neural network is a U-net, the outputs of the first full-convolution neural network and the second full-convolution neural network are used as the inputs of the connecting block, the inputs of the connecting block are used as the inputs of the activating block, the first full-convolution neural network executes two up-sampling operations with the step length of 2 and one single-step operation with the step length of 8 in the deconvolution process, the convolution layer of the second full-convolution neural network is of a U-shaped structure, and the second full-convolution neural network executes four up-sampling operations with the step length of 2 and down-sampling operations in the deconvolution process respectively;
the segmentation module is used for inputting the blood vessel image to be segmented into the image segmentation model to carry out image segmentation.
6. The image segmentation apparatus as set forth in claim 5, wherein the build module is further configured to: before a training sample set is constructed based on a target blood vessel image data set and a corresponding label set, acquiring effective blood vessel image samples with image quality meeting preset quality requirements from a limited number of blood vessel image samples; and carrying out data enhancement processing on the effective blood vessel image samples, and constructing the target blood vessel image data set with the number of samples meeting the preset number requirement.
7. The image segmentation apparatus as set forth in claim 5, wherein the training module is configured to: setting the initial learning rate to 0.0001, setting the minimum batch size to 2, and adopting the training sample set to perform iterative training on a preset mixed deep learning network according to a random gradient descent algorithm; and when the loss function value obtained by iterative training is converged to a preset function value, determining the network model obtained by the current iterative training as a trained image segmentation model.
8. The image segmentation apparatus as set forth in any one of claims 5-7, further comprising: a test module;
the testing module is used for inputting a preset testing sample set into the image segmentation model before inputting the blood vessel image to be segmented into the image segmentation model for image segmentation, so as to obtain a classification label of test output; performing correlation degree calculation on the classification labels output by the test and the classification labels marked by the test sample set; when the correlation is greater than a preset correlation threshold, determining that the image segmentation model is effective;
the segmentation module is specifically used for: and when the image segmentation model is effective, inputting the blood vessel image to be segmented into the image segmentation model for image segmentation.
9. An electronic device, comprising: a processor, a memory, and a communication bus;
the communication bus is used for realizing connection communication between the processor and the memory;
the processor is configured to execute one or more programs stored in the memory to implement the steps of the image segmentation method as set forth in any one of claims 1 to 4.
10. A computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps of the image segmentation method of any one of claims 1-4.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2020/124169 WO2022087853A1 (en) | 2020-10-27 | 2020-10-27 | Image segmentation method and apparatus, and computer-readable storage medium |
| CNPCT/CN2020/124169 | 2020-10-27 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112419271A CN112419271A (en) | 2021-02-26 |
| CN112419271B true CN112419271B (en) | 2023-12-01 |
Family
ID=74777028
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011325510.8A Active CN112419271B (en) | 2020-10-27 | 2020-11-24 | Image segmentation method, device and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112419271B (en) |
| WO (1) | WO2022087853A1 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190934A (en) * | 2021-06-10 | 2021-07-30 | 北京三一智造科技有限公司 | Optimization method and device of pick barrel drill and electronic equipment |
| CN117237260A (en) * | 2022-06-02 | 2023-12-15 | 北京阅影科技有限公司 | Training method of image processing model, image processing method and device |
| CN114818839B (en) * | 2022-07-01 | 2022-09-16 | 之江实验室 | Deep learning-based optical fiber sensing underwater acoustic signal identification method and device |
| CN115170912B (en) * | 2022-09-08 | 2023-01-17 | 北京鹰瞳科技发展股份有限公司 | Method for training image processing model, method for generating image and related product |
| CN115359057B (en) * | 2022-10-20 | 2023-03-28 | 中国科学院自动化研究所 | Deep learning-based freezing electron microscope particle selection method and device and electronic equipment |
| CN115631301B (en) * | 2022-10-24 | 2023-07-28 | 东华理工大学 | Soil-stone mixture image three-dimensional reconstruction method based on improved full convolution neural network |
| CN116503607B (en) * | 2023-06-28 | 2023-09-19 | 天津市中西医结合医院（天津市南开医院） | CT image segmentation method and system based on deep learning |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109584254A (en) * | 2019-01-07 | 2019-04-05 | 浙江大学 | A kind of heart left ventricle's dividing method based on the full convolutional neural networks of deep layer |
| US10304193B1 (en) * | 2018-08-17 | 2019-05-28 | 12 Sigma Technologies | Image segmentation and object detection using fully convolutional neural network |
| CN110660046A (en) * | 2019-08-30 | 2020-01-07 | 太原科技大学 | Industrial product defect image classification method based on lightweight deep neural network |
| CN111028217A (en) * | 2019-12-10 | 2020-04-17 | 南京航空航天大学 | Image crack segmentation method based on full convolution neural network |
| CN111127447A (en) * | 2019-12-26 | 2020-05-08 | 河南工业大学 | Blood vessel segmentation network and method based on generative confrontation network |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6965343B2 (en) * | 2016-10-31 | 2021-11-10 | コニカ ミノルタ ラボラトリー ユー．エス．エー．，インコーポレイテッド | Image segmentation methods and systems with control feedback |
| CN107016681B (en) * | 2017-03-29 | 2023-08-25 | 浙江师范大学 | Brain MRI tumor segmentation method based on full convolution network |
| US10896508B2 (en) * | 2018-02-07 | 2021-01-19 | International Business Machines Corporation | System for segmentation of anatomical structures in cardiac CTA using fully convolutional neural networks |
| CN108876805B (en) * | 2018-06-20 | 2021-07-27 | 长安大学 | End-to-end unsupervised scene passable area cognition and understanding method |
| CN109886307A (en) * | 2019-01-24 | 2019-06-14 | 西安交通大学 | A kind of image detecting method and system based on convolutional neural networks |
| CN111583262A (en) * | 2020-04-23 | 2020-08-25 | 北京小白世纪网络科技有限公司 | Blood vessel segmentation method and system |
-
2020
- 2020-10-27 WO PCT/CN2020/124169 patent/WO2022087853A1/en active Application Filing
- 2020-11-24 CN CN202011325510.8A patent/CN112419271B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10304193B1 (en) * | 2018-08-17 | 2019-05-28 | 12 Sigma Technologies | Image segmentation and object detection using fully convolutional neural network |
| CN109584254A (en) * | 2019-01-07 | 2019-04-05 | 浙江大学 | A kind of heart left ventricle's dividing method based on the full convolutional neural networks of deep layer |
| CN110660046A (en) * | 2019-08-30 | 2020-01-07 | 太原科技大学 | Industrial product defect image classification method based on lightweight deep neural network |
| CN111028217A (en) * | 2019-12-10 | 2020-04-17 | 南京航空航天大学 | Image crack segmentation method based on full convolution neural network |
| CN111127447A (en) * | 2019-12-26 | 2020-05-08 | 河南工业大学 | Blood vessel segmentation network and method based on generative confrontation network |
Non-Patent Citations (1)
| Title |
|---|
| 基于监督的全卷积神经网络视网膜血管分割;王娜;傅迎华;蒋念平;;软件导刊(08);45-48+52 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022087853A1 (en) | 2022-05-05 |
| CN112419271A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112419271B (en) | Image segmentation method, device and computer readable storage medium | |
| CN106940816B (en) | CT image pulmonary nodule detection system based on 3D full convolution neural network | |
| CN110189255B (en) | Face detection method based on two-stage detection | |
| CN110706246B (en) | Blood vessel image segmentation method and device, electronic equipment and storage medium | |
| CN111145181B (en) | Skeleton CT image three-dimensional segmentation method based on multi-view separation convolutional neural network | |
| CN110070531B (en) | Model training method for detecting fundus picture, and fundus picture detection method and device | |
| CN110751636B (en) | Fundus image retinal arteriosclerosis detection method based on improved coding and decoding network | |
| CN112862830B (en) | Multi-mode image segmentation method, system, terminal and readable storage medium | |
| JPWO2019026104A1 (en) | Information processing apparatus, information processing program, and information processing method | |
| CN109685806B (en) | Image significance detection method and device | |
| US11663840B2 (en) | Method and system for removing noise in documents for image processing | |
| CN114758137B (en) | Ultrasonic image segmentation method and device and computer readable storage medium | |
| CN114998595B (en) | Weak supervision semantic segmentation method, semantic segmentation method and readable storage medium | |
| CN112396605B (en) | Network training method and device, image recognition method and electronic equipment | |
| CN112132827A (en) | Pathological image processing method and device, electronic equipment and readable storage medium | |
| CN113221925A (en) | Target detection method and device based on multi-scale image | |
| CN110570394A (en) | medical image segmentation method, device, equipment and storage medium | |
| CN113256670A (en) | Image processing method and device, and network model training method and device | |
| Ma et al. | Towards no-reference image quality assessment based on multi-scale convolutional neural network | |
| CN114241388A (en) | Video instance segmentation method and segmentation device based on space-time memory information | |
| CN114298971A (en) | Coronary artery segmentation method, system, terminal and storage medium | |
| CN116563285B (en) | Focus characteristic identifying and dividing method and system based on full neural network | |
| CN116309571B (en) | Three-dimensional cerebrovascular segmentation method and device based on semi-supervised learning | |
| CN115100731B (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| CN116664829A (en) | RGB-T semantic segmentation method, system, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |