CN108268870B - Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy - Google Patents

Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy Download PDF

Info

Publication number
CN108268870B
CN108268870B CN201810085384.XA CN201810085384A CN108268870B CN 108268870 B CN108268870 B CN 108268870B CN 201810085384 A CN201810085384 A CN 201810085384A CN 108268870 B CN108268870 B CN 108268870B
Authority
CN
China
Prior art keywords
segmentation
network
layer
image
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810085384.XA
Other languages
Chinese (zh)
Other versions
CN108268870A (en
Inventor
崔少国
张建勋
刘畅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Normal University
Original Assignee
Chongqing Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Normal University filed Critical Chongqing Normal University
Priority to CN201810085384.XA priority Critical patent/CN108268870B/en
Publication of CN108268870A publication Critical patent/CN108268870A/en
Application granted granted Critical
Publication of CN108268870B publication Critical patent/CN108268870B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Abstract

The invention provides a multiscale feature fusion ultrasonic image semantic segmentation method based on counterwork learning, which comprises the following steps of: building a multi-scale feature fusion semantic segmentation network model, building an antagonistic identification network model, performing antagonistic training and model parameter learning, and automatically segmenting the breast lesion. The segmentation method provided by the application predicts the pixel class by combining the multi-scale features of the input images with different resolutions, improves the accuracy of pixel class label prediction, adopts the expansion convolution to replace partial pooling to improve the resolution of the segmentation graph, adopts the countermeasure identification network to guide the segmentation graph generated by the segmentation net to be indistinguishable from the segmentation label, and ensures that the segmentation graph has good appearance and spatial continuity, thereby obtaining a more accurate high-resolution ultrasonic mammary gland lesion segmentation graph.

Description

Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy
Technical Field
The invention relates to the technical field of medical image understanding, in particular to a multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy.
Background
Breast cancer is a malignant tumor that occurs in mammary epithelial tissues, seriously threatening the health and quality of life of women. The ultrasonic examination is simple, convenient, economical and radiationless, and becomes an important tool for clinical breast cancer diagnosis. Ultrasonic B-mode images combined with ultrasonic elastography are important means for clinically diagnosing breast diseases. The B mode images the breast tissue structure, the elasticity imaging detects the elasticity of the breast tissue, and the two modal images are mutually contrasted to more accurately detect and position the breast focus.
The breast focal region is accurately identified and segmented from the ultrasonic image, and important support can be provided for accurate diagnosis and treatment of breast diseases. The position, size, shape and number of the focus area are all important bases for clinical diagnosis and treatment. Due to the existence of B-mode image speckle noise and elastic image artifact noise, the traditional image processing algorithm has poor effect when applied to the field, and usually has the problems of under-segmentation or over-segmentation, so that the boundary is not accurate enough. Therefore, it is currently the mainstream method to automatically detect and segment the lesion region from the multi-modal ultrasound image by using an artificial intelligence algorithm.
The deep learning technology makes a significant breakthrough in the series of tasks of computer vision. The ultrasonic image semantic segmentation is to label each pixel semantic category by using image features, and the semantic segmentation precision can be obviously improved by the deep learning technology based on the convolutional neural network. The convolutional neural network can automatically learn hierarchical characteristics facing specific tasks from training data in a supervised learning mode through end-to-end training, and can obtain a better segmentation effect compared with the traditional machine learning. However, the current breast ultrasound image semantic segmentation technology based on deep learning has the following disadvantages: (1) the classification of the pixels is predicted through the single scale features, the local features and the global context features are not fully utilized, and misclassification points are easy to generate; (2) the network remarkably reduces the resolution of the feature map through multi-level pooling (a typical VGG network has five times of pooling, and the feature map is reduced to 1/32 of an original image), so that the final segmentation map is smaller; (3) the prediction of each pixel label does not consider the similarity between the characteristics of each pixel, so that the segmentation graph output by the network lacks high-order spatial continuity.
Disclosure of Invention
Aiming at the problems of the existing method, the invention provides a multiscale feature fusion ultrasonic image semantic segmentation method based on countermeasure learning, which predicts the pixel class by using multiscale feature combination of input images with different resolutions and improves the accuracy of pixel class label prediction; the resolution of the segmentation graph is improved by adopting expansion convolution instead of partial pooling; and the segmentation graph generated by the segmentation net is guided by adopting the countermeasure identification network and cannot be distinguished from the segmentation label, so that the segmentation graph has good appearance and space continuity, and a more accurate high-resolution ultrasonic breast lesion segmentation graph is obtained. .
In order to solve the technical problems, the invention adopts the following technical scheme:
a multiscale feature fusion ultrasonic image semantic segmentation method based on counterwork learning comprises the following steps:
s1, building a multi-scale feature fusion semantic segmentation network model:
s11, the multi-scale feature fusion semantic segmentation network is used for generating a lesion area segmentation map of an input ultrasonic image, the multi-scale feature fusion semantic segmentation network comprises a multi-resolution feature extractor, a cascade feature fusion device and a classifier, the multi-resolution feature extractor is formed by at least two parallel deep convolution network branches, each branch is identical in structure and shared in weight, the multi-resolution feature extractor is used for extracting context features with different scales from input images with different resolutions, the cascade feature fusion device is used for carrying out step-by-step fusion on the multi-scale features obtained by the different branches to generate a comprehensive feature map, and the classifier is used for predicting pixel category labels according to the comprehensive feature map;
s12, each deep convolutional network branch structure including first to fifth convolutional layer groups, first to third largest pooling layers, a first swelling convolutional layer, and a second swelling convolutional layer, the first largest pooling layer being located after the first convolutional layer group, the second largest pooling layer being located after the second convolutional layer group, the third largest pooling layer being located after the third convolutional layer group, the first swelling convolutional layer being located after the fourth convolutional layer, the second swelling convolutional layer being located after the fifth convolutional layer group; the classifier comprises a first characteristic projection layer, a second characteristic projection layer, a category prediction layer and a Softmax regression layer which are sequentially arranged behind a second expansion convolution layer;
s2, establishing a confrontation authentication network model:
s21, the confrontation discrimination network is used for distinguishing whether the input is a segmentation map from a segmentation network or a segmentation map of a manual segmentation label;
s22, the confrontation authentication network comprises a first confrontation convolutional layer, a second confrontation convolutional layer, a third fully-connected layer and a Softmax regression layer which are sequentially arranged;
s3, confrontation training and model parameter learning: alternately training the multi-scale feature fusion semantic segmentation network and the confrontation discrimination network in a confrontation mode by using the expanded training data, designing an objective function to optimize network parameters, and generating an optimal segmentation network model, which specifically comprises the following steps:
s31, initializing the network model parameters by adopting an Xavier method;
s32, obtaining a breast ultrasound multi-modal gray level image training sample and a corresponding artificial segmentation label, wherein the breast ultrasound multi-modal gray level image comprises a B mode image and an elastic image, the B mode image and the elastic image are registered in space, and the training data is expanded according to the ratio of 4: 1, dividing the training set into a training set and a verification set, and training a network model by adopting a five-fold cross verification method;
s33, inputting the B-mode ultrasonic image and the ultrasonic elastic image into a multi-scale feature fusion semantic segmentation network as two channels, generating a pixel class label prediction probability distribution map, and adopting the following mixed loss function as an optimization objective function:
Figure BDA0001562250210000031
the mixed loss function is composed of two classification cross entropy loss functions L1And L2Composition, representing segmentation loss and discrimination loss, respectively; wherein, thetasIs a parameter of a segmented network model, thetaaIs to identify the network model parameter, xnIs the input n-th training sample, ynIs the segmentation label corresponding to the nth training sample, s (-) is the segmentation function, a (-) is the discrimination function, N is the batch size, λ1Is an antagonistic factor;
the cross entropy loss function L in the objective function of the step1Comprises the following steps:
Figure BDA0001562250210000041
wherein the content of the first and second substances,
Figure BDA0001562250210000042
label prediction probability vectors generated by a segmentation function s (.), y is probability vectors corresponding to segmentation labels, H x W is the dimension of the image, C is the number of classes of pixels, and ln (.) is a natural logarithm;
the cross entropy loss function L in the objective function of the step2Comprises the following steps:
Figure BDA0001562250210000043
wherein the content of the first and second substances,
Figure BDA0001562250210000044
the image source label prediction probability vector generated by the countermeasure identification network, and z is the probability vector of the real source label of the input image;
s34, inputting the segmentation labels and the segmentation graph output by the semantic segmentation network into a confrontation discrimination network, predicting whether the input is from the segmentation labels or the segmentation network by the confrontation discrimination network, and adopting the following loss function as an optimization objective function:
Figure BDA0001562250210000045
cross entropy loss function L in the loss function of this step2The same step as step S33;
s35 to obtain Lseg(θs) And Ladv(θa) Training the meaning division network and the countermeasure identification network in an alternative mode by taking the minimum value as an optimization target, optimizing a target function by adopting a random gradient descent algorithm, and updating a network model parameter theta by adopting an error back propagation algorithmsAnd thetaaAnd fixing the challenge discriminating network parameter theta in training the semantic segmentation networkaFixing semantic segmentation network parameters θ in training confrontation discrimination networkss；
S4, automatic segmentation of breast lesions:
s41, inputting the breast ultrasound B-mode images with different resolutions and the corresponding ultrasound elastic image pairs which are registered in space into each branch of the multi-resolution feature extractor as two channels, and respectively generating feature maps with different sizes;
s42, inputting the context features of different scales extracted by the multi-resolution feature extractor into a cascade feature fusion device to generate a comprehensive feature map;
s43, inputting the comprehensive characteristic graph into a classifier to generate a pixel label prediction value graph;
s44, converting the pixel class label prediction score map into a pixel class label prediction probability distribution map by using a Softmax function, wherein the pixel class label prediction probability distribution map represents the probability of each pixel on each class;
s45, taking the subscript component where the maximum probability of each pixel is located as a pixel class label to form a breast ultrasound focus semantic segmentation map;
s46, carrying out 8 times of upsampling on the breast ultrasound focus semantic segmentation map to obtain a final segmentation map with the same size as the original image.
Further, in step S11, the multi-resolution feature extractor is composed of three parallel deep convolution network branches, namely, a high-resolution feature extractor, a medium-resolution feature extractor, and a low-resolution feature extractor.
Further, the high resolution feature extractor branch input image size is the same as the original image size, the medium resolution feature extractor branch input image size is 1/2 of the original image size, and the low resolution feature extractor branch input image size is 1/4 of the original image size.
Further, the cascade feature fusion device firstly performs 2 times of upsampling on the last feature layer of the low-resolution feature extractor branch by using bilinear interpolation, and then adds the upsampling and the last feature layer of the medium-resolution feature extractor branch to generate a fusion feature map 1 with the same size as the medium-resolution feature map; and then performing 2 times of upsampling on the fused feature map 1 by using bilinear interpolation, and adding the upsampled fused feature map to the final feature layer of the high-resolution feature extractor branch to generate a fused feature map 2 with the same size as the high-resolution feature map, wherein the fused feature map 2 is the final comprehensive feature map.
Further, in step S12, each convolution layer group is composed of two convolution layers, the convolution kernel size of each convolution layer is 3 × 3, the step size is 1, and the number of convolution kernels of the first to fifth convolution layer groups is sequentially 64, 128, 256, 512, and 512; the convolution kernel size of each expansion convolutional layer is 3 multiplied by 3, the step length is 1, and the expansion factors of the first expansion convolutional layer and the second expansion convolutional layer are 3 and 6 respectively; the size of the pooling core of each maximum pooling layer is 2 multiplied by 2, and the step length is 2; the convolution kernel size of each characteristic projection layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels of the first characteristic projection layer and the second characteristic projection layer is respectively 256 and 128; the size of the convolution kernel of the category prediction layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels is 2.
Further, in step S22, the convolution kernel size of each pair of deconvolution layers is 5 × 5, the step size is 2, and the number of convolution kernels of the first to sixth pair of convolution layers is 32, 64, 128, 256, 512, 1024 in sequence; the first through third fully-connected layers are 1024, 512, and 2, respectively, where 2 represents two categories of whether the input graph is from a split net or a split label.
Further, the source label of the segmentation graph generated by the segmentation network is 0, and the source label of the segmentation graph is 1.
Further, in the method, an output characteristic diagram Z corresponding to any convolution kerneliThe calculation was performed using the following formula:
Figure BDA0001562250210000061
where f is a non-linear excitation function, r is the input channel index number, k is the input channel number, WirIs the r channel weight matrix of the i convolution kernel,
Figure BDA0001562250210000062
is a convolution operation, XrIs the r-th input channel image.
Further, in step S33, a regularization term is added to the optimization objective function, so that the final optimization objective function is obtained as follows:
Figure BDA0001562250210000063
wherein λ is2Is a regularization factor, Q1Is thetasThe number of parameters (c).
Further, in step S34, a regularization term is also added to the optimization objective function, so that the final optimization objective function is obtained as follows:
Figure BDA0001562250210000064
wherein λ is3Is a regularization factor, Q2Is thetaαThe number of parameters (c).
Compared with the prior art, the counterstudy-based multiscale feature fusion ultrasonic image semantic segmentation method provided by the invention has the following advantages:
1. images with different resolutions are input, context features with different scales can be extracted by adopting a multi-branch feature extractor, and the pixel labels are predicted by utilizing the features together, so that the prediction accuracy can be obviously improved;
2. the weight shared by all the branch feature extractors can reduce network parameters, so that the network is easier to train;
3. expansion convolution is adopted, the dimension of the characteristic diagram is not reduced while the neuron receptive field is enlarged, and the resolution of the characteristic diagram is improved, so that a segmentation diagram with higher resolution is generated;
4. the segmentation network is optimized by applying the countermeasure loss and the pixel label prediction loss together through the countermeasure training, the countermeasure gradient flow enters the segmentation network, the learned segmentation network model can output a segmentation map with good appearance and space continuity, the breast ultrasound image focus segmentation is more accurate, and then a fine-grained segmentation map with more segmentation details is generated.
Drawings
FIG. 1 is a system model schematic diagram of the counterstudy-based multi-scale feature fusion ultrasound image semantic segmentation method provided by the invention (a counterstudy discrimination network is only used in training, and only a semantic segmentation network is needed in segmentation application).
Fig. 2 is a schematic diagram of a multi-scale feature fusion semantic segmentation network structure provided by the present invention (the upper side of the legend indicates the number of feature maps, and the lower left side of the legend indicates the dimension of the feature maps).
Fig. 3 is a schematic diagram of the countermeasure discrimination network structure provided by the present invention (the upper side of the diagram indicates the number of feature maps or the number of neurons, and the lower left side of the diagram indicates the dimension of the feature maps).
FIG. 4 is a schematic diagram of the dilated convolution provided by the present invention. The convolution kernel size in this figure is described in the following embodiments and need not be described further herein.
Detailed Description
In order to make the technical means, the original characteristics, the achieved purposes and the effects of the invention easy to understand, the invention is further described with reference to the specific drawings and the preferred embodiments.
Referring to fig. 1 to 4, the present invention provides a method for segmenting semantic of a multi-scale feature fusion ultrasound image based on countervailing learning, including the following steps:
s1, building a multi-scale feature fusion semantic segmentation network model:
s11, the multi-scale feature fusion semantic segmentation network is used for generating a lesion area segmentation map of an input ultrasonic image, the multi-scale feature fusion semantic segmentation network comprises a multi-resolution feature extractor, a cascade feature fusion device and a classifier, the multi-resolution feature extractor is formed by at least two parallel deep convolution network branches, each branch has the same structure and is shared by weight values, the multi-resolution feature extractor is used for extracting context features with different scales from input images with different resolutions of each branch, the cascade feature fusion device is used for carrying out step-by-step fusion on the multi-scale features obtained by each different branch to generate a comprehensive feature map, and the classifier is used for predicting pixel category labels according to the comprehensive feature map; specifically, the input of the multi-scale feature fusion semantic segmentation network is 2 channels which respectively represent a B-mode ultrasonic image and an ultrasonic elastic image, and the output of the network is also 2 channels which respectively represent the probability that pixels belong to normal tissues and lesion tissues;
s12, each deep convolutional network branch structure includes first to fifth convolutional layer groups, first to third largest pooling layers, a first dilation convolutional layer and a second dilation convolutional layer, the first largest pooling layer is located after the first convolutional layer group, the second largest pooling layer is located after the second convolutional layer group, the third largest pooling layer is located after the third convolutional layer group, the first dilation convolutional layer is located after the fourth convolutional layer, the second dilation convolutional layer is located after the fifth convolutional layer group, in order to ensure that the size of the feature map after convolution is the same as that before convolution, Padding is set to 1 in the convolution process, that is, the image is filled with 0 value during convolution; the classifier comprises a first characteristic projection layer, a second characteristic projection layer, a category prediction layer and a Softmax regression layer which are sequentially arranged behind a second expansion convolution layer.
As a specific embodiment, the multi-resolution feature extractor is composed of three parallel deep convolution network branches, namely a high-resolution feature extractor, a medium-resolution feature extractor and a low-resolution feature extractor, wherein the three branches have the same structure and share weights, and the multi-resolution feature extractor is used for extracting different context features from input images with different sizes of the three branches of the high-resolution feature extractor, the medium-resolution feature extractor and the low-resolution feature extractor; correspondingly, the cascade feature fusion device has the function of carrying out step-by-step fusion on the multi-scale features acquired by three different branches, firstly carrying out 2-time up-sampling on the last feature layer of the low-resolution feature extractor branch by using bilinear interpolation, and then adding the up-sampling and the last feature layer of the medium-resolution feature extractor branch to generate a fusion feature map 1 with the same size as the medium-resolution feature map; and then performing 2 times of upsampling on the fused feature map 1 by using bilinear interpolation, and adding the upsampled fused feature map to the final feature layer of the high-resolution feature extractor branch to generate a fused feature map 2 with the same size as the high-resolution feature map, wherein the fused feature map 2 is the final comprehensive feature map.
As a specific embodiment, it is assumed that the size of the original image is 480 × 480, the size of the high-resolution feature extractor branch input image is 480 × 480, which is the same as the size of the original image, the size of the medium-resolution feature extractor branch input image is 1/2 × 240, which is the size of the original image, and can be obtained by performing 2-fold down-sampling on the original image, and the size of the low-resolution feature extractor branch input image is 1/4, which is the size of the original image, 120 × 120, which can be obtained by performing 4-fold down-sampling on the original image, thereby obtaining input images with different sizes for each depth convolution network branch.
As a specific embodiment, a detailed structure of the multi-scale feature fusion semantic segmentation network model is shown in table 1 below, where table 1 illustrates a high-resolution image size 480 × 480, and the same goes for a medium-resolution image size and a low-resolution image size:
table 1 ultrasonic image semantic segmentation system model parameter table (Padding ═ 1)
Figure BDA0001562250210000091
As can be seen from table 1, in step S12, each convolution layer group is composed of two convolution layers, the convolution kernel size of each convolution layer is 3 × 3, the step size is 1, and the number of convolution kernels of the first to fifth convolution layer groups is sequentially 64, 128, 256, 512, and 512; the convolution kernel size of each expansion convolutional layer is 3 multiplied by 3, the step length is 1, and the expansion factors of the first expansion convolutional layer and the second expansion convolutional layer are 3 and 6 respectively; the size of the pooling core of each maximum pooling layer is 2 multiplied by 2, and the step length is 2; the convolution kernel size of each characteristic projection layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels of the first characteristic projection layer and the second characteristic projection layer is respectively 256 and 128; the size of a convolution kernel of the category prediction layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels is 2; the Softmax regression layer converts the pixel class prediction scores into a probability distribution. The resolution feature maps output by the serial number 15 are subjected to cascade fusion and then input to the serial number 16, namely the input of the serial number 16 is the feature of the serial number 15 subjected to multi-resolution fusion.
S2, establishing a confrontation authentication network model:
s21, the confrontation discrimination network is used for distinguishing whether the input is a segmentation graph from a segmentation network or a manual segmentation label (GT);
s22, the countermeasure identification network includes a first to sixth countermeasure convolutional layers, a first to third full-link layers and a Softmax regression layer arranged in sequence, i.e. the full-link layer is arranged after the countermeasure convolutional layer, and the detailed structure is shown in table 2 below:
table 2 confrontation authentication network model parameter table (Padding ═ 1)
Figure BDA0001562250210000101
As can be seen from table 2, the convolution kernel size of each of the pair of convolutional layers is 5 × 5, the step size is 2, and the number of convolution kernels of the first to sixth pair of convolutional layers is 32, 64, 128, 256, 512, 1024 in sequence; the first through third fully-connected layers are 1024, 512, and 2, respectively, where 2 represents two categories of whether the input graph is from a split net or a split label. Specifically, the input of the confrontation identification network is 2 channels, which respectively represent probability distribution maps of pixels belonging to 2 categories of normal and focus, wherein the segmentation map and the segmentation label of each pair of images (B mode image and elastic image) correspond to two probability distribution maps, one is the probability distribution map of each pixel belonging to normal tissue, the other is the probability distribution map of each pixel belonging to lesion tissue, and the output of the network is a binary prediction result of 0-1. In a preferred embodiment, the segmentation network generates a segmentation map with a source label of 0 and a source label of 1, so that whether the input map is from the segmentation network or the segmentation label can be distinguished by countering the 0-1 binary prediction result output by the discrimination network.
As a specific embodiment, the operation of the network model is as follows:
(1) and (3) convolution operation: output characteristic diagram Z corresponding to any convolution kernel in the methodiThe calculation was performed using the following formula:
Figure BDA0001562250210000111
where f is a non-linear excitation function, r is the input channel index number, k is the input channel number, WirIs the r channel weight matrix of the i convolution kernel,
Figure BDA0001562250210000112
is a convolution operation, XrIs the r-th input channel image.
(2) And (3) expansion convolution:
the expansion convolution is convolution operation carried out by using an expansion convolution kernel, the expansion convolution kernel is that the convolution kernel is subjected to up-sampling, the original position weight of the convolution kernel is not changed, the middle position is supplemented with 0, and the expansion convolution improves the receptive field by adopting different expansion factors without increasing network parameters and calculated quantity. Specifically, referring to fig. 4, black represents the weight of the convolution kernel (the convolution kernel is 3 × 3), and gray represents the padding value of 0. The expansion factor of 1 is equivalent to a normal convolution, and as the expansion factor becomes larger, the receptive field becomes larger and the number of weights does not change.
(3) ReLU nonlinear excitation function:
in the above formula (1), f is a function of a rectifying Linear unit ReLU (rectifying Linear units), wherein ReLU is an activation function of the network, and the rectifying Linear unit is used for outputting a characteristic diagram Z generated by a convolution kerneliIs non-linearly transformed, said rectifying linear unit ReLU being defined as follows:
f(x)＝max(0,x) (2)
where f (x) is the rectified linear unit function and x is an input value.
(4) Softmax classification function:
the Softmax function converts the predicted scores output by the network into a probability distribution, the Softmax function being defined as follows:
Figure BDA0001562250210000121
wherein, OjIs the predicted score, Y, of a pixel in class j that is the last output of the networkjIs the probability that the input pixel belongs to the jth class, C is the number of classes, 2, and exp (.) is an exponential function with a natural constant e as the base.
S3, confrontation training and model parameter learning: alternately training the multi-scale feature fusion semantic segmentation network and the confrontation discrimination network in a confrontation mode by using the expanded training data, designing an objective function to optimize network parameters, and generating an optimal segmentation network model, which specifically comprises the following steps:
s31, initializing the network model parameters by adopting an Xavier method;
s32, acquiring 500 breast ultrasound multi-modal grayscale image training samples and corresponding artificial segmentation labels, where the breast ultrasound multi-modal grayscale image includes a B-mode image and an elastic image, the B-mode image and the elastic image are spatially registered, the sizes of the training samples and the corresponding segmentation labels are 480 × 480, and the training data samples are increased to 10 times of the initial size by using a horizontal flip, vertical flip, clipping, rotation by 45 °, 90 °, 135 °, 180 °, 225 °, 270 °, 315 ° data expansion technique, and then the training data are divided into 4: 1, dividing the training set into a training set and a verification set, and training a network model by adopting a five-fold cross verification method;
s33, inputting the B-mode ultrasonic image and the ultrasonic elastic image into a multi-scale feature fusion semantic segmentation network as two channels, generating a pixel class label prediction probability distribution map, and adopting the following mixed loss function as an optimization objective function:
Figure BDA0001562250210000122
the mixed loss function is composed of two classification cross entropy loss functions L1And L2Composition, representing segmentation loss and discrimination loss, respectively; wherein, thetasIs a parameter of a segmented network model, thetaaIs to identify the network model parameter, xnIs the input n-th training sample, ynIs the segmentation label corresponding to the nth training sample, and s (.) is the segmentation function (using the parameter θ)sFor input image sample xnIs predicted), a () is a discriminant function (using parameter θ)aPredicting the class of the network input, wherein 0 represents a segmentation graph generated by a segmentation network, 1 represents a segmentation label), N is the batch size which is the number of samples used in each iteration in the random gradient descent iteration, and is set as 32, and lambda is1Is an antagonistic factor, set to 0.5;
the cross entropy loss function L in the objective function of the step1Comprises the following steps:
Figure BDA0001562250210000131
wherein the content of the first and second substances,
Figure BDA0001562250210000132
is a label prediction probability vector generated by a segmentation function s (), y is a probability vector corresponding to a segmentation label, H × W is the dimension of the image, the size is 480 × 480, C is the number of classes of pixels, C is 2, and ln () is a natural logarithm;
the cross entropy loss function L in the objective function of the step2Comprises the following steps:
Figure BDA0001562250210000133
wherein the content of the first and second substances,
Figure BDA0001562250210000134
the image source label prediction probability vector generated by the countermeasure identification network, and z is the probability vector of the real source label of the input image;
in order to prevent overfitting, a regularization term is added to the optimization objective function of the formula (4), so that the final optimization objective function is as follows:
Figure BDA0001562250210000135
wherein λ is2Is a regularization factor, set to 0.1, Q1Is thetasThe number of parameters (c).
S34, inputting the segmentation labels and the segmentation graph output by the semantic segmentation network into a confrontation discrimination network, predicting whether the input is from the segmentation labels or the segmentation network by the confrontation discrimination network, and adopting the following loss function as an optimization objective function:
Figure BDA0001562250210000136
cross entropy loss function L in the loss function of this step2The same step as step S33;
also to prevent overfitting, a regularization term is added to the optimization objective function of the countermeasure discrimination network, and the final optimization objective function is obtained as follows:
Figure BDA0001562250210000141
wherein λ is3Is a regularization factor, set to 0.1, Q2Is thetaαThe number of parameters (c).
S35 to obtain Lseg(θs) And Ladv(θa) Training the meaning division network and the countermeasure identification network in an alternating mode by taking the minimum value as an optimization target, alternating every 500 iterations, optimizing a target function by adopting a random gradient descent algorithm, and updating a network model parameter theta by adopting an error back propagation algorithmsAnd thetaa(ii) a The goal of the training of the identification network is to well distinguish the segmentation graph from the segmentation labels, and the goal of the training of the segmentation network is to make the segmentation network accurately predict the pixel labels and make the identifier unable to distinguish the segmentation graph generated by the segmentation network from the corresponding segmentation standard labels, i.e. the segmentation graph generated by the segmentation network is very similar or almost the same as the segmentation standard labels. Specifically, when the segmentation network is trained, the parameter theta of the segmentation network of the model is fixedaTo optimize thetasThe momentum coefficient μ is 0.9, and the initial learning rate is ηt＝1e-3Decrease 1/10 every 1000 iterations until 1e-6Until the end; parameter theta of fixed mould type division net during anti-identification network trainingsTo optimize thetaaThe initial learning rate is set to ηt＝1e-4Decrease 1/10 every 1000 iterations until 1e-7Until now.
S4, automatic segmentation of breast lesions:
s41, inputting the breast ultrasound B-mode images with different resolutions and the corresponding ultrasound elastic image pairs which are registered in space into each branch of the multi-resolution feature extractor as two channels, and respectively generating feature maps with different sizes; for example, the original breast ultrasound B-mode image and the ultrasound elasticity image are input into a high-resolution feature extractor as two channels, and a feature map with the size of the original image 1/8 is generated; then, after 2-fold down sampling and 4-fold down sampling are carried out on the original image, a medium-resolution image and a low-resolution image are formed and are respectively input into a medium-resolution feature extractor and a low-resolution feature extractor, and feature maps with the sizes of the original images 1/16 and 1/32 are respectively generated;
s42, inputting the context features of different scales extracted by the multi-resolution feature extractor into a cascade feature fusion device to generate a comprehensive feature map; performing 2-time upsampling on the low-resolution feature map, performing pixel-by-pixel addition fusion on the low-resolution feature map and the medium-resolution feature map, performing 2-time upsampling on the fused feature map, performing pixel-by-pixel addition fusion on the fused feature map and the high-resolution feature map, and finally generating a comprehensive feature map;
s43, inputting the comprehensive characteristic graph into a classifier to generate 2 pixel class label prediction score graphs which respectively represent the prediction scores of the pixels belonging to normal tissues and lesion tissues;
s44, converting the pixel class label prediction score map into a pixel class label prediction probability distribution map by using a Softmax function, wherein the pixel class label prediction probability distribution map represents the probability of each pixel on each class;
s45, taking a subscript component (0 or 1) where the maximum probability of each pixel is located as a pixel class label to form a breast ultrasound focus semantic segmentation map;
s46, carrying out 8-time upsampling on the breast ultrasound focus semantic segmentation map to obtain a final segmentation map with the same size as the original image; specifically, in the pooling process of this embodiment, the image resolution is reduced to 1/8 of the original image, and 8 times of upsampling is required to obtain a segmentation map with the same size as the original image.
Compared with the prior art, the counterstudy-based multiscale feature fusion ultrasonic image semantic segmentation method provided by the invention has the following advantages:
1. images with different resolutions are input, context features with different scales can be extracted by adopting a multi-branch feature extractor, and the pixel labels are predicted by utilizing the features together, so that the prediction accuracy can be obviously improved;
2. the weight shared by all the branch feature extractors can reduce network parameters, so that the network is easier to train;
3. expansion convolution is adopted, the dimension of the characteristic diagram is not reduced while the neuron receptive field is enlarged, and the resolution of the characteristic diagram is improved, so that a segmentation diagram with higher resolution is generated;
4. the segmentation network is optimized by applying the countermeasure loss and the pixel label prediction loss together through the countermeasure training, the countermeasure gradient flow enters the segmentation network, the learned segmentation network model can output a segmentation map with good appearance and space continuity, the breast ultrasound image focus segmentation is more accurate, and then a fine-grained segmentation map with more segmentation details is generated.
Finally, the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all of them should be covered in the claims of the present invention.

Claims (10)

1. The multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy is characterized by comprising the following steps of:
s1, building a multi-scale feature fusion semantic segmentation network model:
s11, the multi-scale feature fusion semantic segmentation network is used for generating a lesion area segmentation map of an input ultrasonic image, the multi-scale feature fusion semantic segmentation network comprises a multi-resolution feature extractor, a cascade feature fusion device and a classifier, the multi-resolution feature extractor is formed by at least two parallel deep convolution network branches, each branch is identical in structure and shared in weight, the multi-resolution feature extractor is used for extracting context features with different scales from input images with different resolutions, the cascade feature fusion device is used for carrying out step-by-step fusion on the multi-scale features obtained by the different branches to generate a comprehensive feature map, and the classifier is used for predicting pixel category labels according to the comprehensive feature map;
s12, each deep convolutional network branch structure including first to fifth convolutional layer groups, a first maximum pooling layer, a second maximum pooling layer, a third maximum pooling layer, a first expansion convolutional layer, and a second expansion convolutional layer, the first maximum pooling layer being located after the first convolutional layer group, the second maximum pooling layer being located after the second convolutional layer group, the third maximum pooling layer being located after the third convolutional layer group, the first expansion convolutional layer being located after the fourth convolutional layer group, and the second expansion convolutional layer being located after the fifth convolutional layer group; the classifier comprises a first characteristic projection layer, a second characteristic projection layer, a category prediction layer and a Softmax regression layer which are sequentially arranged behind a second expansion convolution layer;
s2, establishing a confrontation authentication network model:
s21, the confrontation discrimination network is used for distinguishing whether the input is a segmentation map from a segmentation network or a segmentation map of a manual segmentation label;
s22, the confrontation authentication network comprises a first confrontation convolutional layer, a second confrontation convolutional layer, a third fully-connected layer and a Softmax regression layer which are sequentially arranged;
s3, confrontation training and model parameter learning: alternately training the multi-scale feature fusion semantic segmentation network and the confrontation discrimination network in a confrontation mode by using the expanded training data, designing an objective function to optimize network parameters, and generating an optimal segmentation network model, which specifically comprises the following steps:
s31, initializing the network model parameters by adopting an Xavier method;
s32, obtaining a breast ultrasound multi-modal gray level image training sample and a corresponding artificial segmentation label, wherein the breast ultrasound multi-modal gray level image comprises a B mode image and an elastic image, the B mode image and the elastic image are registered in space, and the training data is expanded according to the ratio of 4: 1, dividing the training set into a training set and a verification set, and training a network model by adopting a five-fold cross verification method;
s33, inputting the B-mode ultrasonic image and the ultrasonic elastic image into a multi-scale feature fusion semantic segmentation network as two channels, generating a pixel class label prediction probability distribution map, and adopting the following mixed loss function as an optimization objective function:
Figure FDA0002552441720000021
the mixed loss function is composed of two classification cross entropy loss functions L1And L2Composition, representing segmentation loss and discrimination loss, respectively; wherein, thetasIs a parameter of a segmented network model, thetaaIs to identify the network model parameter, xnIs the input n-th training sample, ynIs the segmentation label corresponding to the nth training sample, s (-) is the segmentation function, a (-) is the discrimination function, N is the batch size, λ1Is an antagonistic factor;
the cross entropy loss function L in the objective function of the step1Comprises the following steps:
Figure FDA0002552441720000022
wherein the content of the first and second substances,
Figure FDA0002552441720000023
label prediction probability vectors generated by a segmentation function s (.), y is probability vectors corresponding to segmentation labels, H x W is the dimension of the image, C is the number of classes of pixels, and ln (.) is a natural logarithm;
the cross entropy loss function L in the objective function of the step2Comprises the following steps:
Figure FDA0002552441720000024
wherein the content of the first and second substances,
Figure FDA0002552441720000025
the image source label prediction probability vector generated by the countermeasure identification network, and z is the probability vector of the real source label of the input image;
s34, inputting the segmentation labels and the segmentation graph output by the semantic segmentation network into a confrontation discrimination network, predicting whether the input is from the segmentation labels or the segmentation network by the confrontation discrimination network, and adopting the following loss function as an optimization objective function:
Figure FDA0002552441720000031
cross entropy loss function L in the loss function of this step2The same step as step S33;
s35 to obtain Lseg(θs) And Ladv(θa) Training the meaning division network and the countermeasure identification network in an alternative mode by taking the minimum value as an optimization target, optimizing a target function by adopting a random gradient descent algorithm, and updating a network model parameter theta by adopting an error back propagation algorithmsAnd thetaaAnd fixing the challenge discriminating network parameter theta in training the semantic segmentation networkaFixing semantic segmentation network parameters θ in training confrontation discrimination networkss；
S4, automatic segmentation of breast lesions:
s41, inputting the breast ultrasound B-mode images with different resolutions and the corresponding ultrasound elastic image pairs which are registered in space into each branch of the multi-resolution feature extractor as two channels, and respectively generating feature maps with different sizes;
s42, inputting the context features of different scales extracted by the multi-resolution feature extractor into a cascade feature fusion device to generate a comprehensive feature map;
s43, inputting the comprehensive characteristic graph into a classifier to generate a pixel class label prediction value graph;
s44, converting the pixel class label prediction score map into a pixel class label prediction probability distribution map by using a Softmax function, wherein the pixel class label prediction probability distribution map represents the probability of each pixel on each class;
s45, taking the subscript component where the maximum probability of each pixel is located as a pixel class label to form a breast ultrasound focus semantic segmentation map;
s46, carrying out 8 times of upsampling on the breast ultrasound focus semantic segmentation map to obtain a final segmentation map with the same size as the original image.
2. The method for semantic segmentation of ultrasound images based on multi-scale feature fusion of claim 1, wherein in step S11, the multi-resolution feature extractor is composed of three parallel deep convolutional network branches, namely, a high resolution feature extractor, a medium resolution feature extractor and a low resolution feature extractor.
3. The method of claim 2, wherein the high resolution feature extractor branch input image size is the same as the original image size, the medium resolution feature extractor branch input image size is 1/2 of the original image size, and the low resolution feature extractor branch input image size is 1/4 of the original image size.
4. The method for multi-scale feature fusion ultrasonic image semantic segmentation based on antagonistic learning as claimed in claim 2, wherein the cascade feature fusion device firstly performs 2 times up-sampling on the last feature layer of the low resolution feature extractor branch by using bilinear interpolation, and then adds the up-sampled feature layer and the last feature layer of the middle resolution feature extractor branch to generate a fusion feature map 1 with the same size as the middle resolution feature map; and then performing 2 times of upsampling on the fused feature map 1 by using bilinear interpolation, and adding the upsampled fused feature map to the final feature layer of the high-resolution feature extractor branch to generate a fused feature map 2 with the same size as the high-resolution feature map, wherein the fused feature map 2 is the final comprehensive feature map.
5. The method for semantic segmentation of ultrasound images fused with multi-scale features based on antagonistic learning according to claim 1, wherein in step S12, each convolution layer group consists of two convolution layers, the convolution kernel size of each convolution layer is 3 × 3, the step size is 1, and the number of convolution kernels of the first to fifth convolution layer groups is 64, 128, 256, 512 in sequence; the convolution kernel size of each expansion convolutional layer is 3 multiplied by 3, the step length is 1, and the expansion factors of the first expansion convolutional layer and the second expansion convolutional layer are 3 and 6 respectively; the size of the pooling core of each maximum pooling layer is 2 multiplied by 2, and the step length is 2; the convolution kernel size of each characteristic projection layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels of the first characteristic projection layer and the second characteristic projection layer is respectively 256 and 128; the size of the convolution kernel of the category prediction layer is 1 multiplied by 1, the step length is 1, and the number of the convolution kernels is 2.
6. The method for semantic segmentation of ultrasound images fused with multi-scale features based on counterlearning of claim 1, wherein in step S22, the convolution kernel size of each counterconvolutional layer is 5 × 5, the step size is 2, and the number of convolution kernels of the first to sixth counterconvolutional layers is 32, 64, 128, 256, 512, 1024; the first through third fully-connected layers are 1024, 512, and 2, respectively, where 2 represents two categories of whether the input graph is from a split net or a split label.
7. The method for semantic segmentation of ultrasound images fused with multi-scale features based on antagonistic learning as claimed in claim 6, wherein the segmentation network generates a segmentation map with a source label of 0 and a source label of 1.
8. The method for semantic segmentation of ultrasound image based on multi-scale feature fusion based on counterstudy of claim 1, wherein the output feature graph Z corresponding to any convolution kernel in the methodiThe calculation was performed using the following formula:
Figure FDA0002552441720000051
where f is a non-linear excitation function, r is the input channel index number, k is the input channel number, WirIs the r channel weight matrix of the i convolution kernel,
Figure FDA0002552441720000052
is a convolution operation, XrIs the r-th input channel image.
9. The method for segmenting ultrasound image semantics based on counterlearning and fusion of multi-scale features as claimed in claim 1, wherein in the step S33, a regularization term is further added to the optimization objective function, so that the final optimization objective function is obtained as follows:
Figure FDA0002552441720000053
wherein λ is2Is a regularization factor, Q1Is thetasThe number of parameters (c).
10. The method for segmenting ultrasound image semantics based on multi-scale feature fusion based on counterlearning of claim 1, wherein in the step S34, a regularization term is also added to the optimization objective function, so that the final optimization objective function is obtained as follows:
Figure FDA0002552441720000054
wherein λ is3Is a regularization factor, Q2Is thetaαThe number of parameters (c).
CN201810085384.XA 2018-01-29 2018-01-29 Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy Active CN108268870B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810085384.XA CN108268870B (en) 2018-01-29 2018-01-29 Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810085384.XA CN108268870B (en) 2018-01-29 2018-01-29 Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy

Publications (2)

Publication Number Publication Date
CN108268870A CN108268870A (en) 2018-07-10
CN108268870B true CN108268870B (en) 2020-10-09

Family

ID=62776960

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810085384.XA Active CN108268870B (en) 2018-01-29 2018-01-29 Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy

Country Status (1)

Country Link
CN (1) CN108268870B (en)

Families Citing this family (106)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109166095B (en) * 2018-07-11 2021-06-25 广东技术师范学院 Fundus image cup and disc segmentation method based on generation countermeasure mechanism
CN109034224B (en) * 2018-07-16 2022-03-11 西安电子科技大学 Hyperspectral classification method based on double branch network
CN109064455B (en) * 2018-07-18 2021-06-25 清华大学深圳研究生院 BI-RADS-based classification method for breast ultrasound image multi-scale fusion
CN109086770B (en) * 2018-07-25 2021-12-17 成都快眼科技有限公司 Image semantic segmentation method and model based on accurate scale prediction
CN108961295B (en) * 2018-07-27 2022-01-28 重庆师范大学 Purple soil image segmentation and extraction method based on normal distribution H threshold
CN109166130B (en) * 2018-08-06 2021-06-22 北京市商汤科技开发有限公司 Image processing method and image processing device
CN109241872B (en) * 2018-08-20 2022-03-18 电子科技大学 Image semantic fast segmentation method based on multistage network
US10318842B1 (en) * 2018-09-05 2019-06-11 StradVision, Inc. Learning method, learning device for optimizing parameters of CNN by using multiple video frames and testing method, testing device using the same
WO2020048359A1 (en) * 2018-09-06 2020-03-12 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method, system, and computer-readable medium for improving quality of low-light images
CN109359655B (en) * 2018-09-18 2021-07-16 河南大学 Image segmentation method based on context regularization cycle deep learning
CN109508627A (en) * 2018-09-21 2019-03-22 国网信息通信产业集团有限公司 The unmanned plane dynamic image identifying system and method for shared parameter CNN in a kind of layer
CN109389057B (en) * 2018-09-22 2021-08-06 天津大学 Object detection method based on multi-scale advanced semantic fusion network
CN109345456B (en) * 2018-09-30 2021-01-19 京东方科技集团股份有限公司 Generation countermeasure network training method, image processing method, device, and storage medium
MX2020013580A (en) 2018-09-30 2021-02-26 Boe Technology Group Co Ltd Apparatus and method for image processing, and system for training neural network.
CN111091524A (en) * 2018-10-08 2020-05-01 天津工业大学 Prostate transrectal ultrasound image segmentation method based on deep convolutional neural network
CN109472799B (en) * 2018-10-09 2021-02-23 清华大学 Image segmentation method and device based on deep learning
CN109523569B (en) * 2018-10-18 2020-01-31 中国科学院空间应用工程与技术中心 optical remote sensing image segmentation method and device based on multi-granularity network fusion
CN109509192B (en) * 2018-10-18 2023-05-30 天津大学 Semantic segmentation network integrating multi-scale feature space and semantic space
CN109447990B (en) * 2018-10-22 2021-06-22 北京旷视科技有限公司 Image semantic segmentation method and device, electronic equipment and computer readable medium
WO2020093782A1 (en) 2018-11-08 2020-05-14 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method, system, and computer-readable medium for improving quality of low-light images
CN109598269A (en) * 2018-11-14 2019-04-09 天津大学 A kind of semantic segmentation method based on multiresolution input with pyramid expansion convolution
CN109583454A (en) * 2018-11-14 2019-04-05 天津大学 Image characteristic extracting method based on confrontation neural network
CN109785399B (en) * 2018-11-19 2021-01-19 北京航空航天大学 Synthetic lesion image generation method, device, equipment and readable storage medium
CN109544555B (en) * 2018-11-26 2021-09-03 陕西师范大学 Tiny crack segmentation method based on generation type countermeasure network
WO2020108009A1 (en) * 2018-11-26 2020-06-04 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method, system, and computer-readable medium for improving quality of low-light images
CN109712145B (en) * 2018-11-28 2021-01-08 山东师范大学 Image matting method and system
CN109636817B (en) * 2018-11-30 2020-10-30 华中科技大学 Lung nodule segmentation method based on two-dimensional convolutional neural network
CN109598728B (en) 2018-11-30 2019-12-27 腾讯科技（深圳）有限公司 Image segmentation method, image segmentation device, diagnostic system, and storage medium
CN109472798A (en) * 2018-12-10 2019-03-15 重庆理工大学 Live pig fat content detection model training method and live pig fat content detection method
CN109598732B (en) * 2018-12-11 2022-06-14 厦门大学 Medical image segmentation method based on three-dimensional space weighting
CN109544585A (en) * 2018-12-19 2019-03-29 中国石油大学(华东) A kind of cholelithiasis CT medical image data Enhancement Method based on lightweight convolutional neural networks
CN109784194B (en) * 2018-12-20 2021-11-23 北京图森智途科技有限公司 Target detection network construction method, training method and target detection method
CN111369567B (en) * 2018-12-26 2022-12-16 腾讯科技（深圳）有限公司 Method and device for segmenting target object in three-dimensional image and electronic equipment
CN109784380A (en) * 2018-12-27 2019-05-21 西安交通大学 A kind of various dimensions weeds in field recognition methods based on generation confrontation study
CN109711413B (en) * 2018-12-30 2023-04-07 陕西师范大学 Image semantic segmentation method based on deep learning
CN110008949B (en) * 2019-01-24 2020-03-17 华南理工大学 Image target detection method, system, device and storage medium
CN111488964A (en) * 2019-01-29 2020-08-04 北京市商汤科技开发有限公司 Image processing method and device and neural network training method and device
CN110414631B (en) * 2019-01-29 2022-02-01 腾讯科技（深圳）有限公司 Medical image-based focus detection method, model training method and device
CN109949276B (en) * 2019-02-28 2021-06-11 华中科技大学 Lymph node detection method for improving SegNet segmentation network
CN109902809B (en) * 2019-03-01 2022-08-12 成都康乔电子有限责任公司 Auxiliary semantic segmentation model by using generated confrontation network
CN109948707B (en) * 2019-03-20 2023-04-18 腾讯科技（深圳）有限公司 Model training method, device, terminal and storage medium
CN109978863B (en) * 2019-03-27 2021-10-08 北京青燕祥云科技有限公司 Target detection method based on X-ray image and computer equipment
CN110033422B (en) * 2019-04-10 2021-03-23 北京科技大学 Fundus OCT image fusion method and device
CN110110617B (en) * 2019-04-22 2021-04-20 腾讯科技（深圳）有限公司 Medical image segmentation method and device, electronic equipment and storage medium
CN110074813B (en) * 2019-04-26 2022-03-04 深圳大学 Ultrasonic image reconstruction method and system
CN110222578B (en) * 2019-05-08 2022-12-27 腾讯科技（深圳）有限公司 Method and apparatus for challenge testing of speak-with-picture system
CN110232386B (en) * 2019-05-09 2022-04-05 杭州深睿博联科技有限公司 Pulmonary nodule classification method and device based on local feature pyramid
CN110276354B (en) * 2019-05-27 2023-04-07 东南大学 High-resolution streetscape picture semantic segmentation training and real-time segmentation method
CN110288574A (en) * 2019-06-13 2019-09-27 南通市传染病防治院(南通市第三人民医院) A kind of adjuvant Ultrasonographic Diagnosis hepatoncus system and method
CN110348531B (en) * 2019-07-17 2022-12-30 沈阳亚视深蓝智能科技有限公司 Deep convolution neural network construction method with resolution adaptability and application
CN110363168A (en) * 2019-07-19 2019-10-22 山东浪潮人工智能研究院有限公司 A kind of 3 dimensional drawing identifying system based on convolutional neural networks
CN110414526B (en) * 2019-07-31 2022-04-08 达闼科技（北京）有限公司 Training method, training device, server and storage medium for semantic segmentation network
CN110490884B (en) * 2019-08-23 2023-04-28 北京工业大学 Lightweight network semantic segmentation method based on countermeasure
CN110647889B (en) * 2019-08-26 2022-02-08 中国科学院深圳先进技术研究院 Medical image recognition method, medical image recognition apparatus, terminal device, and medium
CN110765844B (en) * 2019-09-03 2023-05-26 华南理工大学 Automatic non-induction type dinner plate image data labeling method based on countermeasure learning
CN110648311B (en) * 2019-09-03 2023-04-18 南开大学 Acne image focus segmentation and counting network model based on multitask learning
CN110599476B (en) * 2019-09-12 2023-05-23 腾讯科技（深圳）有限公司 Disease grading method, device, equipment and medium based on machine learning
CN112492323B (en) * 2019-09-12 2022-07-19 上海哔哩哔哩科技有限公司 Live broadcast mask generation method, readable storage medium and computer equipment
CN110853039B (en) * 2019-10-16 2023-06-02 深圳信息职业技术学院 Sketch image segmentation method, system and device for multi-data fusion and storage medium
CN110827255A (en) * 2019-10-31 2020-02-21 杨本强 Plaque stability prediction method and system based on coronary artery CT image
CN110910351B (en) * 2019-10-31 2022-07-29 上海交通大学 Ultrasound image modality migration and classification method and terminal based on generation countermeasure network
CN111160413B (en) * 2019-12-12 2023-11-17 天津大学 Thyroid nodule classification method based on multi-scale feature fusion
CN111161279B (en) * 2019-12-12 2023-05-26 中国科学院深圳先进技术研究院 Medical image segmentation method, device and server
CN111179275B (en) * 2019-12-31 2023-04-25 电子科技大学 Medical ultrasonic image segmentation method
CN111161273B (en) * 2019-12-31 2023-03-21 电子科技大学 Medical ultrasonic image segmentation method based on deep learning
CN111259898B (en) * 2020-01-08 2023-03-24 西安电子科技大学 Crop segmentation method based on unmanned aerial vehicle aerial image
CN111275712B (en) * 2020-01-15 2022-03-01 浙江工业大学 Residual semantic network training method oriented to large-scale image data
CN111325093A (en) * 2020-01-15 2020-06-23 北京字节跳动网络技术有限公司 Video segmentation method and device and electronic equipment
CN111368899B (en) * 2020-02-28 2023-07-25 中国人民解放军南部战区总医院 Method and system for segmenting echocardiogram based on recursion aggregation deep learning
CN111292317B (en) * 2020-03-11 2022-06-07 四川大学华西医院 Method for enhancing image local feature type multitask segmentation of in-situ cancer region in mammary duct
CN111402257B (en) * 2020-03-11 2023-04-07 华侨大学 Automatic medical image segmentation method based on multi-task collaborative cross-domain migration
CN111640120B (en) * 2020-04-09 2023-08-29 之江实验室 Pancreas CT automatic segmentation method based on significance dense connection expansion convolution network
CN111723841A (en) * 2020-05-09 2020-09-29 北京捷通华声科技股份有限公司 Text detection method and device, electronic equipment and storage medium
CN111584073B (en) * 2020-05-13 2023-05-09 山东大学 Method for constructing diagnosis models of benign and malignant lung nodules in various pathological types
CN111681166B (en) * 2020-06-02 2023-04-18 重庆理工大学 Image super-resolution reconstruction method of stacked attention mechanism coding and decoding unit
CN111784701A (en) * 2020-06-10 2020-10-16 深圳市人民医院 Ultrasonic image segmentation method and system combining boundary feature enhancement and multi-scale information
CN111784652B (en) * 2020-06-24 2024-02-06 西安电子科技大学 MRI (magnetic resonance imaging) segmentation method based on reinforcement learning multi-scale neural network
CN111899169B (en) * 2020-07-02 2024-01-26 佛山市南海区广工大数控装备协同创新研究院 Method for segmenting network of face image based on semantic segmentation
CN111881768A (en) * 2020-07-03 2020-11-03 苏州开心盒子软件有限公司 Document layout analysis method
CN111915622B (en) * 2020-07-09 2024-01-23 沈阳先进医疗设备技术孵化中心有限公司 Training of image segmentation network model and image segmentation method and device
CN113935928B (en) * 2020-07-13 2023-04-11 四川大学 Rock core image super-resolution reconstruction based on Raw format
CN111870279B (en) * 2020-07-31 2022-01-28 西安电子科技大学 Method, system and application for segmenting left ventricular myocardium of ultrasonic image
CN111985549B (en) * 2020-08-12 2023-03-31 中国科学院光电技术研究所 Deep learning method for automatic positioning and identification of components for given rigid body target
CN112215847B (en) * 2020-09-30 2022-06-24 武汉大学 Method for automatically segmenting overlapped chromosomes based on counterstudy multi-scale features
CN112815493A (en) * 2021-01-11 2021-05-18 珠海格力电器股份有限公司 Air conditioner control method and device, storage medium and air conditioner
CN112837270B (en) * 2021-01-11 2023-04-07 成都圭目机器人有限公司 Synthetic method and network model of road surface image with semantic annotation
CN112750111B (en) * 2021-01-14 2024-02-06 浙江工业大学 Disease identification and segmentation method in tooth full-view film
CN112750124B (en) * 2021-01-22 2021-11-09 推想医疗科技股份有限公司 Model generation method, image segmentation method, model generation device, image segmentation device, electronic equipment and storage medium
US11580646B2 (en) * 2021-03-26 2023-02-14 Nanjing University Of Posts And Telecommunications Medical image segmentation method based on U-Net
CN113239951B (en) * 2021-03-26 2024-01-30 无锡祥生医疗科技股份有限公司 Classification method, device and storage medium for ultrasonic breast lesions
CN113077471B (en) * 2021-03-26 2022-10-14 南京邮电大学 Medical image segmentation method based on U-shaped network
CN113362360B (en) * 2021-05-28 2022-08-30 上海大学 Ultrasonic carotid plaque segmentation method based on fluid velocity field
CN113421270B (en) * 2021-07-05 2022-07-19 上海市精神卫生中心（上海市心理咨询培训中心） Method, system, device, processor and storage medium for realizing medical image domain adaptive segmentation based on single-center calibration data
CN113657388B (en) * 2021-07-09 2023-10-31 北京科技大学 Image semantic segmentation method for super-resolution reconstruction of fused image
CN113298080B (en) * 2021-07-26 2021-11-05 城云科技（中国）有限公司 Target detection enhancement model, target detection method, target detection device and electronic device
CN113703048B (en) * 2021-08-30 2022-08-23 中国科学院地质与地球物理研究所 Method and system for detecting high-resolution earthquake fault of antagonistic neural network
CN113989585B (en) * 2021-10-13 2022-08-26 北京科技大学 Medium-thickness plate surface defect detection method based on multi-feature fusion semantic segmentation
CN114693967B (en) * 2022-03-20 2023-10-31 电子科技大学 Multi-classification semantic segmentation method based on classification tensor enhancement
CN114387346A (en) * 2022-03-25 2022-04-22 阿里巴巴达摩院(杭州)科技有限公司 Image recognition and prediction model processing method, three-dimensional modeling method and device
CN114445430B (en) * 2022-04-08 2022-06-21 暨南大学 Real-time image semantic segmentation method and system for lightweight multi-scale feature fusion
CN115393301B (en) * 2022-08-16 2024-03-12 中山大学附属第一医院 Image histology analysis method and device for liver two-dimensional shear wave elastic image
CN115578360B (en) * 2022-10-24 2023-12-26 电子科技大学 Multi-target semantic segmentation method for ultrasonic cardiac image
CN115619641B (en) * 2022-10-24 2023-06-02 中山大学附属第五医院 FFDM-based breast image processing method, FFDM-based breast image processing system, FFDM-based terminal and FFDM-based breast image processing medium
CN115423806B (en) * 2022-11-03 2023-03-24 南京信息工程大学 Breast mass detection method based on multi-scale cross-path feature fusion
CN116030363B (en) * 2023-02-20 2023-06-23 北京数慧时空信息技术有限公司 Remote sensing image class activation mapping chart optimizing method
CN116563553B (en) * 2023-07-10 2023-09-29 武汉纺织大学 Unmanned aerial vehicle image segmentation method and system based on deep learning

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106886801A (en) * 2017-04-14 2017-06-23 北京图森未来科技有限公司 A kind of image, semantic dividing method and device
CN107220980A (en) * 2017-05-25 2017-09-29 重庆理工大学 A kind of MRI image brain tumor automatic division method based on full convolutional network
CN107346436A (en) * 2017-06-29 2017-11-14 北京以萨技术股份有限公司 A kind of vision significance detection method of fused images classification

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7397945B2 (en) * 2003-11-20 2008-07-08 Hewlett-Packard Development Company, L.P. Method and system of image segmentation using regression clustering
US20080030497A1 (en) * 2005-12-08 2008-02-07 Yangqiu Hu Three dimensional modeling of objects
US8411919B2 (en) * 2008-07-07 2013-04-02 Siemens Aktiengesellschaft Fluid dynamics approach to image segmentation

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106886801A (en) * 2017-04-14 2017-06-23 北京图森未来科技有限公司 A kind of image, semantic dividing method and device
CN107220980A (en) * 2017-05-25 2017-09-29 重庆理工大学 A kind of MRI image brain tumor automatic division method based on full convolutional network
CN107346436A (en) * 2017-06-29 2017-11-14 北京以萨技术股份有限公司 A kind of vision significance detection method of fused images classification

Also Published As

Publication number Publication date
CN108268870A (en) 2018-07-10

Similar Documents

Publication Publication Date Title
CN108268870B (en) Multi-scale feature fusion ultrasonic image semantic segmentation method based on counterstudy
CN109493308B (en) Medical image synthesis and classification method for generating confrontation network based on condition multi-discrimination
CN110930397B (en) Magnetic resonance image segmentation method and device, terminal equipment and storage medium
CN108492297B (en) MRI brain tumor positioning and intratumoral segmentation method based on deep cascade convolution network
CN111784671B (en) Pathological image focus region detection method based on multi-scale deep learning
Jannesari et al. Breast cancer histopathological image classification: a deep learning approach
Lin et al. Automatic retinal vessel segmentation via deeply supervised and smoothly regularized network
CN109754007A (en) Peplos intelligent measurement and method for early warning and system in operation on prostate
CN112102266A (en) Attention mechanism-based cerebral infarction medical image classification model training method
Zheng et al. Stain standardization capsule for application-driven histopathological image normalization
CN112150476A (en) Coronary artery sequence vessel segmentation method based on space-time discriminant feature learning
CN108629772A (en) Image processing method and device, computer equipment and computer storage media
Mhaske et al. Deep learning algorithm for classification and prediction of lung cancer using CT scan images
CN110991254B (en) Ultrasonic image video classification prediction method and system
CN117015796A (en) Method for processing tissue images and system for processing tissue images
CN114648806A (en) Multi-mechanism self-adaptive fundus image segmentation method
WO2023014789A1 (en) System and method for pathology image analysis using a trained neural network and active learning framework
Guo et al. CAFR-CNN: coarse-to-fine adaptive faster R-CNN for cross-domain joint optic disc and cup segmentation
CN111383222A (en) Intervertebral disc MRI image intelligent diagnosis system based on deep learning
Eltoukhy et al. Classification of multiclass histopathological breast images using residual deep learning
CN116664911A (en) Breast tumor image classification method based on interpretable deep learning
CN113870194B (en) Breast tumor ultrasonic image processing device with fusion of deep layer characteristics and shallow layer LBP characteristics
Dovbysh et al. Decision-Making support system for diagnosis of breast oncopathologies by histological images
CN112967295B (en) Image processing method and system based on residual network and attention mechanism
Singh et al. A Shallow Convolution Neural Network for Predicting Lung Cancer Through Analysis of Histopathological Tissue Images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20200518

Address after: 400000 No. 12 Chen Cheng Road, Shapingba District, Chongqing

Applicant after: CHONGQING NORMAL University

Address before: No. 69 lijiatuo Chongqing District of Banan City Road 400054 red

Applicant before: Chongqing University of Technology

GR01 Patent grant
GR01 Patent grant