CN113469186A - Cross-domain migration image segmentation method based on small amount of point labels - Google Patents

Cross-domain migration image segmentation method based on small amount of point labels Download PDF

Info

Publication number
CN113469186A
CN113469186A CN202110734847.2A CN202110734847A CN113469186A CN 113469186 A CN113469186 A CN 113469186A CN 202110734847 A CN202110734847 A CN 202110734847A CN 113469186 A CN113469186 A CN 113469186A
Authority
CN
China
Prior art keywords
domain
network
image
segmentation
counting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110734847.2A
Other languages
Chinese (zh)
Other versions
CN113469186B (en
Inventor
彭佳林
王玉柱
易佳锦
邱达飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huaqiao University
North China University of Water Resources and Electric Power
Original Assignee
Huaqiao University
North China University of Water Resources and Electric Power
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huaqiao University, North China University of Water Resources and Electric Power filed Critical Huaqiao University
Priority to CN202110734847.2A priority Critical patent/CN113469186B/en
Publication of CN113469186A publication Critical patent/CN113469186A/en
Application granted granted Critical
Publication of CN113469186B publication Critical patent/CN113469186B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N10/00—Quantum computing, i.e. information processing based on quantum-mechanical phenomena
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Mathematics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a cross-domain migration image segmentation method based on a small number of point labels, and belongs to the technical field of image processing. In the multi-target segmentation task of the image, only a small number of target central points in target domain data are required to be labeled, then under the guidance of a model trained by source domain data, the target domain data labeled with a small number of points are subjected to learning of a segmentation prediction task, a space counting task and a quantitative counting task, and the model is learned to the characteristic representation of the target domain discriminability by using a countermeasure network in an output space, so that the cross-domain segmentation effect of the target domain is improved, and the automatic image segmentation model with the competitiveness compared with an unsupervised model is obtained. The method disclosed by the invention only needs to utilize the existing marked data and a small amount of point marks on the new application scene data, greatly reduces the labor cost and obtains a competitive segmentation effect, and can be applied to multi-target object segmentation in the fields of natural scene images, medical images and the like.

Description

Cross-domain migration image segmentation method based on small amount of point labels
Technical Field
The invention belongs to the technical field of image processing, relates to a method for segmenting multiple targets in an image, and particularly relates to a cross-domain migration image automatic segmentation method based on a small number of point labels.
Background
Semantic segmentation of a large number of small objects in an image is one of widely applied technologies in the field of image processing, and extraction of image semantic information is achieved by performing pixel-by-pixel classification on the image. With the development of the deep learning technology, the performance of the depth image segmentation technology based on full supervision is greatly improved. But fully supervised methods based on deep learning require a large number of pixel level labels. However, in an actual application scenario, a large amount of manpower and material resources are required to label the targets one by one at a pixel level, and the labeling cost is high.
For the problems, researchers provide an unsupervised domain self-adaptive segmentation method, and model migration trained by a labeled data domain in the existing application scene is applied to a new unlabeled data domain. For example, patent application No. CN 111402257 a, entitled "medical image automatic segmentation method based on multitask system cross-domain migration" obtains good segmentation effect by integrating geometric clues of label domain and visual clues of image domain to guide domain adaptation together, but because data distribution of different data domains is greatly different and new data domain label information is completely lacked, the segmentation performance of the cross-domain segmentation model learned by the method is limited. Besides high-cost pixel-level marks, in the research work of some weak supervision segmentation methods, marking information such as center point marking, boundary box marking, doodle marking and the like can obtain good segmentation effect and greatly reduce marking cost. However, when the amount of the massive image data and the number of the targets in the image are too large, the weak labeling of each target still has a low labeling overhead, and the labeling of a small amount of targets can greatly reduce the labeling cost on the basis of providing a small amount of accurate labeling information. Therefore, a small amount of weak marks on a new target data domain are used for providing real discrimination information for target domain data, the segmentation model trained by the marked source domain data can be guided to be better migrated and generalized to the target domain, and the segmentation precision of the cross-domain segmentation model can be improved under the condition of a small amount of marking cost.
Disclosure of Invention
In order to solve some problems of a cross-domain image semantic segmentation method mentioned in the background technology, the invention provides a cross-domain migration image segmentation method based on a small number of point labels. The invention designs a space counting sub-network and a quantification counting sub-network by considering general source domain data knowledge such as position and quantity information, positions the segmentation targets and restricts the quantity of the segmentation targets, and on the output space, based on the thought of counterstudy, the domain discriminator is utilized to assist the segmentation network to have better segmentation effect on the target domain data.
The invention adopts the following technical scheme:
a cross-domain migration image segmentation method based on a small number of point labels comprises the following steps:
s1, pre-training a cross-domain migration image segmentation model by using source domain data, which comprises 5 steps as follows:
s11, preprocessing source domain image data;
s12, constructing a cross-domain migration image segmentation model;
preferably, the cross-domain migration image segmentation model is divided into a semantic segmentation sub-network, a space counting sub-network and a quantization counting sub-network, and parameters of the quantization counting sub-network are not optimized on source domain data;
s13, designing a loss function;
representing a source domain image as xsIts corresponding pixel level label is ysThe corresponding point is marked with a graph rsThe number of points is recorded as Ts。psAs source domain image xsSegmentation prediction result through semantic segmentation sub-network, qsAs source domain image xsThe predicted outcome across the space counting sub-network,
Figure BDA0003141211990000021
in order to quantify the prediction of the counting sub-network,
Figure BDA0003141211990000022
expressing the mathematic expectation, expressing the category c, and taking K as the total number of image pixel points. The semantic segmentation subnetwork optimization objective loss function under the source domain data is as follows:
Figure BDA0003141211990000023
the space-counting subnetwork optimization objective loss function under the source domain data is as follows:
Figure BDA0003141211990000024
wherein the content of the first and second substances,
Figure BDA0003141211990000025
Figure BDA0003141211990000026
is the variance σ1Gaussian function of gsFor marking points with a map rsThe corresponding gaussian point is marked with a map,
Figure BDA0003141211990000027
representing a pixel value corresponding to a Gaussian point mark image pixel i in a source domain; weight graph
Figure BDA0003141211990000028
Figure BDA0003141211990000029
Is the variance σ2The function of the gaussian function of (a) is,
Figure BDA00031412119900000210
represents a weight map betasThe weighted value corresponding to the middle pixel i;
Figure BDA00031412119900000211
representing a network prediction output result corresponding to the source domain image pixel i; lambda is a weight parameter, and K is the total number of image pixel points;
s14, pre-training a cross-domain migration image segmentation model by using the data set and the loss function;
and S15, storing the parameters of the cross-domain migration image segmentation model, and initializing the parameters of the training model on the subsequent target domain data.
And S2, training a quantitative counting model.
Preferably, the quantitative counting model is initialized by using a feature extraction network parameter of the cross-domain migration image segmentation model;
s21, preprocessing the source domain image data set;
s22, constructing a quantitative counting model;
preferably, the quantization counting model has the same structure as the semantic segmentation sub-network, and the final single-channel prediction graph obtained by final up-sampling passes through a self-adaptive average pooling layer to obtain the final prediction target number;
s23, designing a loss function;
the mean square error is used as the optimization objective function for the target number prediction output, as follows:
Figure BDA0003141211990000031
Figure BDA0003141211990000032
expressing the mathematical expectation, xsRepresenting the source domain image, TsRepresents the true number of mitochondria in the normal state,
Figure BDA0003141211990000033
representing the prediction result of the source domain image passing through a quantization counting network;
s24, training a quantitative counting model by using the data set and the corresponding loss function;
and S25, storing the network parameter model for estimating the number of mitochondria in the target domain.
And S3, training a cross-domain migration image segmentation model based on a small number of point labels on the target domain image data set with the small number of point labels.
Preferably, the cross-domain migration image segmentation model based on the small amount of point labels performs parameter optimization by simultaneously using the source domain image data set, the complete label of the source domain image, the target domain image data set and the small amount of point labels of the target domain data.
S31, preprocessing the data sets of the source domain and the target domain;
s32, designing a cross-domain migration image segmentation model based on a small number of point labels;
s33, designing corresponding loss functions for different sub-networks;
s34, training and optimizing a cross-domain migration image segmentation model by using the data set and the loss function;
s35, storing the generated cross-domain migration image segmentation model for segmentation prediction on the target domain;
further, the cross-domain migration image segmentation model of step S32 includes three sub-networks, and the three sub-networks share the feature extraction network; obtaining semantic information of the image through a semantic segmentation sub-network, and outputting a prediction graph with the same resolution as that of the input two channels; the space counting sub-network has the same network structure as the semantic segmentation sub-network, outputs a prediction graph with a single channel and the same input resolution, and is used for positioning the space position of a target; a quantization counting sub-network for predicting the target number by using the output of the space counting sub-network as input; further, although the source domain data and the target domain data are different, they have a large similarity in the tag space, and therefore it is considered to use a domain discriminator on the output space to make a weak shape constraint on the output space.
Preferably, the present invention constructs a domain discriminator, the domain discriminator inputs the segmentation prediction labels of the source domain and the segmentation prediction labels of the target domain, the output image has the same size as the input image, and each pixel of the output image represents whether the pixel is from the source domain prediction label or the target domain prediction label.
Furthermore, corresponding loss functions are designed for different sub-networks.
The source domain and target domain data are used simultaneously in the training of the cross-domain migration image segmentation model, and the loss function adopted for the output of the source domain data through the network is the same as that described in step S13.
Considering that there is only a small amount of point label information on the target domain, the space count loss function on the target domain is defined as follows:
Figure BDA0003141211990000041
preferably, the loss function introduces a weight map w, each value w of the weight mapiThe definition is as follows:
Figure BDA0003141211990000042
wherein h istThe target domain label only has a small amount of foreground labeling information, so background information is required to be added from the segmentation prediction graph step by step according to the iteration number by a method of setting a soft threshold value, and m istThe medium pixel value is 0 and is represented as a background pixel; the soft threshold varies with the number of model iterations, mtEach value of
Figure BDA0003141211990000043
The definition is as follows:
Figure BDA0003141211990000044
ρ is defined as follows:
Figure BDA0003141211990000045
wherein epsilon is the total iteration times of the model, and epsilon is the current iteration times of the model.
Preferably, the loss of the discriminator is cross entropy loss, as follows:
Figure BDA0003141211990000046
wherein DpredRepresentation domain discriminator, psAnd ptRespectively source domain image xsAnd a target domain image xtAnd (4) performing segmentation prediction results of the sub-network through semantic segmentation.
Preferably, the arbiter network parameters are fixed, and the semantic segmentation sub-network is optimized by minimizing the following loss function, the formula is as follows:
Figure BDA0003141211990000047
preferably, the number of mitochondria in the target domain is constrained by a quantitative counting model pre-trained in the source domain. Predicting mitochondrial number in a target domain using a quantitative counting model trained on source domain data
Figure BDA0003141211990000048
Due to the fact that the source domain and the target domain have certain domain differences and the quantitative counting model trained on the source domain has certain deviation. Considering the error asEpsilon, the predicted number of target domain mitochondria obtained by a quantization counting subnetwork in the cross-domain migration image segmentation model
Figure BDA0003141211990000049
Within a certain range, as shown by the following formula:
Figure BDA00031412119900000410
this loss can place a constraint on the number of mitochondria in the target domain.
In summary, the cross-domain semantic segmentation sub-network is obtained by minimizing an objective function, namely
Figure BDA0003141211990000051
Wherein λpredRepresenting a preset first non-negative hyperparameter; lambda [ alpha ]countIndicating a preset second non-negative hyperparameter.
As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following advantages:
(1) the invention only requires that the central points of partial targets are marked on each picture with multiple targets in the target domain, so that the marking work becomes efficient and simple;
(2) based on general domain knowledge of data to be segmented, such as spatial position information and quantity information of a target, the invention provides a spatial counting sub-network and a quantitative counting sub-network, wherein the spatial counting sub-network assists a model to better position the target, the quantitative counting sub-network performs quantity constraint on the target predicted by the model, and by combining the two sub-networks, the parameters of a segmentation prediction network are optimized, and the cross-domain model is assisted to better segment;
(3) according to the method, a domain discriminator is introduced in an output space, weak shape constraint is carried out on a predicted image of a target domain, model learning is helped to have more discriminative feature expression on the target domain, and the segmentation effect is further improved;
drawings
FIG. 1 is a structural diagram of a cross-domain migration image segmentation method based on a small number of point labels according to the present invention;
FIG. 2 is a flow chart of the method of the present invention;
FIG. 3 is a graph of the segmentation results of the method of the present invention and other segmentation methods; (a) the target domain image and the standard segmentation graph thereof are obtained; (b) a domain-free self-adaptive segmentation result graph is obtained; (c) a segmentation result graph of the DAMT-Net method is shown; (d) is a segmentation result graph of the invention.
Detailed Description
The invention is further described below by means of specific embodiments. It should be noted that the specific embodiments described herein are only for convenience of describing and explaining specific embodiments of the present invention, and are not intended to limit the present invention.
The method provided by the invention can be applied to the segmentation of multiple targets, such as the segmentation of cells and mitochondria in the medical field. The disclosed medical image mitochondrial data set is used as a specific example below. Specifically, the source domain data set is a drosophila melanogaster data set of the third age, and the target domain data set is a mouse hippocampus data set. The data set of fruit flies of the third age consists of 20 1024 × 1024 pictures, there are 389 mitochondria in total, during training, 10 are randomly selected as training sets, and the remaining 10 are test sets; the mouse hippocampal data set comprises a training set and a test set, wherein the training set comprises 165 images of 1024 × 768 and total 2511 mitochondria, the test set comprises 165 images of 1024 × 768 and total 2623 mitochondria, and in the following specific examples, 15% of the mitochondria in the mouse hippocampal training data set, namely 377 mitochondrial center points are randomly selected and labeled as label information.
Referring to fig. 1 and fig. 2, the present invention provides a cross-domain migration image segmentation method based on a small number of point labels, which specifically includes:
and step 10, preprocessing data.
Randomly cutting image data of training data and corresponding label data into 512 x 512 sizes, normalizing the image data to be used as network input, setting foreground pixel values of the label data to be 1,the background pixel value is set to 0; performing Gaussian blur on point label images on a label data set of a training set to obtain a pseudo label as a real label of a training space counting branch, wherein a Gaussian blur parameter sigma1Set to 61;
and step 20, initializing parameters.
Random seeds during network training are fixed, so that the training results of the same experimental model are always kept consistent; the iteration times of the training loop are set to be 10 ten thousand, an SGD optimizer is used for training the cross-domain model, and the initial learning rate is 10-4Momentum parameter of 0.9, weight attenuation coefficient of 5 × 10-5(ii) a Training the discriminator model, using Adam optimizer, initial learning rate set to 5 × 10-5The polynomial attenuation coefficient is 0.9;
and step 30, training a cross-domain migration image segmentation network on the source domain data.
Step 301, defining a network structure. The defined partition prediction network uses the encoding branch and decoding branch of CS-Net, and the network structure is specifically referred to (Peng J, Luo Z. CS-Net: instant-aware cellular segmentation with hierarchical dimension-compensated distributions and slice-active learning [ J ]. arXiv prediction arXiv:2101.02877,2021.). Other similar networks may be used in particular implementations. The CS-Net uses a hierarchical dimension decomposition convolution module to extract multi-level upper and lower information, so that features can be better extracted and the model is lighter. The invention designs and adds a space counting sub-network and a quantification counting sub-network on the basis of the space counting sub-network, the three sub-networks share a feature extraction network, the output of the space counting sub-network sampled for the fourth time of CS-Net is used as the input, and the structure of the space counting sub-network is the same as that of the last up-sampling sub-network. Finally, 1 × 1 convolution with one channel being 1 is used to obtain the final output with the single channel and the original image resolution being the same; the output layer of the quantization counting sub-network uses two hierarchical dimension decomposition convolution modules and a full connection layer, the output of the space counting branch is used as the input of the module, and the predicted number is output;
step 302, inputting the preprocessed source domain training data into the network, and aligning the image data and corresponding data to reduce overfittingPerforming online data enhancement on the tag data, such as random inversion, random brightness transformation, motion blur, etc., wherein the motion blur and the brightness transformation are applied to the image data and not applied to the tag data, and obtaining a segmentation prediction output graph p of the source domain data through a semantic segmentation sub-network and a space counting sub-networksAnd a space counting diagram qs；
Step 303, computing a segmentation prediction map p output by the semantic segmentation subnetworksAnd source domain pixel level label graph ysThe calculation formula of the loss function value of (c) is as follows:
Figure BDA0003141211990000071
step 304, calculating the output q from the space-counting subnetworksAnd source domain Gaussian point label graph gsThe space-counting subnetwork optimization objective loss function is as follows:
Figure BDA0003141211990000072
empirically, λ is set to 3;
step 305, through minimization
Figure BDA0003141211990000073
Optimizing network parameters; empirically, λcountSet equal to 0.1;
step 306, storing the best parameters of the cross-domain migration image segmentation network model according to the test results of the verification set; specifically, the verification set is verified every 1000 iterations during model training, and network model parameters are stored according to the best JAC (Jaccard coefficient) result;
step 40, training a quantitative counting model on the source domain data, specifically comprising the following steps:
step 401, defining a quantization counting network model; the quantization counting model has the same network structure as the semantic segmentation sub-network, and the final single-channel output of the model is processed by a self-adaptive flatThe output size of the self-adaptive average pooling layer is set to be 1, and the final scalar value output is obtained
Figure BDA0003141211990000074
Step 402, inputting the processed source domain training data into a quantitative counting model to obtain a target predicted value of the source domain data
Figure BDA0003141211990000075
The optimization objective function is as follows:
Figure BDA0003141211990000076
step 403, reversely propagating the size of the update parameter;
step 404, saving the best quantitative counting model parameters according to the test result of the verification set every 1000 times of iteration;
step 50, training a cross-domain migration image segmentation model based on a small number of point labels on a target domain image data set with a small number of point labels, and specifically comprising the following steps:
step 501, defining a cross-domain migration image segmentation model, wherein the structure of the model is as described in step 301; defining a quantization count model, the network structure being as described in step 401;
step 502, initializing the parameters of the cross-domain migration image segmentation model by using the network parameters saved in step 306, initializing the quantitative counting model by using the network parameters saved in step 404, and fixing the parameters;
step 503, fixing the network parameters of the arbiter when the network starts training;
step 504, initializing the original domain label of the target domain data to be 0, and initializing the original domain label of the source domain data to be 1;
step 505, inputting the source domain data into the semantic segmentation sub-network to obtain the segmentation prediction output ps；
Step 506, calculating the segmentation loss through the following formula;
Figure BDA0003141211990000081
step 507, inputting the source domain data into the space counting sub-network to obtain the space counting prediction output qs；
Step 508, calculating the mean square error loss according to the following equation;
Figure BDA0003141211990000082
where λ is a weighting parameter, empirically, λ is set to 3;
step 509, input the target domain data into the semantic segmentation sub-network to obtain the segmentation prediction output pt；
Since there is no pixel-level label in the target domain, the loss cannot be directly calculated, but the resulting ptCan be used to pick background pixels;
step 510, inputting the target domain data into a space counting sub-network to obtain a space counting prediction output qt；
Step 511, calculating the loss according to the following function;
Figure BDA0003141211990000083
w is a weight map which varies with the number of iterations, the prediction map p being derived from the segmentationtScreening out background pixels according to a threshold value rhoupperSet to 0.7, as described in step S33, by giving a smaller number of marked dot regions a larger loss weight, λ is set to 3 in this embodiment, that is, the loss weight of the region within a radius of 11 pixels with a dot mark is 6, the loss weight of the ring region within a radius of 31 pixels with a dot mark is 3, the background region is set to 1, and the loss weights of other pixels are set to 0, and no loss calculation is performed;
step 512, according to the quantitative counting model trained in the source domain data, the number of mitochondria in the target domain is constrained by using the following loss function, and the loss calculation formula is as follows:
Figure BDA0003141211990000084
where ε is a perturbation parameter, will
Figure BDA0003141211990000085
Constrained at TtEmpirically set to 3, TtThe number of target domain mitochondria predicted by a quantitative counting model obtained by training source domain data,
Figure BDA0003141211990000086
(ii) a predicted number of mitochondria obtained for a quantitative counting branch of the cross-domain model;
step 513, training the domain discriminator, and updating the parameters of the domain discriminator;
step 514, fixing the network parameters of the domain arbiter;
step 515, partition the source domain data into prediction labels psTarget domain data segmentation prediction label ptInputting the data into a semantic map domain discriminator to respectively obtain a source domain data semantic map domain discrimination label and a target domain data semantic map domain discrimination label;
in step 516, the domain arbiter is implemented by minimizing the following objective function:
Figure BDA0003141211990000091
wherein
Figure BDA0003141211990000092
Expressing the mathematical expectation, wherein DpredRepresentation domain discriminator, psAnd ptRespectively source domain image xsAnd a target domain image xtPartitioning the prediction results of the sub-network by semantic partitioning;
step 517, calculating and minimizing the following loss function values, and enabling the data characteristic distribution of the target domain data to be close to the data characteristic distribution of the source data on the output space;
Figure BDA0003141211990000093
wherein
Figure BDA0003141211990000094
Expressing the mathematical expectation, DpredIndicating a domain discriminator, ptIs a target domain image xtPartitioning the prediction results of the sub-network by semantic partitioning;
in combination with the above, the cross-domain migration image segmentation model is obtained by minimizing an objective function, namely
Figure BDA0003141211990000095
In particular, λcount＝0.1，λpred＝0.001；
518, repeating 505 to 517, and alternately optimizing
Figure BDA0003141211990000096
And updating the parameters by adopting a back propagation method to obtain a final cross-domain migration image segmentation model.
In order to illustrate the effectiveness of the method provided by the invention, corresponding performance evaluation is carried out, DSC (Dice coefficient) and JAC (Jaccard coefficient) indexes are adopted to compare the segmentation quality of the network on the target domain test set with the segmentation quality of the real label G, the DSC and JAC have better segmentation effect, and the DSC and JAC are defined as follows:
Figure BDA0003141211990000097
Figure BDA0003141211990000098
as shown in table 1 and fig. 3, a comparative test result of the present invention using 15% point labeling and non-adaptive method NoAdapt (model trained on source domain is directly applied on target domain), representative unsupervised domain adaptive method DAMT-Net (domain adaptation is guided by integrating geometric clues of tag domain and visual clues of image domain, considering target domain is not tagged).
TABLE 1 comparison of the method of the invention with the baseline method, unsupervised Domain adaptive representative method
Figure BDA0003141211990000099
Figure BDA0003141211990000101
The method described by DAMT-Net is specifically described in the literature "Peng J, Yi J, Yuan Z. Unveruperviced mitochondria segmentation in EM images via domain adaptive multi-task learning [ J ]. IEEE Journal of Selected Topics in Signal Processing,2020,14(6): 1199-.
As can be seen from table 1, compared with the JAC index, the JAC of NoAdapt is only 54.4%, which indicates that a model trained on a drosophila melanogaster dataset of the third age cannot be well applied to a hippocampus data set of mice, the JAC of a representative unsupervised domain adaptive method DAMT-Net is only 60.8%, but the JAC of the method provided by the invention reaches 77.6%, only 15% of center point labels are utilized, and an effect improvement of 23.2% is achieved on NoAdapt, and an effect improvement of 16.8% is achieved compared with the DAMT-Net method, which indicates that the segmentation effect can be greatly improved by introducing a small number of center point labels, and the effectiveness of the provided method is also proved.
The present invention has been described above by way of a specific embodiment, but the design concept of the present invention is not limited thereto, and any insubstantial modification of the present invention using this concept shall fall within the scope of infringing on the scope of the present invention.

Claims (10)

1. A cross-domain migration image segmentation method based on a small amount of point labels is characterized in that the high-performance segmentation of a target domain image is realized by using the knowledge of the existing completely labeled source domain image and a small amount of point label information of the target domain image to be segmented, and the method specifically comprises the following steps:
s1, training a cross-domain migration image segmentation model on the source domain image data set with the pixel level marks;
s2, training a quantitative counting model on the source domain image data set with the pixel level marks;
and S3, training a cross-domain migration image segmentation model based on a few point labels by combining the source domain image data set and the target domain image data set, wherein the method specifically comprises the following steps:
s31, inputting target domain images marked by a small number of points and source domain images marked by a complete pixel level into a pre-trained cross-domain migration image segmentation model on source domain data to obtain corresponding segmentation output as the input of a domain discriminator, based on the thought of counterstudy, the discriminator is used for distinguishing whether predicted images come from a source domain or a target domain, and the domain discriminator of an output space is used for optimizing parameters of the cross-domain migration image segmentation model;
s32, inputting target domain images marked by a small number of points into a pre-trained cross-domain migration image segmentation model on source domain data to obtain corresponding space counting output, picking out reliable pseudo background information by utilizing the output of a semantic segmentation sub-network, and learning a target domain space counting task by combining a small number of target central point pixels;
s33, learning a quantization counting task on the target domain by using a quantization counting model of the source domain; and fixing the network parameters of the pre-trained quantitative counting model on the source domain data, inputting the target domain image into the network to obtain the target estimation quantity in the target domain image, and constraining the quantitative counting prediction on the target domain by using the quantitative counting model in the step S2 to learn the quantitative counting task of the target domain.
And S34, saving the model parameters of the cross-domain migration image segmentation model for the segmentation prediction of the new target domain image.
2. The method as claimed in claim 1, wherein in the multi-target segmentation, the small number of point labels is to label the central points of the small number of targets in the target domain image.
3. The method for image segmentation based on cross-domain migration with a small number of point labels as claimed in claim 2, wherein the semantic segmentation sub-network, the spatial counting sub-network and the quantization counting sub-network form a cross-domain migration image segmentation model based on a small number of point labels; the semantic segmentation sub-network and the space counting sub-network share the characteristic extraction network parameters; the semantic division sub-network, the space counting sub-network and the quantization counting sub-network all adopt a decoder-encoder structure, the semantic division sub-network and the space counting sub-network respectively carry out image division result prediction and space counting prediction, and the quantization counting sub-network carries out quantization counting task learning on prediction output of space counting.
4. The method for image segmentation based on cross-domain migration of a small number of point labels as claimed in claim 3, wherein the semantic segmentation sub-network, the spatial counting sub-network and the quantization counting sub-network are pre-trained using source domain images with pixel-level labels;
the semantic segmentation subnetwork optimization objective loss function is as follows:
Figure FDA0003141211980000021
wherein the content of the first and second substances,
Figure FDA0003141211980000022
expressing the mathematical expectation, xsRepresenting the source domain image, ysA pixel level label corresponding to the source domain image, c represents a category,
Figure FDA0003141211980000023
representing a source domain image; c pixel level labels of the category;
Figure FDA0003141211980000024
representing the segmentation prediction result of the c category obtained by the source domain image through a semantic segmentation sub-network;
the space-counting subnetwork optimization objective loss function is as follows:
Figure FDA0003141211980000025
wherein the content of the first and second substances,
Figure FDA0003141211980000026
Figure FDA0003141211980000027
is the variance σ1Gaussian function of gsFor marking points with a map rsThe corresponding gaussian point is marked with a map,
Figure FDA0003141211980000028
representing a pixel value corresponding to a Gaussian point mark image pixel i in a source domain; weight graph
Figure FDA0003141211980000029
Figure FDA00031412119800000210
Is the variance σ2The function of the gaussian function of (a) is,
Figure FDA00031412119800000211
represents a weight map betasThe weighted value corresponding to the middle pixel i;
Figure FDA00031412119800000212
representing a network prediction output result corresponding to the source domain image pixel i; λ is the weight parameter and K is the image pixelCounting the total number of points;
the quantization count network optimizes the objective loss function as follows:
Figure FDA00031412119800000213
wherein the content of the first and second substances,
Figure FDA00031412119800000214
expressing the mathematical expectation, TsRepresents the true number of mitochondria in the normal state,
Figure FDA00031412119800000215
representing the prediction result of the source domain image through the quantization counting network.
5. The method for image segmentation based on cross-domain migration with a small number of point labels as claimed in claim 4, wherein a quantization counting model of source domain training is the same as a semantic segmentation sub-network structure, a feature extraction network shared by the semantic segmentation sub-network and a spatial counting sub-network is used as an initialization parameter, a final output is obtained by passing the final output through an adaptive average pooling layer, and a corresponding target loss function is used for training.
6. The method for image segmentation based on cross-domain migration with small number of point labels as claimed in claim 5, wherein the semantic segmentation sub-network and the space counting sub-network are initialized with parameters of a pre-trained model of the source domain, and then are trained with the target domain image labeled with a small number of points and the pixel-level labeled source domain image; the quantization counting sub-network is trained only by using the target domain image, and the pre-trained quantization counting model of the source domain fixes the parameters thereof as a single quantity estimation model.
7. The method as claimed in claim 6, wherein the target domain image labeled with a few dots and the source domain image labeled with a complete pixel level are divided into twoInputting the segmentation predictions of a source domain and a target domain into a domain discriminator respectively in a cross-domain migration image segmentation model initialized by source domain pre-training; optimizing a semantic segmentation sub-network and a domain discriminator based on counterlearning; representing the target domain image as xt，ptIs a target domain image xtThrough the segmentation prediction result of the semantic segmentation sub-network, the optimized target loss of the cross-domain migration image segmentation model is as follows:
Figure FDA00031412119800000216
the domain arbiter optimization penalty is as follows:
Figure FDA0003141211980000031
wherein D ispredRepresentation domain discriminator, psAnd ptRespectively source domain image xsAnd a target domain image xtPartitioning the prediction results of the sub-network by semantic partitioning; dpred(pt) The prediction output of the target domain segmentation prediction result through the discriminator is shown; dpred(ps) And the prediction output of the source domain division prediction result through the discriminator is shown.
8. The method as claimed in claim 7, wherein the target domain image with a small number of point labels is input into a spatial counting sub-network pre-trained in a source domain, and a pseudo background and a real sparse point label are selected in combination with the prediction output in the semantic segmentation sub-network for learning a spatial counting task, and a spatial counting loss function on the target domain is as follows:
Figure FDA0003141211980000032
wherein, the Gaussian point mark map
Figure FDA0003141211980000033
Figure FDA0003141211980000034
Is the variance σ1Gaussian function of rtThe map is marked for the center point,
Figure FDA0003141211980000035
mark diagram g for representing Gaussian pointstA pixel value corresponding to the pixel i; weight graph
Figure FDA0003141211980000036
Figure FDA0003141211980000037
Is the variance σ2The function of the gaussian function of (a) is,
Figure FDA0003141211980000038
represents a weight map betatA weighted value corresponding to the pixel i; w is a weighted graph with a balance of pseudo-background and sparse foreground, wiRepresenting the weight value corresponding to the pixel i in the balance weight map w;
Figure FDA0003141211980000039
a prediction output representing a target domain spatial count output image pixel i; k represents the total number of image pixel points; λ is a weight parameter.
9. The method as claimed in claim 8, wherein the quantity of target domain images is estimated as T by using a quantitative counting model of a source domaintThe sub-net prediction output of the quantization count of the target domain is noted
Figure FDA00031412119800000310
Will TtReference estimate value pair as target domain
Figure FDA00031412119800000311
And (3) carrying out quantity constraint, carrying out quantitative counting task learning of a target domain, and optimizing a target loss function as follows:
Figure FDA00031412119800000312
where ε represents a perturbation parameter, will
Figure FDA00031412119800000313
Constrained at TtWithin the perturbation range of (a).
10. The method of claim 9, wherein the sub-network of semantic segmentation is obtained by minimizing an objective function that is
Figure FDA00031412119800000314
Wherein λ ispredRepresenting a preset first non-negative hyperparameter; lambda [ alpha ]countIndicating a preset second non-negative hyperparameter.
CN202110734847.2A 2021-06-30 2021-06-30 Cross-domain migration image segmentation method based on small number of point labels Active CN113469186B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110734847.2A CN113469186B (en) 2021-06-30 2021-06-30 Cross-domain migration image segmentation method based on small number of point labels

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110734847.2A CN113469186B (en) 2021-06-30 2021-06-30 Cross-domain migration image segmentation method based on small number of point labels

Publications (2)

Publication Number Publication Date
CN113469186A true CN113469186A (en) 2021-10-01
CN113469186B CN113469186B (en) 2023-06-23

Family

ID=77874346

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110734847.2A Active CN113469186B (en) 2021-06-30 2021-06-30 Cross-domain migration image segmentation method based on small number of point labels

Country Status (1)

Country Link
CN (1) CN113469186B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114239753A (en) * 2022-02-23 2022-03-25 山东力聚机器人科技股份有限公司 Migratable image identification method and device
CN114240955A (en) * 2021-12-22 2022-03-25 电子科技大学 Semi-supervised cross-domain self-adaptive image segmentation method
CN114445413A (en) * 2022-04-07 2022-05-06 宁波康达凯能医疗科技有限公司 Inter-frame image semantic segmentation method and system based on domain self-adaptation
CN114638964A (en) * 2022-03-07 2022-06-17 厦门大学 Cross-domain three-dimensional point cloud segmentation method based on deep learning and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275713A (en) * 2020-02-03 2020-06-12 武汉大学 Cross-domain semantic segmentation method based on countermeasure self-integration network
CN111402257A (en) * 2020-03-11 2020-07-10 华侨大学 Medical image automatic segmentation method based on multi-task collaborative cross-domain migration
AU2020103905A4 (en) * 2020-12-04 2021-02-11 Chongqing Normal University Unsupervised cross-domain self-adaptive medical image segmentation method based on deep adversarial learning
CN112819873A (en) * 2021-02-05 2021-05-18 四川大学 High-generalization cross-domain road scene semantic segmentation method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275713A (en) * 2020-02-03 2020-06-12 武汉大学 Cross-domain semantic segmentation method based on countermeasure self-integration network
CN111402257A (en) * 2020-03-11 2020-07-10 华侨大学 Medical image automatic segmentation method based on multi-task collaborative cross-domain migration
AU2020103905A4 (en) * 2020-12-04 2021-02-11 Chongqing Normal University Unsupervised cross-domain self-adaptive medical image segmentation method based on deep adversarial learning
CN112819873A (en) * 2021-02-05 2021-05-18 四川大学 High-generalization cross-domain road scene semantic segmentation method and system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
候伟宁: "基于深度学习的遥感影像去雾与舰船目标检测", 《中国优秀硕士学位论文全文数据库 (工程科技Ⅱ辑)》 *
彭佳林: "基于序列间先验约束和多视角信息融合的肝脏CT图像分割", 《电子与信息学报》 *
贾颖霞;郎丛妍;冯松鹤;: "基于类别相关的领域自适应交通图像语义分割方法", 计算机研究与发展, no. 04 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114240955A (en) * 2021-12-22 2022-03-25 电子科技大学 Semi-supervised cross-domain self-adaptive image segmentation method
CN114240955B (en) * 2021-12-22 2023-04-07 电子科技大学 Semi-supervised cross-domain self-adaptive image segmentation method
CN114239753A (en) * 2022-02-23 2022-03-25 山东力聚机器人科技股份有限公司 Migratable image identification method and device
CN114638964A (en) * 2022-03-07 2022-06-17 厦门大学 Cross-domain three-dimensional point cloud segmentation method based on deep learning and storage medium
CN114445413A (en) * 2022-04-07 2022-05-06 宁波康达凯能医疗科技有限公司 Inter-frame image semantic segmentation method and system based on domain self-adaptation

Also Published As

Publication number Publication date
CN113469186B (en) 2023-06-23

Similar Documents

Publication Publication Date Title
Fan et al. Watching a small portion could be as good as watching all: Towards efficient video classification
CN113469186A (en) Cross-domain migration image segmentation method based on small amount of point labels
CN110580496A (en) Deep migration learning system and method based on entropy minimization
CN111402257B (en) Automatic medical image segmentation method based on multi-task collaborative cross-domain migration
CN112132149B (en) Semantic segmentation method and device for remote sensing image
CN113326731B (en) Cross-domain pedestrian re-identification method based on momentum network guidance
CN114139676A (en) Training method of domain adaptive neural network
CN112052818A (en) Unsupervised domain adaptive pedestrian detection method, unsupervised domain adaptive pedestrian detection system and storage medium
CN113128478A (en) Model training method, pedestrian analysis method, device, equipment and storage medium
CN114863091A (en) Target detection training method based on pseudo label
CN113313166A (en) Ship target automatic labeling method based on feature consistency learning
CN114692732A (en) Method, system, device and storage medium for updating online label
CN115270797A (en) Text entity extraction method and system based on self-training semi-supervised learning
CN116431597A (en) Method, electronic device and computer program product for training a data classification model
US20220156519A1 (en) Methods and systems for efficient batch active learning of a deep neural network
CN114495114B (en) Text sequence recognition model calibration method based on CTC decoder
CN115797701A (en) Target classification method and device, electronic equipment and storage medium
CN115730656A (en) Out-of-distribution sample detection method using mixed unmarked data
CN114078203A (en) Image recognition method and system based on improved PATE
CN112347826B (en) Video continuous sign language recognition method and system based on reinforcement learning
CN114241380A (en) Multi-task attribute scene recognition method based on category label and attribute annotation
CN112819098A (en) Domain self-adaption method based on triple and difference measurement
CN113033212B (en) Text data processing method and device
CN114821337B (en) Semi-supervised SAR image building area extraction method based on phase consistency pseudo tag
CN115546567B (en) Unsupervised domain adaptive classification method, system, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant