CN112164077A - Cell example segmentation method based on bottom-up path enhancement - Google Patents
Cell example segmentation method based on bottom-up path enhancement Download PDFInfo
- Publication number
- CN112164077A CN112164077A CN202011019366.5A CN202011019366A CN112164077A CN 112164077 A CN112164077 A CN 112164077A CN 202011019366 A CN202011019366 A CN 202011019366A CN 112164077 A CN112164077 A CN 112164077A
- Authority
- CN
- China
- Prior art keywords
- network
- branch
- mask
- dimension
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 53
- 238000012360 testing method Methods 0.000 claims abstract description 33
- 238000012795 verification Methods 0.000 claims abstract description 29
- 230000004927 fusion Effects 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 238000010586 diagram Methods 0.000 claims description 28
- 230000002708 enhancing effect Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 3
- 230000005764 inhibitory process Effects 0.000 claims description 2
- 230000037361 pathway Effects 0.000 claims 2
- 238000010276 construction Methods 0.000 abstract 1
- 210000004027 cell Anatomy 0.000 description 32
- 210000004872 soft tissue Anatomy 0.000 description 4
- 210000002919 epithelial cell Anatomy 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 210000004969 inflammatory cell Anatomy 0.000 description 3
- 230000007170 pathology Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 210000005170 neoplastic cell Anatomy 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 210000004881 tumor cell Anatomy 0.000 description 2
- 206010061218 Inflammation Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000338 in vitro Methods 0.000 description 1
- 230000004054 inflammatory process Effects 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30024—Cell structures in vitro; Tissue sections in vitro
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Abstract
A cell instance segmentation method based on bottom-up path enhancement comprises the steps of data preprocessing, instance segmentation network construction, instance segmentation network training, model storage, instance segmentation network verification and instance segmentation network testing. The method comprises the steps of extracting features in a picture by adopting a backbone network branch in the step of constructing an example segmentation network, predicting positioning, classification and mask coefficient information of a head branch generation example, generating a prototype mask by a prototype mask branch, and carrying out example segmentation on cells in the picture by linearly combining the mask coefficient and the prototype mask. Compared with the existing yoract method, the method has the advantages that the characteristic fusion part in the example segmentation network is improved and constructed, the information in the picture is fully utilized, the accuracy of positioning and classifying the object is improved, and the method can be used for positioning, classifying and example segmentation of cells.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to cell classification, positioning and example segmentation in pictures.
Background
Example segmentation is the combination of target detection and semantic segmentation, the target is detected in the image, and then each pixel is labeled; with the development of the internet and the proliferation of personal intelligent mobile devices, people are generating, storing and using a large number of pictures; example segmentation techniques are fundamental and challenging in computer vision. Due to the wide application scene and research value, the technology attracts more and more attention in both academic and industrial fields.
Currently mainstream object instance segmentation methods can be divided into two major categories: the method is based on a top-down method (such as Mask R-CNN) of the ROI and a bottom-up method based on pixel-by-pixel clustering, and the main problems of the methods are that the method has multiple stages, multiple super parameters, and the segmentation result is not fine enough.
In the technical field of cell instance segmentation, a technical problem to be solved at present is to provide a cell instance segmentation method which is simple, fast in positioning speed and high in positioning precision.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a cell example segmentation method based on bottom-up path enhancement, which is simple, high in positioning speed and high in positioning accuracy.
The technical scheme adopted for solving the technical problems comprises the following steps:
(1) data pre-processing
Taking an original picture from the PanNuke data set and dividing the original picture into a verification set, a test set and a training set, wherein the number ratio of the verification set to the test set to the training set is 1: 1: and 4, the pictures in the training set, the verification set and the test set are 550 multiplied by 550 pixels in size, each cell in the training set, the verification set and the test set is extracted to generate a new mask picture containing a single cell, the mask picture is 550 multiplied by 550 pixels in size, and the pictures in the training set, the verification set and the test set and the mask pictures of the corresponding data sets are converted into tag files by a pycocools tool.
(2) Building instance split networks
The example segmentation network is formed by connecting a backbone network branch (1), a prediction head branch (2) and a prototype mask branch (3), wherein the output of the backbone network branch (1) is connected with the inputs of the prediction head branch (2) and the prototype mask branch (3), and the backbone network branch (1), the prediction head branch (2) and the prototype mask branch (3) are constructed by the following method:
1) building backbone network branches
a characteristic pyramid network extraction characteristic diagram
And inputting each picture in the training set into a basic feature extraction network (1-1) of the feature pyramid network to extract a feature map, so as to obtain a basic feature map set { C1, C2, C3, C4 and C5 }.
b fusion of feature maps
Inputting the basic feature map set { C1, C2, C3, C4, C5} into a feature pyramid network (1-2), and performing feature fusion to obtain a fused feature map set { P }3，P4，P5，P6，P7And (5) performing upsampling by using a nearest neighbor upsampling method to obtain a feature map set of 256 channels, sequentially connecting 5 convolutional layers with the convolutional kernel size of 3 multiplied by 3 and the step length of 2 on the feature map output by the feature pyramid network to form a path enhancement network (1-3), and obtaining a path enhancement network feature map set { N (N) } according to the following formula3，N4，N5，N6，N7Constructing a backbone network branch (1).
Wherein R is a rule activation function, C3The convolution kernel size is 3 x 3, the step size is2, U is a matrix addition, i ∈ {3, 4,5, 6, 7 }.
2) Building a probing tip branch
The prediction head branch comprises a positioning branch, a classification branch and a mask coefficient branch.
a constructing a positioning branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, the anchor characteristic diagram is input into a convolution layer (2-1) and a convolution layer (2-2), and the convolution operation with convolution kernel size of 3 multiplied by 3 and step length of 1 is carried out for 2 times to obtain (A multiplied by S, L)j，Lj) Is (A x S, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjAxS), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by S) to obtain (L)j×LjX A, S), wherein A is 3 anchors set for each pixel point, and S is coordinates of a center point and an upper left corner point of each anchor; will obtain (L)j×LjX a, S) a first dimension of the prediction results is column-stitched to obtain a positioning result of (C, S), where C is determined by:
in the formula LjE {69,35,18,9,5}, j e {1,2,3,4,5}, a localization branch (2-3) is constructed.
b constructing a classification branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, which is input into the convolution layer (2-1) and the convolution layer (2-2) viaConvolution operation with convolution kernel size of 3 × 3 and step size of 1 yields (A × D, L)j，Lj) Is (A x D, L)j，Lj) Exchanging the first dimension and the third dimension of the feature map to obtain (L)j，LjA × D), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by D) to obtain (L)j×LjX A, D) where D represents the number of cell classes ex vivo in the training set, and (L) obtainedj×LjAnd multiplying by A, D) the first dimension of the prediction result is subjected to column splicing operation to obtain the classification result of (C, D), and a classification branch (2-4) is constructed.
c constructing a mask coefficient branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 form an anchor characteristic diagram, which is input into the convolution layer (2-1) and convolution layer (2-2) and is processed by convolution operation with convolution kernel size of 3 x 3 and step size of 1 to obtain (A x K, L)j，Lj) Is (A x K, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjA × K), and then (L)j，LjA x K) in the first dimension, the second dimension and the third dimension of the feature map to obtain (L)j×LjX a, K), where K is the number of generated mask coefficients, K is 32 or 64; will obtain (L)j×LjAnd x A, K) performing column splicing operation on the first dimension of the prediction result to obtain mask coefficients of (C, K), and constructing a mask coefficient branch (2-5).
3) Constructing prototype mask branches
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7N in3Inputting convolutional layer (3-1) and convolutional layer 3-2), performing convolution operation with convolution kernel size of 3 × 3 and step size of 1, and using the mostNeighbor up-sampling method for convolving N 31/4 of the original picture size is sampled, the feature diagram of (32,138,138) is obtained through 2 times of repeated convolution layers (3-3) and (3-4), the first dimension and the third dimension of the feature diagram of (32,138,138) are subjected to position exchange, and a prototype mask P of (138, 138 and 32) is obtained to construct a prototype mask branch (3);
(3) training instance segmentation network
(a) Determining a loss function
The Loss function Loss comprises category Loss, prediction box regression Loss and mask Loss, and is determined by the following formula:
Loss＝αLcls+βLbox+θLmask
in the formula, alpha, beta and theta are different weights of three losses, and are respectively 1, 1.5 and 6.125.
(b) Training instance segmentation network
Inputting a training set, a corresponding label file and a pre-training model on an ImageNet network into an example segmentation network for training, setting the learning rate gamma of the example segmentation network to be 0.0001 in the training process, setting an optimizer to be an adaptive moment estimation optimizer, iterating for M times, using the number of pictures for each iteration to be B, wherein M and B are finite positive integers, and training until a Loss function Loss of the example segmentation network converges.
(4) Saving weight files
And storing the weight file for 1 time in F times of iteration, wherein F is less than or equal to M, and obtaining the storage weight file.
(5) Validating an instance split network
And inputting the verification set and the corresponding label file into an example segmentation network for verification, wherein the network performs verification once every F times.
(6) Test case split network
Inputting the storage weight file, the test data set and the corresponding label file into an example segmentation network for testing, and obtaining an example mask Q according to the following formula:
Q＝σ(PWT)
and in the formula, P is a prototype mask, W is a mask coefficient with the size of N multiplied by K reserved after non-maximum value inhibition and score threshold processing, N is the number of the reserved mask coefficients, T is the transposition of a matrix, sigma is a sigmoid nonlinear activation function, and the example mask Q is cut and thresholded by 0.5 by using the positioning result of (C, S) obtained in the step (2) of constructing the predictive head branch, so that example segmentation of the cell is obtained.
In the step (3) of segmenting the network according to the training example of the present invention, the number of iterations M is preferably 200000, and B is preferably 8.
In the step (4) of saving the weight file, F is preferably 10000.
The invention divides the picture data set into a training set, a verification set and a test set, and adopts an example division network, wherein the example division network is formed by adding a path enhancement network on a characteristic pyramid network, the training set trains in the example division network, the verification set verifies in the training process, the test set tests by using a weight file stored in the training process, performs example division on cells in the picture, fully utilizes the shallow information in the picture, performs example division on the cells in the picture, and solves the problems of low precision and low speed of example division of dense pictures. Compared with the prior art, the method has the advantages of simplicity, high positioning speed, high positioning precision and the like, and can be used for example segmentation of cells in a dense picture.
Drawings
FIG. 1 is a flowchart of example 1 of the present invention.
Fig. 2 is a schematic structural diagram of a split network according to an example of embodiment 1 of the present invention.
Fig. 3 is a picture of a medical pathology in a PanNuKe dataset.
Fig. 4 is a graph of the results of the segmentation of fig. 3 using the method of example 1.
FIG. 5 is a graph of the results of a comparative experiment of the method of example 1 of the present invention with a Yolact network.
Detailed Description
The present invention will be described in further detail below with reference to the drawings and examples, but the present invention is not limited to the following embodiments.
Example 1
Taking 8160 isolated medical pathological pictures from PanNuke data set as an example, the cell example segmentation method based on bottom-up path enhancement in the embodiment comprises the following steps (see fig. 1).
(1) Data pre-processing
8160 isolated medical pathology original pictures are taken from the PanNuke data set and divided into a verification set, a test set and a training set, wherein the number ratio of the verification set to the test set to the training set is 1: 1: and 4, the pictures in the training set, the verification set and the test set are 550 multiplied by 550 pixels in size, each cell in the training set, the verification set and the test set is extracted to generate a new mask picture containing a single cell, the mask picture is 550 multiplied by 550 pixels in size, and the pictures in the training set, the verification set and the test set and the mask pictures of the corresponding data sets are converted into tag files by a pycocools tool.
(2) Building instance split networks
In fig. 2, the example segmented network of this embodiment is formed by connecting a backbone branch (1), a prediction header branch (2), and a prototype mask branch (3), wherein an output of the backbone branch (1) is connected to inputs of the prediction header branch (2) and the prototype mask branch (3), and the backbone branch (1), the prediction header branch (2), and the prototype mask branch (3) are constructed by the following methods:
1) building backbone network branches
a characteristic pyramid network extraction characteristic diagram
And inputting each picture in the training set into a basic feature extraction network (1-1) of the feature pyramid network to extract a feature map, so as to obtain a basic feature map set { C1, C2, C3, C4 and C5 }.
b fusion of feature maps
Inputting the basic feature map set { C1, C2, C3, C4, C5} into a feature pyramid network (1-2), and performing feature fusion to obtain a fused feature map set { P }3，P4，P5，P6，P7And (4) upsampling by using a nearest neighbor upsampling method to obtain a feature map set of 256 channels, and sequentially connecting 5 convolution kernels with the size of 3 multiplied by 3 and the step length of 3 multiplied by 3 to the feature map output by the feature pyramid network2, constituting a path enhancing network (1-3), and obtaining a set of path enhancing network feature maps { N } by the following formula3，N4，N5，N6，N7Constructing a backbone network branch (1):
wherein R is a rule activation function, C3For a convolutional layer with a convolutional kernel size of 3 x 3 and a step size of 2, U is the matrix addition, i ∈ {3, 4,5, 6, 7 }.
2) Building a probing tip branch
The prediction head branch comprises a positioning branch, a classification branch and a mask coefficient branch.
a constructing a positioning branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, the anchor characteristic diagram is input into a convolution layer (2-1) and a convolution layer (2-2), and the convolution operation with convolution kernel size of 3 multiplied by 3 and step length of 1 is carried out for 2 times to obtain (A multiplied by S, L)j，Lj) Is (A x S, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjAxS), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by S) to obtain (L)j×LjX A, S), wherein A is 3 anchors set for each pixel point, and S is coordinates of a center point and an upper left corner point of each anchor; will obtain (L)j×LjX a, S) a first dimension of the prediction results is column-stitched to obtain a positioning result of (C, S), where C is determined by:
in the formula LjE {69,35,18,9,5}, j e {1,2,3,4,5}, a localization branch (2-3) is constructed.
b constructing a classification branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, the anchor characteristic diagram is input into a convolution layer (2-1) and a convolution layer (2-2), and (A multiplied by D, L) is obtained through convolution operation with convolution kernel size of 3 multiplied by 3 and step length of 1j，Lj) Is (A x D, L)j，Lj) Exchanging the first dimension and the third dimension of the feature map to obtain (L)j，LjA × D), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by D) to obtain (L)j×LjX A, D) where D represents the number of cell classes ex vivo in the training set, and (L) obtainedj×LjThe first dimension of the prediction results of the XA and D) is subjected to column splicing operation to obtain classification results of the (C and D) to construct a classification branch (2-4); in this embodiment, D is 5, which is specifically divided into: tumor Cells (Neoplastic Cells), Inflammatory Cells (inflammation), Soft tissue Cells (Soft tissue Cells), Dead Cells (Dead Cells), Epithelial Cells (Epithelial).
c constructing a mask coefficient branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 form an anchor characteristic diagram, which is input into the convolution layer (2-1) and convolution layer (2-2) and is processed by convolution operation with convolution kernel size of 3 x 3 and step size of 1 to obtain (A x K, L)j，Lj) Is (A x K, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjA × K), and then (L)j，LjA x K) in the first dimension, the second dimension and the third dimension of the feature map to obtain (L)j×LjX a, K), where K is the number of mask coefficients generated, and K in this embodiment is 32. Will obtain (L)j×LjAnd x A, K) performing column splicing operation on the first dimension of the prediction result to obtain mask coefficients of (C, K), and constructing a mask coefficient branch (2-5).
3) Constructing prototype mask branches
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7N in3Inputting convolutional layer (3-1) and convolutional layer 3-2), performing convolution operation with convolution kernel size of 3 × 3 and step length of 1, and performing nearest neighbor upsampling on convolved N 31/4 which is up-sampled to the original picture size, and the feature map of (32,138,138) is obtained through 2 times of repeated convolution layers (3-3) and (3-4), the first dimension and the third dimension of the feature map of (32,138,138) are subjected to position exchange to obtain the prototype mask P of (138, 138 and 32), and the prototype mask branch (3) is constructed.
The invention adopts the example segmentation network, is formed by adding the path enhancement network on the characteristic pyramid network, fully utilizes the shallow information in the picture to segment the examples of the cells in the picture, and solves the problems of low precision and low speed of the example segmentation of the dense picture.
(3) Training instance segmentation network
(a) Determining a loss function
The Loss function Loss comprises category Loss, prediction box regression Loss and mask Loss, and is determined by the following formula:
Loss＝αLcls+βLbox+θLmask
in the formula, alpha, beta and theta are different weights of three losses, and are respectively 1, 1.5 and 6.125.
(b) Training instance segmentation network
Inputting a training set, a corresponding label file and a pre-training model on an ImageNet network into an example segmentation network for training, setting the learning rate gamma of the example segmentation network to be 0.0001 in the training process, setting an optimizer to be an adaptive moment estimation optimizer, iterating for M times, using the number of pictures for each iteration to be B, wherein M and B are finite positive integers, and training until a Loss function Loss of the example segmentation network converges.
(4) Saving weight files
Saving the weight file for 1 time in each iteration for F times, wherein F is less than or equal to M, and obtaining a saving weight file; in this example, F is 10000.
(5) Validating an instance split network
And inputting the verification set and the corresponding label file into an example segmentation network for verification, wherein the network performs verification once every F times.
(6) Test case split network
Inputting the storage weight file, the test data set and the corresponding label file into an example segmentation network for testing, and obtaining an example mask Q according to the following formula:
Q＝σ(PWT)
in the formula, P is a prototype mask, W is a mask coefficient with the size of N multiplied by K reserved after non-maximum value suppression and score threshold processing, N is the number of the reserved mask coefficients, T is the transposition of a matrix, sigma is a sigmoid nonlinear activation function, and the example mask Q is cut and thresholded with the threshold value of 0.5 by using the positioning result of (C, S) obtained in the step (2) of constructing a prediction head branch. The segmentation of the cell example is completed, fig. 3, fig. 4 is a medical pathology picture in the PanNuKe data set, and fig. 4 is a graph of the example segmentation result of fig. 3. As can be seen from fig. 3 and 4, the mixed Cells in the in vitro pictures are subjected to example segmentation, and are segmented into five Cells, namely tumor Cells (Neoplastic Cells), Inflammatory Cells (Inflammatory Cells), Soft tissue Cells (Soft tissue Cells), Dead Cells (Dead Cells) and Epithelial Cells (Epithelial Cells).
The invention divides the picture data set into a training set, a verification set and a test set, and adopts an example division network, wherein the example division network is formed by adding a path enhancement network on a characteristic pyramid network, the training set trains in the example division network, the verification set verifies in the training process, the test set tests by storing a weight file in the training process, and the shallow information in the picture is fully utilized to carry out example division on cells in the picture, thereby solving the technical problems of low precision and low speed of example division of dense pictures. Compared with the prior art, the method has the advantages of simplicity, high positioning speed, high positioning precision and the like, and can be used for example segmentation of cells in a dense picture.
Example 2
Taking 8160 isolated medical pathological pictures from the PanNuke data set as an example, the cell example segmentation method based on bottom-up path enhancement in the embodiment comprises the following steps.
(1) Data pre-processing
This procedure is the same as in example 1.
(2) Building instance split networks
The example segmentation network is formed by connecting a backbone network branch (1), a prediction head branch (2) and a prototype mask branch (3), wherein the output of the backbone network branch (1) is connected with the inputs of the prediction head branch (2) and the prototype mask branch (3), and the backbone network branch (1), the prediction head branch (2) and the prototype mask branch (3) are constructed by the following method:
1) building backbone network branches
This procedure is the same as in example 1.
2) Building a probing tip branch
The prediction head branch comprises a positioning branch, a classification branch and a mask coefficient branch.
a constructing a positioning branch
This procedure is the same as in example 1.
b constructing a classification branch
This procedure is the same as in example 1.
c constructing a mask coefficient branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 form an anchor characteristic diagram, which is input into the convolution layer (2-1) and convolution layer (2-2) and processed by convolution kernelA convolution operation of 3X 3 with a step size of 1 yields (A X K, L)j，Lj) Is (A x K, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjA × K), and then (L)j，LjA x K) in the first dimension, the second dimension and the third dimension of the feature map to obtain (L)j×LjX a, K), where K is the number of mask coefficients generated, and K in this embodiment is 64. Will obtain (L)j×LjAnd x A, K) performing column splicing operation on the first dimension of the prediction result to obtain mask coefficients of (C, K), and constructing a mask coefficient branch (2-5).
3) Constructing prototype mask branches
This procedure is the same as in example 1.
Other steps are the same as in example 1, and the cell instance segmentation is completed.
In order to verify the beneficial effects of the invention, the inventor uses the method of embodiment 1 of the invention to perform a comparison experiment with yolact network, tests the model precision by using a given evaluation code on the same test set of PanNuke data set by using a trained model, uses an average accuracy rate mAP of a non-maximum value for inhibiting different intersection comparison thresholds as an evaluation index, and the experimental result is shown in fig. 5. in fig. 5, the abscissa represents the target positioning mAP segmented by an example and the mAP evaluation index of an example mask, and the ordinate represents the average accuracy rate mAP value of the different intersection comparison thresholds. As can be seen from fig. 5, the target mAP and the example mask mAP of yolact network are 32.77% and 32.83%, respectively, and the target mAP and the example mask mAP of the present invention are 34.03% and 34.05%, respectively.
Claims (3)
1. A cell instance segmentation method based on bottom-up path enhancement is characterized by comprising the following steps;
(1) data pre-processing
Taking an original picture from the PanNuke data set and dividing the original picture into a verification set, a test set and a training set, wherein the number ratio of the verification set to the test set to the training set is 1: 1: 4, the size of the pictures in the training set, the verification set and the test set is 550 multiplied by 550 pixels, each cell in the training set, the verification set and the test set picture is extracted to generate a new mask picture containing a single cell, the size of the mask picture is 550 multiplied by 550 pixels, and the pictures in the training set, the verification set and the test set and the mask pictures of the corresponding data sets are converted into tag files by a pycocools tool;
(2) building instance split networks
The example segmentation network is formed by connecting a backbone network branch (1), a prediction head branch (2) and a prototype mask branch (3), wherein the output of the backbone network branch (1) is connected with the inputs of the prediction head branch (2) and the prototype mask branch (3), and the backbone network branch (1), the prediction head branch (2) and the prototype mask branch (3) are constructed by the following method:
1) building backbone network branches
a characteristic pyramid network extraction characteristic diagram
Inputting each picture in the training set into a basic feature extraction network (1-1) of the feature pyramid network to extract a feature map, and obtaining a basic feature map set { C1, C2, C3, C4 and C5 };
b fusion of feature maps
Inputting the basic feature map set { C1, C2, C3, C4, C5} into a feature pyramid network (1-2), and performing feature fusion to obtain a fused feature map set { P }3，P4，P5，P6，P7And (5) performing upsampling by using a nearest neighbor upsampling method to obtain a feature map set of 256 channels, sequentially connecting 5 convolutional layers with the convolutional kernel size of 3 multiplied by 3 and the step length of 2 on the feature map output by the feature pyramid network to form a path enhancement network (1-3), and obtaining a path enhancement network feature map set { N (N) } according to the following formula3，N4，N5，N6，N7Constructing a backbone network branch (1);
wherein R is a rule activation function, C3Is a convolution layer with convolution kernel size of 3 × 3 and step length of 2, U isMatrix addition, i ∈ {3, 4,5, 6, 7 };
2) building a probing tip branch
The prediction head branch comprises a positioning branch, a classification branch and a mask coefficient branch;
a constructing a positioning branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, the anchor characteristic diagram is input into a convolution layer (2-1) and a convolution layer (2-2), and the convolution operation with convolution kernel size of 3 multiplied by 3 and step length of 1 is carried out for 2 times to obtain (A multiplied by S, L)j，Lj) Is (A x S, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjAxS), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by S) to obtain (L)j×LjX A, S), wherein A is 3 anchors set for each pixel point, and S is coordinates of a center point and an upper left corner point of each anchor; will obtain (L)j×LjX a, S) a first dimension of the prediction results is column-stitched to obtain a positioning result of (C, S), where C is determined by:
in the formula LjBelongs to {69,35,18,9,5}, j belongs to {1,2,3,4,5}, and a positioning branch (2-3) is constructed;
b constructing a classification branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 respectively form an anchor characteristic diagram, which is input into the convolution layer (2-1) and convolution layer (2-2) and has a convolution kernel size ofConvolution operation with step size of 1 by 3X 3 to obtain (A X D, L)j，Lj) Is (A x D, L)j，Lj) Exchanging the first dimension and the third dimension of the feature map to obtain (L)j，LjA × D), and then (L)j，LjA is multiplied in the first dimension, the second dimension and the third dimension of the characteristic diagram of A multiplied by D) to obtain (L)j×LjX A, D) where D represents the number of cell classes ex vivo in the training set, and (L) obtainedj×LjThe first dimension of the prediction results of the XA and D) is subjected to column splicing operation to obtain classification results of the (C and D) to construct a classification branch (2-4);
c constructing a mask coefficient branch
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7The area of each pixel point on each characteristic graph in the graph is [24,48,96,192,384]The anchors with length and width ratios of 1, 1/2 and 2 form an anchor characteristic diagram, which is input into the convolution layer (2-1) and convolution layer (2-2) and is processed by convolution operation with convolution kernel size of 3 x 3 and step size of 1 to obtain (A x K, L)j，Lj) Is (A x K, L)j，Lj) The first dimension and the third dimension of the feature map are exchanged to obtain (L)j，LjA × K), and then (L)j，LjA x K) in the first dimension, the second dimension and the third dimension of the feature map to obtain (L)j×LjX a, K), where K is the number of generated mask coefficients, K is 32 or 64; will obtain (L)j×LjThe first dimension of the prediction results of the XA and the K) is subjected to column splicing operation to obtain mask coefficients of the C and the K, and the mask coefficients are constructed into mask coefficient branches (2-5);
3) constructing prototype mask branches
Enhancing a Path with a set of network profiles N3，N4，N5，N6，N7N in3Inputting convolutional layer (3-1) and convolutional layer 3-2), performing convolution operation with convolution kernel size of 3 × 3 and step length of 1, and performing nearest neighbor upsamplingLast N31/4 of the original picture size is sampled, the feature diagram of (32,138,138) is obtained through 2 times of repeated convolution layers (3-3) and (3-4), the first dimension and the third dimension of the feature diagram of (32,138,138) are subjected to position exchange, and a prototype mask P of (138, 138 and 32) is obtained to construct a prototype mask branch (3);
(3) training instance segmentation network
(a) Determining a loss function
The Loss function Loss comprises category Loss, prediction box regression Loss and mask Loss, and is determined by the following formula:
Loss＝αLcls+βLbox+θLmask
in the formula, alpha, beta and theta are three different lost weights which are respectively 1, 1.5 and 6.125;
(b) training instance segmentation network
Inputting a training set, a corresponding label file and a pre-training model on an ImageNet network into an example segmentation network for training, setting the learning rate gamma of the example segmentation network to be 0.0001 in the training process, setting an optimizer to be an adaptive moment estimation optimizer, iterating for M times, using the number of pictures for each iteration to be B, wherein M and B are finite positive integers, and training until Loss function Loss of the example segmentation network converges;
(4) saving weight files
Saving the weight file for 1 time in each iteration for F times, wherein F is less than or equal to M, and obtaining a saving weight file;
(5) validating an instance split network
Inputting the verification set and the corresponding label file into an example segmentation network for verification, wherein the network performs verification once every F times;
(6) test case split network
Inputting the storage weight file, the test data set and the corresponding label file into an example segmentation network for testing, and obtaining an example mask Q according to the following formula:
Q＝σ(PWT)
and in the formula, P is a prototype mask, W is a mask coefficient with the size of N multiplied by K reserved after non-maximum value inhibition and score threshold processing, N is the number of the reserved mask coefficients, T is the transposition of a matrix, sigma is a sigmoid nonlinear activation function, and the example mask Q is cut and thresholded by 0.5 by using the positioning result of (C, S) obtained in the step (2) of constructing the predictive head branch, so that example segmentation of the cell is obtained.
2. The bottom-up pathway enhancement based cell instance segmentation method of claim 1, wherein: in the step (3) of training the example segmentation network, the number of iterations M is 200000, and B is 8.
3. The bottom-up pathway enhancement based cell instance segmentation method of claim 1, wherein: in the step (4) of saving the weight file, F is 10000.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011019366.5A CN112164077B (en) | 2020-09-25 | 2020-09-25 | Cell instance segmentation method based on bottom-up path enhancement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011019366.5A CN112164077B (en) | 2020-09-25 | 2020-09-25 | Cell instance segmentation method based on bottom-up path enhancement |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112164077A true CN112164077A (en) | 2021-01-01 |
| CN112164077B CN112164077B (en) | 2023-12-29 |
Family
ID=73862959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011019366.5A Active CN112164077B (en) | 2020-09-25 | 2020-09-25 | Cell instance segmentation method based on bottom-up path enhancement |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112164077B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112484A (en) * | 2021-04-19 | 2021-07-13 | 山东省人工智能研究院 | Ventricular image segmentation method based on feature compression and noise suppression |
| CN113569860A (en) * | 2021-07-29 | 2021-10-29 | 北京地平线信息技术有限公司 | Example segmentation method, training method of example segmentation network and device thereof |
| CN114639102A (en) * | 2022-05-11 | 2022-06-17 | 珠海横琴圣澳云智科技有限公司 | Cell segmentation method and device based on key point and size regression |
| CN116681892A (en) * | 2023-06-02 | 2023-09-01 | 山东省人工智能研究院 | Image precise segmentation method based on multi-center polar mask model improvement |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007076893A1 (en) * | 2005-12-30 | 2007-07-12 | Telecom Italia S.P.A. | Edge-guided morphological closing in segmentation of video sequences |
| CN101371273A (en) * | 2005-12-30 | 2009-02-18 | 意大利电信股份公司 | Video sequence partition |
| US10140544B1 (en) * | 2018-04-02 | 2018-11-27 | 12 Sigma Technologies | Enhanced convolutional neural network for image segmentation |
| CN109584248A (en) * | 2018-11-20 | 2019-04-05 | 西安电子科技大学 | Infrared surface object instance dividing method based on Fusion Features and dense connection network |
| CN109711413A (en) * | 2018-12-30 | 2019-05-03 | 陕西师范大学 | Image, semantic dividing method based on deep learning |
| CN110378345A (en) * | 2019-06-04 | 2019-10-25 | 广东工业大学 | Dynamic scene SLAM method based on YOLACT example parted pattern |
| US10467500B1 (en) * | 2018-12-31 | 2019-11-05 | Didi Research America, Llc | Method and system for semantic segmentation involving multi-task convolutional neural network |
| CN110619632A (en) * | 2019-09-18 | 2019-12-27 | 华南农业大学 | Mango example confrontation segmentation method based on Mask R-CNN |
| CN111091095A (en) * | 2019-12-19 | 2020-05-01 | 华中科技大学 | Method for detecting ship target in remote sensing image |
| US10713794B1 (en) * | 2017-03-16 | 2020-07-14 | Facebook, Inc. | Method and system for using machine-learning for object instance segmentation |
| CN111461114A (en) * | 2020-03-03 | 2020-07-28 | 华南理工大学 | Multi-scale feature pyramid text detection method based on segmentation |
| CN111552300A (en) * | 2020-06-09 | 2020-08-18 | 南开大学 | Crop picking system based on instance segmentation and path planning |
| CN111553414A (en) * | 2020-04-27 | 2020-08-18 | 东华大学 | In-vehicle lost object detection method based on improved Faster R-CNN |
| CN111597920A (en) * | 2020-04-27 | 2020-08-28 | 东南大学 | Full convolution single-stage human body example segmentation method in natural scene |
-
2020
- 2020-09-25 CN CN202011019366.5A patent/CN112164077B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101371273A (en) * | 2005-12-30 | 2009-02-18 | 意大利电信股份公司 | Video sequence partition |
| WO2007076893A1 (en) * | 2005-12-30 | 2007-07-12 | Telecom Italia S.P.A. | Edge-guided morphological closing in segmentation of video sequences |
| US10713794B1 (en) * | 2017-03-16 | 2020-07-14 | Facebook, Inc. | Method and system for using machine-learning for object instance segmentation |
| US10140544B1 (en) * | 2018-04-02 | 2018-11-27 | 12 Sigma Technologies | Enhanced convolutional neural network for image segmentation |
| CN109584248A (en) * | 2018-11-20 | 2019-04-05 | 西安电子科技大学 | Infrared surface object instance dividing method based on Fusion Features and dense connection network |
| CN109711413A (en) * | 2018-12-30 | 2019-05-03 | 陕西师范大学 | Image, semantic dividing method based on deep learning |
| US10467500B1 (en) * | 2018-12-31 | 2019-11-05 | Didi Research America, Llc | Method and system for semantic segmentation involving multi-task convolutional neural network |
| CN110378345A (en) * | 2019-06-04 | 2019-10-25 | 广东工业大学 | Dynamic scene SLAM method based on YOLACT example parted pattern |
| CN110619632A (en) * | 2019-09-18 | 2019-12-27 | 华南农业大学 | Mango example confrontation segmentation method based on Mask R-CNN |
| CN111091095A (en) * | 2019-12-19 | 2020-05-01 | 华中科技大学 | Method for detecting ship target in remote sensing image |
| CN111461114A (en) * | 2020-03-03 | 2020-07-28 | 华南理工大学 | Multi-scale feature pyramid text detection method based on segmentation |
| CN111553414A (en) * | 2020-04-27 | 2020-08-18 | 东华大学 | In-vehicle lost object detection method based on improved Faster R-CNN |
| CN111597920A (en) * | 2020-04-27 | 2020-08-28 | 东南大学 | Full convolution single-stage human body example segmentation method in natural scene |
| CN111552300A (en) * | 2020-06-09 | 2020-08-18 | 南开大学 | Crop picking system based on instance segmentation and path planning |
Non-Patent Citations (5)
| Title |
|---|
| DANIEL BOLYA 等: "YOLACT: Real-Time Instance Segmentation", 《2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)》 * |
| YOUNGWAN LEE 等: "CenterMask: Real-Time Anchor-Free Instance Segmentation", 《2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》 * |
| 党倩 等: "基于二级改进LeNet-5的交通标志识别算法", 《陕西师范大学学报(自然科学版)》 * |
| 陈栋 等: "卷积神经网络在炮兵对抗训练系统中的应用", 《兵工自动化 》 * |
| 高烨 等: "基于猫群优化算法的图像多阈值分割方法", 《中国体视学与图像分析》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112484A (en) * | 2021-04-19 | 2021-07-13 | 山东省人工智能研究院 | Ventricular image segmentation method based on feature compression and noise suppression |
| CN113112484B (en) * | 2021-04-19 | 2021-12-31 | 山东省人工智能研究院 | Ventricular image segmentation method based on feature compression and noise suppression |
| CN113569860A (en) * | 2021-07-29 | 2021-10-29 | 北京地平线信息技术有限公司 | Example segmentation method, training method of example segmentation network and device thereof |
| CN113569860B (en) * | 2021-07-29 | 2024-02-27 | 北京地平线信息技术有限公司 | Instance segmentation method, training method of instance segmentation network and device thereof |
| CN114639102A (en) * | 2022-05-11 | 2022-06-17 | 珠海横琴圣澳云智科技有限公司 | Cell segmentation method and device based on key point and size regression |
| CN116681892A (en) * | 2023-06-02 | 2023-09-01 | 山东省人工智能研究院 | Image precise segmentation method based on multi-center polar mask model improvement |
| CN116681892B (en) * | 2023-06-02 | 2024-01-26 | 山东省人工智能研究院 | Image precise segmentation method based on multi-center polar mask model improvement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112164077B (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11373305B2 (en) | Image processing method and device, computer apparatus, and storage medium | |
| CN109241982B (en) | Target detection method based on deep and shallow layer convolutional neural network | |
| CN112164077B (en) | Cell instance segmentation method based on bottom-up path enhancement | |
| CN112884064B (en) | Target detection and identification method based on neural network | |
| CN110348384B (en) | Small target vehicle attribute identification method based on feature fusion | |
| CN111612008A (en) | Image segmentation method based on convolution network | |
| CN111489357A (en) | Image segmentation method, device, equipment and storage medium | |
| Wang et al. | FE-YOLOv5: Feature enhancement network based on YOLOv5 for small object detection | |
| CN112750129A (en) | Image semantic segmentation model based on feature enhancement position attention mechanism | |
| CN110555461A (en) | scene classification method and system based on multi-structure convolutional neural network feature fusion | |
| CN109492610A (en) | A kind of pedestrian recognition methods, device and readable storage medium storing program for executing again | |
| CN115063833A (en) | Machine room personnel detection method based on image layered vision | |
| CN114972753A (en) | Lightweight semantic segmentation method and system based on context information aggregation and assisted learning | |
| CN114782355A (en) | Gastric cancer digital pathological section detection method based on improved VGG16 network | |
| Liu et al. | Student behavior recognition from heterogeneous view perception in class based on 3-D multiscale residual dense network for the analysis of case teaching | |
| CN110633706A (en) | Semantic segmentation method based on pyramid network | |
| CN114092467A (en) | Scratch detection method and system based on lightweight convolutional neural network | |
| CN113111740A (en) | Characteristic weaving method for remote sensing image target detection | |
| CN113033371A (en) | CSP model-based multi-level feature fusion pedestrian detection method | |
| CN117152438A (en) | Lightweight street view image semantic segmentation method based on improved deep LabV3+ network | |
| CN116912595A (en) | Cross-domain multi-mode remote sensing image classification method based on contrast learning | |
| CN116778164A (en) | Semantic segmentation method for improving deep V < 3+ > network based on multi-scale structure | |
| TWI809957B (en) | Object detection method and electronic apparatus | |
| CN116758477A (en) | Kitchen personnel dressing detection method based on improved YOLOv7 model | |
| CN116596966A (en) | Segmentation and tracking method based on attention and feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |