CN115082420A - Example segmentation method of histopathological cell nucleus based on deep learning - Google Patents

Abstract

The invention discloses a method for segmenting an example of a histopathology cell nucleus based on deep learning, and relates to the problem of accurate segmentation of the example of the histopathology cell nucleus in histopathology image analysis. Example segmentation of histopathological nuclei not only distinguishes nuclei from the background of the picture, but also precisely segments each individual nucleus contour. This poses a challenge for accurate segmentation of histopathological nuclear examples due to the complex image background, lack of sharp boundaries between nuclei, and large nuclear size and morphological changes. The invention provides a histopathology cell nucleus example segmentation method based on a CondInst model. Experiments show that the method effectively solves the problem of missing segmentation or mistaken segmentation of the histopathology cell nucleus, and the outline of the cell nucleus is more fit with the outline of the real cell nucleus. The invention is applied to the precise segmentation of the histopathological nucleus example.

Description

Example segmentation method of histopathological cell nucleus based on deep learning

Technical Field

The invention relates to example segmentation of histopathological nuclei.

Background

Histopathological images can provide a great deal of useful information for the classification and classification of diagnosed cancer, and therefore histopathological image analysis is particularly important in the process of cancer diagnosis. Since the nucleus is the most obvious structure in the tissue, abnormal changes in the nucleus can convey a lot of information about the disease. The traditional diagnostic method is for pathologists to manually examine and analyze the diseased cell nuclei under a microscope. Such manual diagnosis is a tedious and time consuming task, and specificity and sensitivity can be influenced by the experience of the pathologist. The problem can be effectively solved by computer-aided image analysis, and accurate segmentation of cell nuclei is a key step of computer-aided histopathology image analysis. Nuclear segmentation is considered a prerequisite for determining cell phenotype, nuclear morphometry, cell classification, and provides strong support for cancer staging and prognosis.

Example segmentation of histopathological nuclei not only distinguishes nuclei from the background of the picture, but also precisely segments each individual nucleus contour. Due to the fact that noise exists in images, the types of cell nuclei are various, and the sizes and the shapes of the cell nuclei are different, difficulty is caused in feature extraction, missing segmentation or mistaken segmentation of the cell nuclei occurs, clear boundaries between the cell nuclei are not provided, and the challenges are brought to accurate segmentation of histopathological cell nucleus examples. The scheme of the histopathological cell nucleus example segmentation method based on the CondInst (conditional volumes for Instance segmentation) model can effectively solve the problem. The network consists of an effective feature extraction module, a target detection head, a self-adaptive receptive field segmentation branch and a conditional convolution segmentation branch. The effective feature extraction module is responsible for extracting image features and fusing effective information of a multi-scale feature map so as to solve the problems of missing segmentation and error segmentation, then sending the feature map obtained by the backbone network into a detection head of a single-stage FCOS (fuzzy consistent dependent-state object detector) network without an anchor frame to obtain parameters required by a target frame and a conditional convolution, wherein the conditional convolution branch is an original segmentation branch of CondInst, the self-adaptive receptive field segmentation branch cuts out an example image with a corresponding size at a corresponding position of an original image according to the result of the detection frame, more accurate cell contour information is extracted through deformable convolution, the segmentation results of the two branches are subjected to consistency loss calculation so as to enhance the segmentation result of the conditional convolution, and finally the segmentation result of the conditional convolution branch is obtained.

Disclosure of Invention

The invention aims to solve the problem of accurate segmentation of a histopathological nucleus example, and provides a histopathological nucleus example segmentation method based on a CondInst model.

The above object of the invention is mainly achieved by the following technical scheme:

s1, performing data enhancement on the data set;

and performing data enhancement on the data set through operations of cutting, overlapping and overturning.

S2, sending the picture obtained in the step S1 into an effective feature extraction module to obtain a multi-scale feature map;

as shown in fig. 2, the effective feature extraction module is composed of a Global Average Pooling module (GAP) and a Gate module, the GAP module is responsible for extracting Global semantic features of a high-dimensional feature map and discarding location features when multi-scale information is fused, and the Gate module filters low-dimensional features to obtain high-quality features.

GAP Module As shown in FIG. 3, a two-dimensional matrix is pooled by global averagingCompressed into a real number, the real number has global semantic features, and for the feature map

，

Is the number of the channels,

and

the feature map length and width, respectively, are obtained:

(1)

then pass through

The convolution changes the channels of the feature map into

，

Is the original number of channels of the feature map,

to be over-parametric, and finally passed

Convolution restores the channel number so as to fuse the information of different channels; adding the features obtained by the GAP module into the bottom layer feature graph pixel by pixel:

(2)

wherein

In order to obtain a feature map by the GAP module,

the characteristic diagram is an original characteristic diagram,

as a result of feature fusion;

the Gate module can control the passing of effective features in the low-dimensional feature map; through one

The depth of the feature map can be separated from the convolution and sigmoid functions to obtain a new feature map, and the new feature map is multiplied by the original feature map to obtain a feature map fused with low-dimensional high-quality features.

S3, sending the multi-scale characteristic diagram obtained in the S2 into a target detection head to obtain a target frame and parameters required by conditional convolution;

and (4) sending the multi-scale feature map obtained in the step (S2) into a detection head of a target detection network FCOS, learning parameters and predicted target frame information (classification, regression, centrality and the like) required by conditional convolution pixel by pixel, and screening to obtain a predicted frame of a positive sample and parameters required by conditional convolution. The parameters obtained here correspond one-to-one to the prediction boxes.

S4, sending the feature map with the highest resolution in the multi-scale feature map obtained in the S2 into a conditional convolution segmentation branch to obtain a cell nucleus example segmentation result;

sending the feature map with the highest resolution in the multi-scale feature map obtained in step S2 to a conditional convolution segmentation branch of CondInst, combining the feature map with a coordinate map, where the coordinate map is a relative coordinate from a position on the feature map to a position (x, y) (i.e., a position where a convolution kernel of a mask head is generated), and then sending the combined feature map to the mask head for conditional convolution operation, where parameters required by the mask head are parameters obtained in step S3, the parameters are in one-to-one correspondence with prediction boxes, and the feature map is convolved with the parameters corresponding to each prediction box to obtain an example matching the prediction boxes. The mask is a very compact FCN structure with three 1 x 1 convolutions, each convolution having 8 channels, and using ReLU as the activation function (the last one excluded).

S5, cutting out a picture with a corresponding size at the corresponding position of the picture obtained in S1 by using the target frame obtained in S3, and sending the cut picture into a self-adaptive receptive field segmentation branch to obtain a cell nucleus example segmentation result; and according to the position and the size of the target frame obtained in the step S3, cutting the corresponding area in the picture obtained in the step S1, and adjusting the picture after cutting to be a uniform size through RoIAlign. The small pictures only containing a single cell nucleus example are subjected to convolution operation, and each small picture contains a single example, so that the small pictures are not influenced by other cell nuclei during segmentation, and the segmented cell nucleus outline is closer to the real cell nucleus outline. Since the cell nucleus is mostly elliptical and the sizes and shapes of different examples are different, it is preferable to adaptively determine the size of the receptive field, so that the deformable convolution is selected here, and a direction offset is added to each element in the convolution kernel, so that the convolution kernel can be changed into any shape.

S6, calculating consistency loss by the segmentation result obtained in the S4 and the segmentation result obtained in the S5, and reinforcing the segmentation result of the S4;

the consistency loss calculation uses focal loss in the specific form:

(3)

order to

(4)

Wherein the content of the first and second substances,

in order to be able to adjust the factor,

prediction phase representing two segmentation resultsIn the same way, the first and second electrodes are connected,

the same probability is predicted for both segmentation results,

the two segmentation results representing the model have the same predicted value of probability,

the larger the two segmentation results, the closer the segmented nuclear contour of the conditional convolution is to the true contour.

Effects of the invention

The invention provides a histopathology cell nucleus example segmentation method based on a CondInst model, which can effectively solve the problem. The network can realize accurate segmentation of cell nucleus examples, wherein the effective feature extraction module is responsible for extracting picture features and fusing effective information of multi-scale feature maps, so that the conditions of missing segmentation and error segmentation are solved, the adaptive receptive field segmentation branch can extract more accurate cell contour information through deformable convolution, and the segmentation result of the adaptive receptive field segmentation branch and the original conditional convolution segmentation branch result of CondInst are subjected to consistency loss calculation, so that the segmentation result of conditional convolution is enhanced.

Drawings

FIG. 1 is a network structure of a histological cell nucleus example segmentation method based on a CondInst model;

FIG. 2 is a diagram of a backbone network architecture;

FIG. 3 GAP module;

FIG. 4 is a Gate module;

detailed description of the invention

The first embodiment is as follows:

in order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, the segmentation method for the example of the pathological cell nucleus provided herein performs the following steps:

s1, performing data enhancement on the data set;

s2, sending the picture obtained in the S1 into an effective feature extraction module to obtain a multi-scale feature map;

s3, sending the multi-scale feature map obtained in the S2 into a target detection head to obtain parameters required by the target frame and the conditional convolution;

s4, sending the feature map with the highest resolution in the multi-scale feature map obtained in the S2 into a conditional convolution segmentation branch to obtain a cell nucleus example segmentation result;

s5, cutting out a picture with a corresponding size at the corresponding position of the picture obtained in S1 by using the target frame obtained in S3, and sending the cut picture into a self-adaptive receptive field segmentation branch to obtain a cell nucleus example segmentation result;

s6, the consistency loss is calculated by using the segmentation result obtained in S4 and the segmentation result obtained in S5, and the segmentation result in S4 is strengthened.

The embodiment of the invention comprises an effective characteristic extraction module, a target detection head, a self-adaptive receptive field segmentation branch and a conditional convolution segmentation branch. The effective characteristic extraction module is responsible for extracting picture characteristics and fusing effective information of a multi-scale characteristic diagram so as to solve the problems of missing segmentation and error segmentation, then the characteristic diagram obtained by the backbone network is sent into a detection head of a single-stage FCOS network without an anchor frame to obtain parameters required by a target frame and a conditional convolution, the conditional convolution branch is an original CondInst segmentation branch, the self-adaptive receptive field segmentation branch cuts out example pictures with corresponding sizes at corresponding positions of an original image according to the result of the detection frame, more accurate cell contour information is extracted through deformable convolution, the segmentation results of the two branches are subjected to consistency loss calculation so as to enhance the segmentation result of the conditional convolution, and finally the segmentation result of the conditional convolution branch is obtained.

The following examples illustrate the invention in detail:

the model training as shown in fig. 1 comprises the steps of:

s1, performing data enhancement on the data set;

MoNuSeg dataset is resolution of

The multi-organ H & E staining image comprises 30 training sets and 14 testing sets, and the picture of the training sets is cut into

Size, data enhancement by adopting horizontal turning, vertical turning and horizontal and vertical turning, image normalization, and cutting test set pictures into

Size, and image normalization.

S2, sending the picture obtained in the step S1 into an effective feature extraction module to obtain a multi-scale feature map;

the pictures processed in S1 are sent to a ResNet (Deep residual network) network to obtain feature maps of different sizes, and sent to an effective feature extraction module shown in fig. 2, where the effective feature extraction module is composed of a Global Average Pooling module (GAP) and a Gate module, the GAP module is responsible for extracting Global semantic features of a high-dimensional feature map when multi-scale information is fused, discarding location features, and the Gate module filters low-dimensional features to obtain high-quality features.

The GAP module compresses a two-dimensional matrix into a real number through global average pooling as shown in fig. 3, the real number has global semantic features, and for the feature map, the real number is

，

Is the number of the channels,

and

the feature map length and width, respectively, are obtained:

(1)

then pass through

The convolution changes the channels of the feature map into

，

Is the original number of channels of the feature map,

to be over-parametric, and finally passed

Convolution restores the channel number so as to fuse the information of different channels; adding the features obtained by the GAP module into the bottom layer feature graph pixel by pixel:

(2)

wherein

Is a feature map obtained by the GAP module,

the characteristic diagram is an original characteristic diagram,

is characterized by meltingThe result after synthesis;

the Gate module can control the passing of effective features in the low-dimensional feature map; through one

The depth of the feature map can be separated from the convolution and sigmoid functions to obtain a new feature map, and the new feature map is multiplied by the original feature map to obtain a feature map fused with low-dimensional high-quality features. The sizes of the multi-scale feature maps obtained by the valid feature extraction module are S1, respectively, to obtain 1/8, 1/16, 1/32, 1/64, 1/128 of the pictures.

S3, sending the multi-scale feature map obtained in the S2 into a target detection head to obtain parameters required by the target frame and the conditional convolution;

and (4) sending the multi-scale feature map obtained in the step (S2) into a detection head of a target detection network FCOS, learning parameters and predicted target frame information (classification, regression, centrality and the like) required by conditional convolution pixel by pixel, and screening to obtain a predicted frame of a positive sample and parameters required by conditional convolution. The parameters obtained here correspond one-to-one to the prediction boxes.

S4, sending the characteristic diagram with the highest resolution in the multi-scale characteristic diagram obtained in the S2 into a conditional convolution segmentation branch to obtain a cell nucleus instance segmentation result;

sending the feature map with the highest resolution in the multi-scale feature map obtained in the step S2 to a conditional convolution segmentation branch of the CondInst, combining the feature map with a coordinate map, wherein the coordinate map is a relative coordinate from a position on the feature map to a position (x, y) (i.e., a position where a convolution kernel of a mask head is generated), and then sending the combined feature map to the mask head for conditional convolution operation, wherein parameters required by the mask head are parameters obtained in the step S3, the parameters are in one-to-one correspondence with prediction boxes, and the feature map is convolved with the parameters corresponding to each prediction box to obtain an example matched with the prediction boxes. The mask is a very compact FCN structure with three 1 x 1 convolutions, each convolution having 8 channels, and using ReLU as the activation function (the last one excluded).

S5, cutting out a picture with a corresponding size at the corresponding position of the picture obtained in S1 by using the target frame obtained in S3, and sending the cut picture into a self-adaptive receptive field segmentation branch to obtain a cell nucleus example segmentation result;

cropping the corresponding region in the picture obtained in S1 according to the position and size of the target frame obtained in S3, and adjusting the picture after cropping to a uniform size by RoIAlign (

). These small pictures containing only a single instance of the nucleus were subjected to four deformable convolution operations (convolution kernel size 3 × 3, number of channels 16, 8, 1, respectively).

S6, calculating consistency loss by the segmentation result obtained in the S4 and the segmentation result obtained in the S5, and reinforcing the segmentation result of the S4;

clipping the corresponding region in the segmentation result obtained in S4 according to the position and size of the target frame obtained in S3, and adjusting the picture after clipping to a uniform size by RoIAlign: (

) And the segmentation result obtained from S5 is subjected to consistency loss calculation through local loss to strengthen the segmentation result of the conditional convolution branch, so that the contour of the predicted cell nucleus is more fit with the real contour.

The focal loss is in a specific form:

(3)

order to

(4)

Wherein the content of the first and second substances,

in order to be able to adjust the factor,

it is represented that the two segmentation results are predicted to be the same,

the same probability is predicted for both segmentation results,

the two segmentation results representing the model have the same predicted value of probability,

the larger the two segmentation results, the closer the two segmentation results are, and the closer the segmented nuclear contour of the conditional convolution is to the real contour. And finally, outputting the segmentation result of the conditional convolution branch.

The present invention is capable of other embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention.

Claims (4)

1. An example segmentation method of histopathological cell nucleus based on deep learning is characterized by comprising the following steps:

s1, performing data enhancement on the data set;

s2, sending the picture obtained in the step S1 into an effective feature extraction module to obtain a multi-scale feature map;

s3, sending the multi-scale feature map obtained in the S2 into a target detection head to obtain parameters required by the target frame and the conditional convolution;

s4, sending the feature map with the highest resolution in the multi-scale feature map obtained in the S2 into a conditional convolution segmentation branch to obtain a cell nucleus example segmentation result;

s5, cutting out a picture with a corresponding size at the corresponding position of the picture obtained in S1 by using the target frame obtained in S3, and sending the cut picture into a self-adaptive receptive field segmentation branch to obtain a cell nucleus example segmentation result;

s6, the consistency loss is calculated by using the segmentation result obtained in S4 and the segmentation result obtained in S5, and the segmentation result in S4 is strengthened.

2. The method for example segmentation of histopathological nuclei based on deep learning as claimed in claim 1, wherein the effective feature extraction module in step S2 is as follows:

the effective feature extraction module consists of a Global Average Pooling module (GAP) and a Gate module, wherein the GAP module is responsible for extracting Global semantic features of a high-dimensional feature map during multi-scale information fusion, abandoning position features, and the Gate module filters low-dimensional features to obtain high-quality features;

in the GAP module, a two-dimensional matrix is compressed into a real number through global average pooling, the real number has global semantic features, and a feature graph is subjected to feature mapping

，

Is the number of the channels,

and

the feature map length and width, respectively, are obtained:

(1)

then pass through

The convolution changes the channels of the feature map into

，

Is the original number of channels of the feature map,

to be over-parametric, and finally passed

Convolution restores the channel number so as to fuse the information of different channels; adding the features obtained by the GAP module into the bottom layer feature graph pixel by pixel:

(2)

wherein

In order to obtain a feature map by the GAP module,

the characteristic diagram is an original characteristic diagram,

as a result of feature fusion;

the Gate module can control the passing of effective features in the low-dimensional feature map; through one

The depth of the feature map can be separated from the convolution and sigmoid functions to obtain a new feature map, and the new feature map is multiplied by the original feature map to obtain a feature map fused with low-dimensional high-quality features.

3. The method for example segmentation of histopathological nuclei based on deep learning as claimed in claim 1, wherein the adaptive receptor field segmentation branch in step S5 includes the following steps:

cutting a corresponding area in the original image according to the position and the size of a target frame generated by target detection, and adjusting the cut image to be in a uniform size through RoIAlign; and performing deformable convolution operation on the small pictures only containing the single cell nucleus example to obtain a segmentation result.

4. The method for example segmentation of histopathological nuclei based on deep learning as claimed in claim 1, wherein the method for enhancing segmentation result by consistency loss in step S6 is as follows:

clipping the segmentation result obtained by the conditional convolution according to the position and the size of a target frame generated by target detection, adjusting the clipped picture to be uniform in size through RoIAlign, and calculating consistency loss with the segmentation result obtained by the deformable convolution through focal loss to strengthen the segmentation result of the conditional convolution branch, wherein the focal loss is specifically as follows:

(3)

order to

(4)

Wherein the content of the first and second substances,

in order to be able to adjust the factor,

it is represented that the two segmentation results are predicted to be the same,

the same probability is predicted for both segmentation results,

the two segmentation results representing the model have the same predicted value of probability,

the larger the two segmentation results, the closer the segmented nuclear contour of the conditional convolution is to the true contour.

Images