CN117132774B - Multi-scale polyp segmentation method and system based on PVT - Google Patents

Abstract

The invention discloses a multi-scale polyp segmentation method and system based on PVT, and relates to the technical field of deep learning medical image semantic segmentation. The invention includes the following steps: acquiring a colorectal endoscope image to be detected and performing image preprocessing; using a PVTv2 backbone network to perform multi-scale feature extraction on the preprocessed image; and using a parallel Sobel edge decoder to extract original feature maps of different scales generated by PVT. Perform step-by-step fusion to obtain a global prediction map; use a multi-scale parallel dilated convolution attention module to extract features of multiple receptive fields from the original feature map; use the global prediction map to gradually guide and generate a multi-stage prediction map step by step; convert the global prediction The prediction loss is obtained by comparing the multi-stage prediction map and the ground truth map, and the last level prediction map is the final polyp segmentation prediction map. This application can accurately identify and segment polyps in colorectal images, providing doctors with effective help in correct diagnosis.

Description

A multi-scale polyp segmentation method and system based on PVT

Technical field

The invention belongs to the technical field of deep learning medical image semantic segmentation, and specifically relates to a multi-scale polyp segmentation method and system based on PVT.

Background technique

Colorectal cancer is a common malignant tumor, and its early detection and treatment are of great significance to improving the survival rate of patients. Since colorectal cancer has no typical symptoms in its early stages, the importance of screening for colorectal cancer has gradually received attention, and one of the screening methods is colorectal examination. In colorectal examination, polyps are similar in color to the normal tissue in the surrounding area, with variable shapes and sizes. It is even common that multiple small polyps may be adhered together, and the boundaries of the polyps are usually Blurred. This makes polyp segmentation based on colonoscopic imaging results multiple challenges. Traditional medical image segmentation methods, such as threshold segmentation and region growing segmentation, often require manual annotation assisted by professional doctors, and the segmentation results are affected by lighting conditions, doctor experience and subjective factors, making segmentation not only time-consuming and labor-intensive, but also There are large errors and instability. Therefore, how to achieve automatic segmentation of colonoscopy images to obtain clearer and more effective polyp segmentation results has become one of the hot topics in medical image segmentation.

In recent years, deep learning technology has been widely used in medical image segmentation. In colorectal examination, polyp segmentation methods based on convolutional neural networks have been widely used. There are two main typical architectures of convolutional neural networks used for polyp segmentation: U-shaped structure based on U-Net, and PraNet architecture. For example, U-Net uses an encoder-decoder structure to combine low-level and high-level features through skip connections, which can effectively retain spatial local information, but is susceptible to noise and occlusion. PraNet first uses a Parallel Part Decoder (PPD) to aggregate elevation features and generate a global map to roughly locate polyps, and then utilizes a reverse attention (RA) module to gradually refine regions and boundaries; however, due to convolutional neural Due to the limitations of the network itself, there are still certain problems with the segmentation accuracy and robustness of the model. Therefore, existing models need to be improved to improve their polyp segmentation performance in imaging results during colorectal examination.

Recently, the successful application of Transformer in the field of natural language processing (NLP) has inspired computer vision researchers, which has led to certain application and development of Transformer in computer vision research tasks. Since Transformer-based networks are good at capturing long-range dependencies of image targets through global self-attention, Transformer can be considered to be applied to the polyp segmentation task, that is: in the polyp segmentation task, Transformer is used to learn the differences in colorectal examination images. dependencies between regions, and use this information to improve the segmentation performance and robustness of the model; in addition, the model training process can be accelerated and the model's convergence speed can be improved by using advanced optimization algorithms. Through the application of these technologies, the segmentation effect of polyps in colorectal examination images can be further improved, providing clinicians with more accurate and reliable diagnostic results.

Contents of the invention

In view of the shortcomings of the existing technology, the present invention proposes a multi-scale polyp segmentation method and system based on PVT to effectively solve the problem of the inability to accurately identify polyp areas in the existing technology and further improve the polyp segmentation in colorectal examination images. The accuracy of boundaries and semantic completeness enable accurate, fast, and automatic segmentation of polyps.

In order to achieve the above objects, the present invention provides the following solutions:

A multi-scale polyp segmentation method based on PVT, including the following steps:

S1. Obtain the colonoscopy image to be detected and preprocess the colonoscopy image;

S2. Use the PVTv2 backbone network to perform multi-scale feature extraction on the preprocessed colorectal image, and obtain original feature maps of different scales;

S3. Use a parallel Sobel edge decoder to fuse the original feature maps step by step to obtain a global prediction map;

S4. Use a multi-scale parallel dilated convolutional attention module to extract features of multiple receptive fields from the original feature map;

S5. Use the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generate multi-stage prediction maps step by step;

S6. Compare the global prediction map and the multi-stage prediction map with the true value map, calculate the loss, and the final level prediction map obtained is the final polyp segmentation prediction map.

Preferably, in step S1, the method for preprocessing the colorectal image includes:

Random rotation, vertical flipping, horizontal flipping and normalization techniques were used to enhance the colorectal image data. Finally, the colorectal image was uniformly cropped to a size of 352×352, and {0.75, 1, 1.25} was used. A multi-scale strategy is used to scale the colorectal image.

Preferably, in S2, the method of using the PVTv2 backbone network to perform multi-scale feature extraction on the preprocessed colorectal image includes:

Determine whether the preprocessed colorectal image input to the PVTv2 backbone network is a 3-channel image. If it is a 3-channel image, it is directly sent to the network for feature extraction. If it is not a 3-channel image, 1×1 is used once. Convolution, adjust the number of channels of the image to 3;

Four-stage feature extraction is performed using the pre-trained model of PVTv2-B2.

Preferably, in S3, the method of using a parallel Sobel edge decoder to fuse the original feature maps step by step to obtain a global prediction map includes:

S31: Use a 1×1 convolution in the first branch to compress the feature map channel;

S32: In the second branch, first use 1×1 convolution to compress the feature map channel, and then use 1×3, 3×1 asymmetric convolution and 3×3 convolution with hole rate 3 respectively. Perform feature extraction on the feature map;

S33: In the third branch, first use 1×1 convolution to compress the feature map channel, and then use 1×5, 5×1 asymmetric convolution and 3×3 convolution with hole rate 5 respectively. Perform feature extraction on the feature map;

S34: In the fourth branch, first use 1×1 convolution to compress the feature map channel, and then use 1×7, 7×1 asymmetric convolution and 3×3 convolution with hole rate 7 respectively. Perform feature extraction on the feature map;

S35: Splice the compressed feature map of the first branch with the feature extracted feature maps of the second, third, and fourth branches in the channel dimension, and then use 1×1 convolution to compress the feature map channel. ;

S36: Add the compressed spliced feature map and the original feature map channel-compressed with the help of convolution pixel by pixel, and then pass through the ReLU nonlinear activation function before inputting into the Sobel operation;

S37: Add the feature maps after gradient sharpening with the Sobel operator pixel by pixel, and then undergo a 1×1 convolution and use bilinear interpolation upsampling operation to generate an initial polyp segmentation global prediction map.

Preferably, in S4, the method of using a multi-scale parallel dilated convolutional attention module to extract features of multiple receptive fields from the original feature map includes:

S41: Channel-compress the original feature maps of the four levels of the PVT encoder through a 1×1 convolution to obtain the number of channels of the original feature map. times of multi-channel feature maps;

S42: Group the channel-compressed feature maps evenly into channels and send them to four branches for processing. The processing method is: using 3×3 convolutions with hole rates of 1, 3, 5, and 7 respectively. Feature extraction of the corresponding branches, and then channel splicing of the processing results of the four branches;

S43: Perform 1×1 convolution on the channel spliced feature map, and then perform nonlinear operations of batch normalized BN and ReLU activation functions sequentially to obtain the processed feature map;

S44: Send the processed feature map to the CBAM module, perform further attention weighting on the feature map, and obtain a more distinguishable feature map.

Preferably, in S5, the method of using the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generating multi-stage prediction maps step by step includes:

S51: Spatially downsample the global prediction map so that the resolution is consistent with the resolution of the feature map in the fourth stage of PVT; then send it to the RA module for anti-attention operation to generate an attention map; then, and The feature map in the fourth stage of PVT is multiplied element-by-element, and further uses three 3×3 convolutions to reduce the feature dimension and adds it to the prediction map of the previous stage pixel-by-pixel to generate the prediction map of this stage;

S52: Send the prediction map of this stage to the next stage, and perform the same operation as the S51 stage to guide the generation of the last level feature map.

Preferably, in S6, the method of comparing the global prediction map and the multi-stage prediction map with the true value map, calculating the loss, and the obtained last-level prediction map is the final polyp segmentation prediction map includes:

The global prediction map and the multi-stage prediction map are subjected to a spatial upsampling operation of bilinear interpolation, the size of all prediction maps is adjusted to the same size as the ground truth map corresponding to the input image, and the weighted BCE and weighted BCE are calculated. Mixed losses of IOU;

The weighted BCE loss calculation method is defined as:

Among them, G represents the true value map; P represents the prediction map; (x, y) represents any pixel position in the image, and the corresponding weight coefficient ω (x, y) is used to represent the importance of the pixel (x, y). The specific calculation will be/> set to 5;

The weighted IOU loss calculation method is defined as:

The method of combining the weighted BCE and weighted IOU losses to obtain the hybrid loss of the prediction map relative to the true value map is:

The invention also provides a multi-scale polyp segmentation system based on PVT, including: a preprocessing module, a first feature extraction module, a fusion module, a second feature extraction module, a guidance module and a prediction module;

The preprocessing module is used to obtain the colorectal endoscope image to be detected and preprocess the colorectal endoscope image;

The first feature extraction module is used to perform multi-scale feature extraction on the preprocessed colorectal endoscope image using the PVTv2 backbone network to obtain original feature maps of different scales;

The fusion module is used to fuse the original feature maps step by step using a parallel Sobel edge decoder to obtain a global prediction map;

The second feature extraction module is used to extract features of multiple receptive fields from the original feature map using a multi-scale parallel atrous convolution attention module;

The guidance module is configured to use the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generate multi-stage prediction maps step by step;

The prediction module is used to compare the global prediction map and the multi-stage prediction map with the true value map, and calculate the loss. The obtained last-level prediction map is the final polyp segmentation prediction map.

Compared with the prior art, the beneficial effects of the present invention are:

1. The present invention uses PVTv2 as the backbone network, replacing the ResNet backbone network in PraNet, so that the network has better global feature extraction capabilities.

2. The present invention proposes a parallel Sobel edge decoder to fuse feature maps at different scales, thereby improving the model's segmentation effect on polyps of different sizes.

3. The present invention also proposes a multi-scale parallel dilated convolution attention module to extract features from feature maps of different scales under multiple receptive fields, and uses CBAM to redistribute weights on the feature maps to extract regions of interest.

4. The present invention also uses a new loss function to train the model by introducing a combination of weighted BCE loss and weighted IOU loss, which eliminates the impact of uneven distribution of positive and negative samples, thereby further improving the segmentation accuracy of the model. and robustness.

Description of drawings

In order to explain the technical solutions of the present invention more clearly, the drawings required to be used in the embodiments are briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary people in the art, Technical personnel can also obtain other drawings based on these drawings without exerting creative labor.

Figure 1 is a schematic flow chart of the implementation of the PVT-based multi-scale polyp segmentation method provided in the embodiment of the present invention;

Figure 2 is a schematic diagram of the PraNet network model structure;

Figure 3 is a schematic structural diagram of the network model of the PVT-based multi-scale polyp segmentation method constructed in the embodiment of the present invention;

Figure 4 shows the parallel Sobel edge decoder module of the present invention;

Figure 5 is a schematic diagram of RFB operation in the parallel Sobel edge decoder module of the present invention;

Figure 6 shows the multi-scale parallel dilated convolution attention module of the present invention;

Figure 7 is an experimental visualization comparison diagram of the multi-scale polyp segmentation method based on PVT in the embodiment.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

Embodiment 1

Figure 1 is a schematic flow chart of the implementation of the PVT-based multi-scale polyp segmentation method provided in the embodiment of the present invention. As shown in Figure 1, the PVT-based multi-scale polyp segmentation method includes the following steps:

Step S1: Obtain the colorectal endoscope images to be detected in Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS and CVC-T, and preprocess the images.

In practice, preprocessing may include random rotation, vertical flipping, horizontal flipping, and normalization of the colorectal image to be detected, so as to process the colorectal image to be detected into an image that meets the detection requirements. Then the size of the image is uniformly cropped to 352 rows × 352 columns, and the multi-scale strategy of {0.75, 1, 1.25} is used to scale the image. These preprocessing techniques can provide more reliable input data for the neural network model, making the network Polyps of different sizes can be processed, thereby improving polyp segmentation during colorectal endoscopy.

Step S2: Use the PVTv2 backbone network to extract multi-scale features from the preprocessed image.

Refer to Figure 2 for the PraNet network model structure. PraNet is a parallel reverse attention network that can accurately segment polyps from colonoscopy images. The network first uses the Parallel Part Decoder (PPD) to aggregate high-level features to generate an initial global prediction map to guide subsequent steps; then it uses the reverse attention module to establish the relationship between the target area and the boundary, fully Leverage the complementarity between edges and regions. However, due to the limited receptive field of the PraNet backbone network, it can only capture local information, while ignoring the spatial context and global information. The present invention improves it. Figure 3 is a schematic structural diagram of the network model of the PVT-based multi-scale polyp segmentation method constructed in the embodiment of the present invention.

In the embodiment of the present invention, the PVTv2 backbone network is used for multi-scale feature extraction. PVTv2 is one of the most advanced pre-training models currently. Because it uses the Pyramid Vision Transformer architecture, it can provide more accurate and robust feature extraction capabilities for input images. This model can include image classification, target detection and segmentation. Excellent performance in a variety of visual tasks. Therefore, the PVTv2 backbone network can be used to better handle polyps with different resolutions in colorectal images.

In order to model local continuity information, PVTv2 uses overlapping patch embeddings to label images. We enlarge the patch windows so that adjacent windows overlap half of the area and pad the feature maps with zeros to maintain resolution. To ensure that PVTv2 has the same linear complexity as CNN, we use zero-padding convolutions to implement overlapping patch embeddings. The specific operations of using the PVTv2 backbone network for multi-scale feature extraction are as follows:

S201: First check whether the preprocessed colorectal image input by the network is a 3-channel image.

S202, if it is a 3-channel image, it is directly sent to the network for feature extraction; if it is not a 3-channel image, channel adjustment is performed with a 1×1 convolution, so that the adjusted number of image channels is changed to 3.

S203. This model uses the PVTv2-B2 pre-training model to perform four-stage feature extraction. In these four stages, the number of layers of PVTv2 basic units are 3, 3, 6, and 3 respectively. The sizes of the original feature maps _Xi of the four levels generated (i.e., the number of channels × the number of rows × the number of columns) are: 64×(H/4)×(W/4), 128×(H/8)×(W/8), 320×(H/16)×(W/16), 512×(H/32)× (W/32).

Step S3: Use a parallel Sobel edge decoder to fuse the original feature maps of different scales generated by PVT step by step to obtain a global prediction map. As shown in Figure 4, it is the parallel Sobel edge decoder module of the present invention. Its specific operation is as follows:

As shown in Figure 4, X ₁ , X ₂ , X ₃ , and X ₄ in the figure are the original feature maps of the four levels of the PVT stage. We perform RFB operations in parallel. The specific operations are shown in Figure 5. Taking X ₁ as an example, four branch convolution operations are required for X ₁ . The operations are as follows:

S301, use a 1×1 convolution in the first branch to compress the number of channels of the feature map. To facilitate calculation, we compress the number of channels of all features to 32.

S302, in the second branch, first use 1×1 convolution to compress the number of channels of the feature map, and then use 1×3, 3×1 asymmetric convolution and 3× with hole rate 3 respectively. 3 convolution performs feature extraction on the feature map.

S303, in the third branch, first use 1×1 convolution to compress the number of channels of the feature map, and then use 1×5, 5×1 asymmetric convolution and 3× with hole rate 5 respectively. 3 convolution performs feature extraction on the feature map.

S304, in the fourth branch, first use 1×1 convolution to compress the number of channels of the feature map, and then use 1×7, 7×1 asymmetric convolution and 3× with hole rate 7 respectively. 3 convolution performs feature extraction on the feature map.

S305, in the fourth branch, first use 1×1 convolution to compress the number of channels of the feature map, and then use 1×7, 7×1 asymmetric convolution and 3× with hole rate 7 respectively. 3 convolution performs feature extraction on the feature map.

S306: Splice the feature maps of these four branches in the channel dimension, and then use 1×1 convolution to compress the number of channels of the feature map.

S307: Add the feature map and the original feature map compressed by 1×1 convolution channel pixel by pixel, and then perform the nonlinear operation of the ReLU activation function, and then perform the Sobel operation.

The RFB operation of the X ₂ , X ₃ , and X ₄ feature maps is also the same as above.

For the four feature maps processed above, the Sobel operator is used for gradient sharpening and then added pixel by pixel, and then undergoes a 1×1 convolution and uses a spatial upsampling operation based on bilinear interpolation to generate the initial polyp segmentation. Global prediction map.

Step S4: Use the multi-scale parallel atrous convolution attention module to extract features of multiple receptive fields from the original feature map. As shown in Figure 6, it is the multi-scale parallel dilated convolution attention module of the present invention. Its specific operation is as follows:

S401, send the original feature maps of the four levels of the PVT encoder to the multi-scale parallel atrous convolution attention module for further feature extraction. First, they undergo a 1×1 convolution channel compression, so that the processed feature maps The number of channels is the number of channels in the original feature map.

S402, divide the processed feature maps evenly into four groups according to channels, and send them to four branches for processing, that is, use 3×3 convolutions with hole rates of 1, 3, 5, and 7 for feature extraction, and then The processing results are channel spliced.

S403: Perform 1×1 convolution on the channel spliced feature map, and then sequentially perform nonlinear activation of batch normalized BN and ReLU functions.

S404: Input the feature map processed in S403 into the CBAM module to further enhance the feature map with attention. The CBAM module mainly contains two parts: channel attention and spatial attention. The channel attention module assigns global importance to the feature map between channels to reduce redundant calculations; the spatial attention module retains more effective local feature information in the space by paying varying degrees of attention to local spatial information. The feature map after the CBAM module is further optimized to obtain a more distinguishable feature map. The final generated four sizes (ie: number of channels × number of rows × number of columns) are 64×(H/4)×(W/4), 128×(H/8)×(W/8), 320×( Feature maps of H/16)×(W/16) and 512×(H/32)×(W/32).

Step S5: Use the global prediction map to gradually guide and generate multi-stage prediction maps step by step. Its operation is the same as PraNet, and the specific operations are as follows:

S501: Spatially downsample the global prediction map so that its resolution is consistent with the resolution of the feature map in the fourth stage of PVT, and then send it to the RA module for anti-attention operation to generate an attention map, which is then combined with the resolution of the feature map in the fourth stage of PVT. The feature maps of the four stages are multiplied element by element, and then three 3×3 convolutions are used to reduce the feature dimension. They are further added to the prediction map of the previous stage pixel by pixel to generate the prediction map of this stage.

S502, send the prediction map of this stage to the next stage, and perform the same operation as the S501 stage to guide the generation of the last level feature map.

Use the global prediction map generated by the parallel Sobel edge decoder to gradually guide the four feature maps generated by S4 and generate prediction maps step by step. The specific operations are as follows:

S501: Use the global prediction map generated by S3 and the feature map with a size of 512×(H/32)×(W/32) generated by S4 for guided prediction. First, perform a spatial downsampling operation on the global prediction map generated by S3. Its size is reduced to the size of the S4 feature map; then it is fused with the S4 feature map to generate a prediction map with a size of 1×(H/32)×(W/32). This guided prediction method can combine global and local information to more accurately segment polyps in colorectal images.

S502: Use the prediction map generated by S501 to conduct guided prediction on the feature map with a size of 320×(H/16)×(W/16) generated by S4. First, use the prediction map generated by S501 to perform a spatial upsampling operation to reduce its size. Expand to the size of the S4 feature map; then fuse it with the S4 feature map to generate a prediction map with a size of 1 × (H/16) × (W/16); then, guide the prediction for the next stage in sequence, and generate the size in sequence The two polyp segmentation prediction maps are 1×(H/8)×(W/8) and 1×(H/4)×(W/4) respectively.

Step S6: Compare the generated global prediction map and multi-stage prediction map with the true value map to calculate the prediction loss. The last level prediction map is the final polyp segmentation prediction map. The specific operations are as follows:

S601, perform the spatial upsampling operation of bilinear interpolation on these five prediction maps, adjust the size of the prediction map to the same size as the ground truth map corresponding to the input image, and calculate a hybrid loss composed of weighted BCE and weighted IOU. L _seg .

The weighted BCE loss is calculated as follows:

G represents the true value map, P represents the predicted map, and (x, y) represents any pixel position in the image.

The weight coefficient ω(x,y) represents the importance of pixel (x,y), and its calculation method is:

For specific calculations, we will Set to 5.

The weighted IOU loss is calculated as follows:

The hybrid loss function of the final predicted map P relative to the ground truth map G is:

in and/> are the weighted IOU loss and the weighted BCE loss.

In order to illustrate the effectiveness of this method, the Kvasir-SEG and CVC-ClinicDB data sets are used as training sets to train this method, and then five data sets of Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS and CVC-T are used Tests are conducted and the test results are compared with mainstream polyp segmentation algorithms in the existing technology. During the specific experiment, based on the PyTorch 1.8 deep learning framework, an NVIDIA RTX 2080Ti graphics card was used for training, with a video memory of 11G; the input image size was set to 352×352, the initial learning rate was 1e-4, the AdamW optimizer was used, and the batch size was set to 6. The total number of training rounds is 100 epochs. The test results on the Kvasir-SEG and CVC-ClinicDB data sets are shown in Table 1:

Table 1 Test results of polyp segmentation of the present invention and other 7 methods in Kvasir-SEG and CVC-ClinicDB data sets

Table 1

The experimental results on CVC-ClinicDB, CVC-ColonDB, and ETIS data sets are shown in Table 2:

Table 2 Test results of polyp segmentation of the present invention and other 7 methods in CVC-ClinicDB, CVC-ColonDB, and ETIS data sets

Table 2

Among the 7 evaluation indicators, the first 2 are commonly used evaluation indicators in semantic segmentation tasks, and the closer the value of the first 6 indicators is to 1, the better the segmentation result is; the value of the 7th indicator is not less than 0, and the value is The closer to 0 the better.

Figure 7 is a visual comparison of the experimental results of the method of the present invention and other methods. From the results, it can be concluded that the polyp segmentation method proposed by the present invention can obtain segmentation results with more accurate boundaries and more complete semantic structure.

Embodiment 2

The invention also provides a multi-scale polyp segmentation system based on PVT, including: a preprocessing module, a first feature extraction module, a fusion module, a second feature extraction module, a guidance module and a prediction module;

The preprocessing module is used to obtain the colorectal endoscope image to be detected and preprocess the colorectal endoscope image;

The first feature extraction module is used to perform multi-scale feature extraction on the preprocessed colorectal endoscope image using the PVTv2 backbone network to obtain original feature maps of different scales;

The fusion module is used to fuse the original feature maps step by step using a parallel Sobel edge decoder to obtain a global prediction map;

The second feature extraction module is used to extract features of multiple receptive fields from the original feature map using a multi-scale parallel atrous convolution attention module;

The guidance module is configured to use the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generate multi-stage prediction maps step by step;

The prediction module is used to compare the global prediction map and the multi-stage prediction map with the true value map, and calculate the loss. The obtained last-level prediction map is the final polyp segmentation prediction map.

The above-described embodiments are only descriptions of preferred modes of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, those of ordinary skill in the art can make various modifications to the technical solutions of the present invention. All deformations and improvements shall fall within the protection scope determined by the claims of the present invention.

Claims (5)

1. A multi-scale polyp segmentation method based on PVT, characterized by including the following steps: S1. Obtain the colonoscopy image to be detected and preprocess the colonoscopy image; S2. Use the PVTv2 backbone network to perform multi-scale feature extraction on the preprocessed colorectal image, and obtain original feature maps of different scales; S3. Use a parallel Sobel edge decoder to fuse the original feature maps step by step to obtain a global prediction map; In the S3, the method of using a parallel Sobel edge decoder to fuse the original feature maps step by step to obtain a global prediction map includes: S31: Use a 1×1 convolution in the first branch to compress the feature map channel; S32: In the second branch, first use 1×1 convolution to compress the feature map channel, and then use 1×3, 3×1 asymmetric convolution and 3×3 convolution with hole rate 3 respectively. Perform feature extraction on the feature map; S33: In the third branch, first use 1×1 convolution to compress the feature map channel, and then use 1×5, 5×1 asymmetric convolution and 3×3 convolution with hole rate 5 respectively. Perform feature extraction on the feature map; S34: In the fourth branch, first use 1×1 convolution to compress the feature map channel, and then use 1×7, 7×1 asymmetric convolution and 3×3 convolution with hole rate 7 respectively. Perform feature extraction on the feature map; S35: Splice the compressed feature map of the first branch with the feature extracted feature maps of the second, third, and fourth branches in the channel dimension, and then use 1×1 convolution to compress the feature map channel. ; S36: Add the compressed spliced feature map and the original feature map channel-compressed with the help of convolution pixel by pixel, and then pass through the ReLU nonlinear activation function before inputting into the Sobel operation; S37: Add the feature maps after gradient sharpening by the Sobel operator pixel by pixel, and then undergo a 1×1 convolution and use bilinear interpolation upsampling operation to generate an initial polyp segmentation global prediction map; S4. Use a multi-scale parallel dilated convolutional attention module to extract features of multiple receptive fields from the original feature map; In S4, the method of using a multi-scale parallel dilated convolutional attention module to extract features of multiple receptive fields from the original feature map includes: S41: Channel-compress the original feature maps of the four levels of the PVT encoder through a 1×1 convolution to obtain the number of channels of the original feature map. times of multi-channel feature maps; S42: Group the channel-compressed feature maps evenly into channels and send them to four branches for processing. The processing method is: use 3×3 convolutions with hole rates of 1, 3, 5, and 7 to perform corresponding branches. Feature extraction, and then channel splicing of the processing results of the four branches; S43: Perform 1×1 convolution on the channel spliced feature map, and then perform nonlinear operations of batch normalized BN and ReLU activation functions sequentially to obtain the processed feature map; S44: Send the processed feature map to the CBAM module to further weight the feature map with attention to obtain a more distinguishable feature map; S5. Use the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generate multi-stage prediction maps step by step; In S5, the method of using the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction and generating multi-stage prediction maps step by step includes: S51: Spatially downsample the global prediction map so that the resolution is consistent with the resolution of the feature map in the fourth stage of PVT; then send it to the RA module for anti-attention operation to generate an attention map; then, and The feature map in the fourth stage of PVT is multiplied element-by-element, and further uses three 3×3 convolutions to reduce the feature dimension and adds it to the prediction map of the previous stage pixel-by-pixel to generate the prediction map of this stage; S52: Send the prediction map of this stage to the next stage, and perform the same operation as the S51 stage to guide the generation of the last level feature map; S6. Compare the global prediction map and the multi-stage prediction map with the true value map, calculate the loss, and the final level prediction map obtained is the final polyp segmentation prediction map. 2. The multi-scale polyp segmentation method based on PVT according to claim 1, characterized in that, in the S1, the method for preprocessing the colorectal endoscope image includes: Random rotation, vertical flipping, horizontal flipping and normalization techniques were used to enhance the colorectal image data. Finally, the colorectal image was uniformly cropped to a size of 352×352, and {0.75, 1, 1.25} was used. A multi-scale strategy is used to scale the colorectal image. 3. The multi-scale polyp segmentation method based on PVT according to claim 1, characterized in that, in the S2, a method of using the PVTv2 backbone network to perform multi-scale feature extraction on the preprocessed colorectal endoscope image include: Determine whether the preprocessed colorectal image input to the PVTv2 backbone network is a 3-channel image. If it is a 3-channel image, it is directly sent to the network for feature extraction. If it is not a 3-channel image, 1×1 is used once. Convolution, adjust the number of channels of the image to 3; Four-stage feature extraction is performed using the pre-trained model of PVTv2-B2. 4. The multi-scale polyp segmentation method based on PVT according to claim 1, characterized in that, in the S6, the global prediction map and the multi-stage prediction map are compared with the true value map, and the loss is calculated to obtain The last level prediction map is the final polyp segmentation prediction map. Methods include: The global prediction map and the multi-stage prediction map are subjected to a spatial upsampling operation of bilinear interpolation, the size of all prediction maps is adjusted to the same size as the ground truth map corresponding to the input image, and the weighted BCE and weighted BCE are calculated. Mixed losses of IOU; The weighted BCE loss calculation method is defined as: Among them, G represents the true value map; P represents the prediction map; (x, y) represents any pixel position in the image, and the corresponding weight coefficient ω (x, y) is used to represent the importance of the pixel (x, y). The specific calculation will be/> set to 5; The weighted IOU loss calculation method is defined as: The method of combining the weighted BCE and weighted IOU losses to obtain the hybrid loss of the prediction map relative to the true value map is: 5. A multi-scale polyp segmentation system based on PVT, characterized by comprising: a preprocessing module, a first feature extraction module, a fusion module, a second feature extraction module, a guidance module and a prediction module; The preprocessing module is used to obtain the colorectal endoscope image to be detected and preprocess the colorectal endoscope image; The first feature extraction module is used to perform multi-scale feature extraction on the preprocessed colorectal endoscope image using the PVTv2 backbone network to obtain original feature maps of different scales; The fusion module is used to fuse the original feature maps step by step using a parallel Sobel edge decoder to obtain a global prediction map; The process of using a parallel Sobel edge decoder to fuse the original feature maps step by step to obtain a global prediction map includes: In the first branch, a 1×1 convolution is used to compress the feature map channel; In the second branch, 1×1 convolution is first used to compress the feature map channel, and then 1×3, 3×1 asymmetric convolution and 3×3 convolution pair with hole rate 3 are used respectively. Feature map for feature extraction; In the third branch, 1×1 convolution is first used to compress the feature map channel, and then 1×5, 5×1 asymmetric convolution and 3×3 convolution pair with hole rate 5 are used respectively. Feature map for feature extraction; In the fourth branch, 1×1 convolution is first used to compress the feature map channel, and then 1×7, 7×1 asymmetric convolution and 3×3 convolution pair with hole rate 7 are used respectively. Feature map for feature extraction; Splice the compressed feature map of the first branch with the feature extracted feature maps of the second, third, and fourth branches in the channel dimension, and then use 1×1 convolution to compress the feature map channel; The compressed spliced feature map and the original feature map channel-compressed with the help of convolution are added pixel by pixel, and then passed through the ReLU nonlinear activation function and then input into the Sobel operation; The feature maps after gradient sharpening by the Sobel operator are added pixel by pixel, and then undergo a 1×1 convolution and use bilinear interpolation upsampling operation to generate an initial polyp segmentation global prediction map; The second feature extraction module is used to extract features of multiple receptive fields from the original feature map using a multi-scale parallel atrous convolution attention module; The process of using multi-scale parallel dilated convolutional attention modules to perform multi-receptive field feature extraction on the original feature map includes: The original feature maps of the four levels of the PVT encoder are channel-compressed through a 1×1 convolution to obtain the number of original feature map channels. times of multi-channel feature maps; The channel-compressed feature maps are evenly grouped into channels and sent to four branches for processing. The processing method is: using 3×3 convolutions with hole rates of 1, 3, 5, and 7 to perform the features of the corresponding branches. Extract, and then perform channel splicing on the processing results of the four branches; Perform 1×1 convolution on the channel spliced feature map, and then perform nonlinear operations of batch normalized BN and ReLU activation functions sequentially to obtain the processed feature map; The processed feature map is sent to the CBAM module, and the feature map is further weighted with attention to obtain a more distinguishable feature map; The guidance module is configured to use the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction, and generate multi-stage prediction maps step by step; The process of using the global prediction map to gradually guide the original feature map after multi-receptive field feature extraction and gradually generating a multi-stage prediction map includes: The global prediction map is spatially downsampled to make the resolution size consistent with the resolution of the feature map in the fourth stage of PVT; then it is sent to the RA module for anti-attention operation to generate an attention map; after that, it is compared with the feature map in the fourth stage of PVT. The feature maps of the four stages are multiplied element by element, and then the feature dimensionality reduction using three times 3×3 convolution is performed and added pixel by pixel to the prediction map of the previous stage to generate the prediction map of this stage; The prediction map of this stage is sent to the next stage, and the same operation as the previous stage is performed to guide the generation of the last level feature map; The prediction module is used to compare the global prediction map and the multi-stage prediction map with the true value map, and calculate the loss. The obtained last-level prediction map is the final polyp segmentation prediction map.