CN112200795A - Large intestine endoscope polyp detection method based on deep convolutional network - Google Patents

Large intestine endoscope polyp detection method based on deep convolutional network Download PDF

Info

Publication number
CN112200795A
CN112200795A CN202011143365.1A CN202011143365A CN112200795A CN 112200795 A CN112200795 A CN 112200795A CN 202011143365 A CN202011143365 A CN 202011143365A CN 112200795 A CN112200795 A CN 112200795A
Authority
CN
China
Prior art keywords
feature
polyp
image
neural network
convolutional neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011143365.1A
Other languages
Chinese (zh)
Inventor
曹鱼
王德纯
刘本渊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Huiwei Intelligent Medical Technology Co ltd
Original Assignee
Suzhou Huiwei Intelligent Medical Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Huiwei Intelligent Medical Technology Co ltd filed Critical Suzhou Huiwei Intelligent Medical Technology Co ltd
Priority to CN202011143365.1A priority Critical patent/CN112200795A/en
Publication of CN112200795A publication Critical patent/CN112200795A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Abstract

The invention relates to a large intestine endoscope polyp detection method based on a deep convolutional network, which comprises the following steps: s1, constructing a convolutional neural network, pre-training the convolutional neural network in ImageNet through a backbone network, and training the convolutional neural network through a training image data set containing polyp types and position information; s2, preprocessing the color endoscope image; s3, inputting the preprocessed color endoscope image into a convolutional neural network, and extracting picture characteristic layer information; s4, feature enhancement and view field promotion are carried out on the picture feature layer information; s5, decoding the picture characteristic layer information into the type of polyp and the position information thereof, the method for detecting the polyp of the large intestine endoscope based on the deep convolutional network has convenient operation, can improve the reasoning speed and the sensitivity to the polyp image, thereby being capable of quickly positioning the polyp position in the endoscope image and avoiding the inconsistency of label setting caused by different rotation angles in the same polyp object under the endoscope image.

Description

Large intestine endoscope polyp detection method based on deep convolutional network
Technical Field
The invention belongs to the technical field of polyp detection technology, and particularly relates to a method for detecting polyps of a large intestine endoscope based on a deep convolutional network.
Background
Colon cancer belongs to high-incidence malignant tumors all over the world, 1.8 ten thousand cases of colon cancer and 881000 deaths are expected in 2018, intestinal cancer develops from benign polyp lesions, and therefore, early screening discovery is a very effective prevention means; for example, benign adenomatous polyps and hyperplastic polyps can be effectively prevented from developing malignant tumors by using an endoscope for removal.
In the article, "fans in the mistake Rate of Polyps in a Back-To-top-Back Colonoscopy Study" published by Leufkens et al in 2012, 25% of adenomatous Polyps in the experiment are missed by doctors, and in the actual operation, many Factors cause the missed detection of Polyps, which can be summarized as: polyp undersize, polyp is covered by the foreign matter, it is too fast to retreat the mirror, polyp does not appear in the camera lens, polyp appears in the camera lens but is neglected by the doctor, although some factors can reduce the miss detection rate through intestinal tract preparation before the operation, promote the equipment performance and promote doctor's operation, because endoscope operation relies on doctor's vision very much to look for the colon polyp that the characteristic is not especially obvious in the intestinal tract, because the missing detection that artificial factors such as endoscope equipment operator fatigue, experience and sensitivity cause is very difficult to avoid, therefore automatic polyp detects and is used for helping the doctor to reduce the miss detection rate very effective tool.
Ameling et al in Texture-based polyp detection in colonoscopy uses a method of distinguishing normal intestinal lining from polyps based on the Texture, color and shape of the polyp surface, but this method can cause a large amount of false positives in real environments, such as some normal tissues, bumps, spots on the lens, with obvious vascular regions and intestinal foreign bodies.
Regarding the Detection of colon cancer pictures by using the existing techniques such as deep learning, Mo et al use fast-RCNN for polyp Detection in "Polyps Detection in Endoscopic video Based on fast R-CNN", a dual-stage anchor frame-Based object detector, which is different from the traditional image processing method, deep learning calls a large-scale convolutional neural network CNN to extract image feature information, and uses more features to distinguish easily confused morphological features of Polyps from morphological features of normal intestinal walls, but because the dual-stage architecture is used, the first stage generates more candidate frames, which does not meet the requirement of real-time Detection.
Therefore, the two existing detection methods have some defects and cannot meet the actual use requirements.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a method for detecting polyps of a large intestine endoscope based on a deep convolutional network, which has high detection method precision, low false alarm rate and high detection speed.
In order to achieve the purpose, the invention adopts the technical scheme that: a large intestine endoscope polyp detection method based on a deep convolutional network comprises the following steps:
s1: constructing a convolutional neural network, wherein the convolutional neural network is pre-trained in ImageNet by a backbone network and then trained by a training image data set containing polyp categories and position information;
s2: preprocessing a color endoscope image;
s3: inputting the preprocessed color endoscope image into a convolutional neural network and extracting picture characteristic layer information;
s4: carrying out feature enhancement and visual field domain promotion on the image feature layer information;
s5: the picture feature layer information is decoded into the category of polyps and its location information.
Further, the convolutional neural network comprises sixteen layers of convolution, and each layer of network updates the weight parameters thereof through back propagation;
the convolutional neural network is divided into different stages with the step length of 4,8,16,32 and 64 by taking max pool as a boundary;
further, the step of preprocessing the color endoscope image is as follows:
cutting a black edge of the color endoscope picture;
the image size is adjusted to a uniform size and pixel mean normalization is performed.
Further, in S3, the preprocessed color endoscope image is input into a convolutional neural network for convolution operation, so as to obtain image feature layer information with sizes of 80x80,40x40,20x20,10x10, and 5x5, which respectively correspond to outputs with step sizes of 4,8,16,32, and 64 in the convolutional neural network.
Further, the step in S4 is as follows:
a, performing feature enhancement on the preprocessed picture feature layer information;
and b, inputting the image characteristic layer information subjected to characteristic enhancement into a visual field lifting module to obtain the image characteristic layer information subjected to visual field lifting.
Further, the processing method of the feature enhancement is as follows:
extracting k intermediate layers from a convolutional neural network, wherein k is 5;
secondly, when feature fusion is carried out on the feature layer Si, wherein i is 0-k-1, firstly, smoothing and dimensionality reduction operation is carried out on the feature layer Si +1 by using a convolution layer with a convolution kernel of 1x 1;
thirdly, performing upsampling by using the transposed convolution with the convolution kernel of 2x2 and the step length of 2, and adding the result and the characteristic layer Si to obtain an enhanced characteristic Ei, wherein the formula is as follows:
Ei＝Si+deconv(smooth(Si+1))；
wherein, deconv () is a transposed convolution and smooth () is a convolution operation of 1x 1;
further, the horizon promoting module comprises three branches, each branch processes 1/3 channels respectively, and comprises 1,2,3 sparse convolutions with 3 convolution kernels, wherein 3 × 3 step length is 1 sparse convolution and 3 sparse convolution; and after the picture characteristic layer information enters the visual field lifting module, outputting the convolution result of each branch in the cascade connection of the channel level.
Further, the decoding process in S5 is as follows:
for a bounding box regression feature layer, the output of a feature point x, y is predicted to be
Figure BDA0002738892920000031
Wherein the content of the first and second substances,
Figure BDA0002738892920000032
Figure BDA0002738892920000041
decoded to obtain the predicted object position cx，cy，w，h]Wherein c isx，cyThe value x, y, w, h of the center point of the object frame is the width and height of the object frame, and then the classification and position information [ s, cx, xy, w, h ] of the polyp is obtained by combining the classification information of the object]；
Where s is the object class.
Further, the training method of the convolutional neural network is as follows:
firstly, pre-training is carried out on ImageNet by a backbone network, and then training is carried out by a training image data set containing polyp types and position information to obtain a convolutional neural network;
preprocessing a color endoscope image;
inputting the preprocessed color endoscope image into a convolution neural network to extract image characteristic layer information;
fourthly, performing feature enhancement and visual field promotion on the image feature information extracted by the convolutional neural network;
giving different labels to the feature points of different feature layers according to the size of the target polyp;
suppose a polyp target bounding box b ═ cx，cy，w，h),(cx，cy) Is the center point of the bounding box, and (w, h) is width and height; it is allocated to the scaling step size slCharacteristic layer K oflThe feature layer contains m2A feature point, where m is the feature layer side length, set
Figure BDA0002738892920000042
Representing feature points in an original drawingIn which xi＝sl×i，yj＝sl×j；
The positive sample region is
Figure BDA0002738892920000043
The non-positive sample is
Figure BDA0002738892920000044
Wherein
Figure BDA0002738892920000045
Figure BDA0002738892920000046
Area size according to epsilonp＝0.75，εnScaling is performed as 1.25;
when the feature point is in the area of the positive sample, determining as the positive sample; when the feature point is between the regions of the positive sample and the non-positive sample, determining to ignore the sample; when the feature point is outside the region of the non-positive sample, determining as a negative sample;
sixthly, polyp target distribution of the characteristic points and polyp target screening of the current layer are carried out;
when positive and negative samples are judged, assuming that an object b belongs to a feature layer i, i belongs to [0, k-1], and d is a value of the distance scaling size between the current feature layer and the optimal feature layer, wherein the value is as follows:
δ1＝max(Lcosin(λ，d，k)，0)；
Figure BDA0002738892920000051
the new non-positive samples for the current layer are therefore:
Figure BDA0002738892920000052
wherein, deltalThe calculated scaling non-positive sample proportion;
and for different polyp targets, different distance heat force values can be generated for different feature layers to assist the determination of the polyp target at the current position by the feature points, and the specific formula is as follows:
Figure BDA0002738892920000053
wherein:
Figure BDA0002738892920000054
Figure BDA0002738892920000055
min (x, y) is the minimum value of x and y, and max (x, y) is the maximum value of x and y in the same way;
wherein left, right, top and bottom are respectively expressed as the distance from the feature point to the left frame, the right frame, the upper frame and the lower frame of the sample frame;
using binary cross entry as a loss function in the training phase of the generated thermodynamic diagram, and performing point multiplication on the generated thermodynamic diagram and the object class predicted value obtained in the step 8;
in the training stage, different weights are generated by using a formula when classification loss functions are calculated for different characteristic layers for different polyp targets; different distances are given different weights in calculating the loss by a Gaussian distribution formula
Figure BDA0002738892920000056
The concrete formula is as follows:
Figure BDA0002738892920000057
Figure BDA0002738892920000061
wherein alpha is 1, cbIs the position of the target object，cpIs the position of the characteristic point;
ninthly, calculating a bounding box regression loss L by using a smooth L1 loss function as a fitting loss function for the position sizes of the feature points in the training phaselocAnd encoding the feature points of the bounding box by using a function:
Figure BDA0002738892920000062
Figure BDA0002738892920000063
wherein the content of the first and second substances,
Figure BDA0002738892920000064
as a regression target, β is 0.45;
the r training phase uses the following loss function for the classification of feature points:
LF(pt)＝-αt(1-pt)γlog(pt)；
Figure BDA0002738892920000065
wherein L isF(pt) is focal length. L isCESorting the loss function for cross entry, LclsAs a final classification loss function
Due to the application of the technical scheme, compared with the prior art, the invention has the following advantages:
the method for detecting the polyp of the large intestine endoscope based on the deep convolutional network has the advantages that the operation is convenient, the reasoning speed and the sensitivity to the polyp image can be improved, the polyp position can be rapidly positioned in the endoscope image, the inconsistency of label setting caused by different rotation angles in the same polyp object under the endoscope image is avoided, the actual use requirements are met, and the method has high practicability and popularization value.
Drawings
The technical scheme of the invention is further explained by combining the accompanying drawings as follows:
FIG. 1 is a block diagram of a first embodiment of the present invention;
FIG. 2 is a schematic diagram of the process of using the present invention;
FIG. 3 is a diagram illustrating the practical use of the method for determining positive and negative samples in the present invention;
FIG. 4 is a diagram of the actual detection effect of the present invention;
FIG. 5 is a diagram of another practical test result of the present invention;
FIG. 6 is a graph of the recognition effect of the present invention;
wherein: polyp 1, first solid frame 11, second solid frame 13.
Detailed Description
The invention is described in further detail below with reference to the figures and the embodiments.
Referring to fig. 1, the method for detecting polyps of a large intestine endoscope based on a deep convolutional network according to the present invention includes the following steps: s1: constructing a convolutional neural network, wherein the convolutional neural network is pre-trained in ImageNet by a backbone network and then trained by a training image data set containing polyp categories and position information; s2: preprocessing a color endoscope image; s3: inputting the preprocessed color endoscope image into a convolutional neural network and extracting picture characteristic layer information; s4: carrying out feature enhancement and visual field domain promotion on the image feature layer information; s5: the picture feature layer information in S4 is decoded into the category of polyp and its position information.
Referring to fig. 2, at S1, the backbone network includes but is not limited to: alexnet, VGGnet, Shufflenet, densenert, Squeezenet, ResNet; in the first embodiment, a convolutional neural network is constructed by using the pretrained VGG16 based on ImageNet, and a new network layer and module are additionally added on the convolutional neural network; the VGG16 convolutional neural network comprises 16 layers of convolution, each layer of which updates its weight parameters by back-propagation. The convolutional neural network is divided into different stages with step sizes of 4,8,16,32,64, bounded by max pool.
As the network deepens and the size of the feature map is reduced by half after each pooling, the number of channels is doubled continuously.
Moreover, the RELU activation function is used for each layer of convolution result, and the problem of gradient disappearance of the deep neural network can be effectively overcome and the overfitting problem can be relieved through the linearity of the segmented excitation function.
In S2, before extracting the image feature information, preprocessing the original color endoscopic image, including: and cutting the black edge of the input picture, adjusting the image size to be uniform in size and carrying out pixel mean value normalization.
This is because, in practical application, due to different endoscope devices, the output image resolution is also different, and in order to facilitate subsequent feature extraction, the present invention first scales the color endoscopic image to the image resolution of 320 × 320, and then performs pixel mean normalization on the image, so that each dimension of the data has zero mean and unit variance.
In S3, inputting the preprocessed color endoscope image into a convolutional neural network, and extracting picture characteristic layer information; inputting the preprocessed pictures into a convolutional neural network for convolution operation, and obtaining the outputs of picture feature layer information with the sizes of 80x80,40x40,20x20,10x10 and 5x5 corresponding to backbone networks Conv3_3, Conv4_3, Conv5_3, Conv6_2 and Conv7_2 respectively.
In S4, in order to increase the detection rate of small objects, feature enhancement is performed using the backbone networks Conv3_3, Conv4_3, Conv5_3, Conv6_2, and Conv7_ 2.
In order to solve the problems that after deep layers are subjected to repeated pooling, semantic information is increased but is insensitive to small objects, shallow layer semantic information is few, and accuracy is low, the method uses a top-down method of a characteristic pyramid structure to enhance the semantic information of a shallow layer characteristic layer, and accuracy of the small objects is improved.
The method comprises the following specific steps:
in order to extract k intermediate layers from the convolutional neural network, k is 5 in this embodiment; when the value [0, k-1] of i is taken as the characteristic layer Si, the characteristic layer Si +1 is subjected to smoothing and dimensionality reduction operation by using a convolution layer with a convolution kernel of 1x1, then the transposed convolution with the convolution kernel of 2x2 and the step length of 2 is used for up-sampling, and the result is added with the characteristic layer Si to obtain an enhanced characteristic Ei;
the formula is as follows: ei＝Si+deconv(smooth(Si+1))
Where deconv () is a transposed convolution and smooth () is a convolution operation of 1x 1.
Next, in order to further improve the view field under the condition of saving computing resources, the feature-enhanced Ei is input to a view field improving module, please refer to fig. 2, which includes three branches, each branch processes 1/3 channels respectively and includes 1,2,3 sparse convolutions with 3 convolution kernels of 3 × 3 step length 1 and sparse convolution of 3, and the output of the module is the cascade of convolution results of each branch at the channel level.
In operation, by using two or three 3x3 convolutions, the effect can reach a 5x5 or 7x7 convolution visual field and use less computing resources, and the feature layer after the step contains more visual field, so that semantic information is increased, and the accuracy and the recall rate of small objects are improved.
In S5, decoding the extracted picture feature layer information: and decoding the digital characteristic information extracted by the network into the polyp category and the polyp position information, and updating the network weight by using a loss function in cooperation with back propagation in the training process of the convolutional neural network.
For a bounding box regression feature layer, the output of a feature point x, y is predicted to be
Figure BDA0002738892920000091
Wherein the content of the first and second substances,
Figure BDA0002738892920000092
decoded to obtain the predicted object position cx，cy，w，h]，cx，cyThe values of x and y are the central points of the object frame, and w and h are the width and height of the object frame.
In addition, when the convolutional neural network is actually tested and used, the convolutional neural network needs to be trained for many times, so that the convolutional neural network is fixed, and the test accuracy of the convolutional neural network meets the actual use requirement.
In this embodiment one, the training method of the convolutional neural network is as follows:
(ii) constructing a convolutional neural network based on ImageNet's pre-trained VGG 16.
And secondly, preprocessing the color endoscope image.
Inputting the preprocessed color endoscope image into a convolution neural network to extract the characteristic layer information of the image.
And fourthly, performing feature enhancement and visual field promotion on the image feature information extracted by the convolutional neural network.
The training stage gives different labels to the feature points of different feature layers according to the size of the target polyp, the labels are used for the subsequent calculation of function calculation and regression, and the method is different from the traditional method of IoU (overlapping rate) calculation based on an anchor frame, and the method uses a more concise means to calculate the labels.
We propose a method for determining positive and negative samples based on the region of the target bounding box, which saves the extra hyper-parameters and computational load due to the anchor box, and to represent this process more intuitively, we assume a polyp target bounding box b ═ (c)x，cy，w，h)，(cx，cy) Is the center point of the bounding box, and (w, h) is width and height; allocate it to scale to s by sizelCharacteristic layer K oflThe feature layer contains m2A feature point, where m is the feature layer side length, set
Figure BDA0002738892920000101
Indicating the position of the feature point in the original image, where xi＝sl×i，yj＝sl×j。
The positive sample region is
Figure BDA0002738892920000102
The non-positive sample is
Figure BDA0002738892920000103
Wherein
Figure BDA0002738892920000104
Area size according to epsilonp＝0.75，εnScaling is performed as 1.25.
When the feature point is in the area of the positive sample, determining as the positive sample; when the feature point is between the regions of the positive sample and the non-positive sample, determining to ignore the sample; when the feature point is outside the region of the non-positive sample, determining as a negative sample; a practical example is shown in fig. 3, where the second solid frame is a non-positive sample region, the first solid frame is a positive sample region, and the squares are feature points.
And sixthly, fitting polyp targets with different sizes for the feature layers with different scaling in a training stage, wherein the training stage comprises polyp target distribution of feature points and polyp target screening of the current layer.
In determining positive and negative samples, different polyp object sizes are assigned to different feature layers, but feature points of some adjacent feature layers, especially those located at the center of the object, have sufficient semantic information to predict the object position.
In order to use the characteristic points, a scaling formula is designed, so that the non-positive sample region belonging to a certain layer can be scaled to different characteristic layers according to the distance size, and the recall rate is improved.
Suppose that object b belongs to a feature layer i, i ∈ [0, k-1]]And d ═ i-l | is the scaling value of the distance between the current characteristic layer l and the optimal characteristic layer: delta1＝max(Lcosin(λ，d，k)，0)；
Figure BDA0002738892920000111
The new non-positive samples for the current layer are therefore:
Figure BDA0002738892920000112
wherein, deltalThe scaled non-positive sample proportion is calculated.
The training stage can generate different distance heat force values for different polyp targets for different feature layers to assist the feature points in judging the polyp target at the current position, and the specific formula is as follows:
Figure BDA0002738892920000113
wherein:
Figure BDA0002738892920000114
Figure BDA0002738892920000115
min (x, y) is the minimum value of x and y, and max (x, y) is the maximum value of x and y in the same way.
Wherein left, right, top and bottom are respectively expressed as the distance from the feature point to the left frame, the right frame, the upper frame and the lower frame of the sample frame;
the generated thermodynamic diagram uses binary cross entry as a loss function in a training stage, and the prediction stage and the object class prediction value obtained in the step 8 are subjected to point multiplication.
And in the training stage, different weights are generated by using a formula when the classification loss function is calculated for different characteristic layers for different polyp targets.
When the characteristic point is closer to the central point of the target object boundary frame, the loss of the characteristic point is considered to be more important to a detection result in prediction, and therefore a Gaussian distribution formula is designed to give different weights to different distances in loss calculation
Figure BDA0002738892920000116
The concrete formula is as follows:
Figure BDA0002738892920000121
Figure BDA0002738892920000122
wherein alpha is 1, cbIs the target object position, cpIs the position of the feature point.
Ninthly, calculating a bounding box regression loss L by using a smooth L1 loss function as a fitting loss function for the position sizes of the feature points in the training phaselocAnd encoding the feature points of the bounding box by using a function:
Figure BDA0002738892920000123
Figure BDA0002738892920000124
Figure BDA0002738892920000125
wherein the content of the first and second substances,
Figure BDA0002738892920000126
for regression objective, β is 0.45.
The r training phase uses the following loss function for the classification of feature points:
LF(pt)＝-αt(1-pt)γlog(pt)；
Figure BDA0002738892920000127
wherein L isF(pt) Is focal length. L isCEIs cross entry scoreA class loss function; l isclsIs the final classification loss function.
In practical use, the detection effect graphs of fig. 4 and 5 can accurately locate the position of polyp, fig. 6 is a graph of the recognition effect of the present invention, and it can be seen that when the recognition accuracy of the polyp image recognition system of the present invention is 89.8%, the recall rate is 83.87%.
The method for detecting the polyp of the large intestine endoscope based on the deep convolutional network has the advantages that the operation is convenient, the reasoning speed and the sensitivity to the polyp image can be improved, the polyp position can be rapidly positioned in the endoscope image, the inconsistency of label setting caused by different rotation angles in the same polyp object under the endoscope image is avoided, the actual use requirements are met, and the method has high practicability and popularization value.
The above is only a specific application example of the present invention, and the protection scope of the present invention is not limited in any way. All the technical solutions formed by equivalent transformation or equivalent replacement fall within the protection scope of the present invention.

Claims (9)

1. A large intestine endoscope polyp detection method based on a deep convolutional network is characterized by comprising the following steps:
s1: constructing a convolutional neural network, wherein the convolutional neural network is pre-trained in ImageNet by a backbone network and then trained by a training image data set containing polyp categories and position information;
s2: preprocessing a color endoscope image;
s3: inputting the preprocessed color endoscope image into a convolutional neural network and extracting picture characteristic layer information;
s4: carrying out feature enhancement and visual field domain promotion on the image feature layer information;
s5: the picture feature layer information is decoded into the category of polyps and its location information.
2. The deep convolutional network-based large intestine endoscope polyp detection method according to claim 1, characterized in that: the convolutional neural network comprises sixteen layers of convolution, and each layer of network updates the weight parameters thereof through back propagation;
the convolutional neural network is divided into different stages with step sizes of 4,8,16,32,64, bounded by max pool.
3. The deep convolutional network-based large intestine endoscope polyp detection method as claimed in claim 1, wherein the step of preprocessing the color endoscope image is as follows:
cutting a black edge of the color endoscope picture;
the image size is adjusted to a uniform size and pixel mean normalization is performed.
4. The deep convolutional network-based large intestine endoscope polyp detection method according to claim 1, characterized in that: in S3, the preprocessed color endoscope image is input into a convolutional neural network for convolution operation, so as to obtain image feature layer information with sizes of 80x80,40x40,20x20,10x10, and 5x5, which respectively correspond to outputs with step sizes of 4,8,16,32, and 64 in the convolutional neural network.
5. The deep convolutional network-based large intestine endoscope polyp detection method as claimed in claim 4, wherein the step in S4 is as follows:
a, performing feature enhancement on the preprocessed picture feature layer information;
b: and inputting the image feature layer information subjected to feature enhancement into a visual field lifting module to obtain the image feature layer information subjected to visual field lifting.
6. The deep convolutional network-based large intestine endoscope polyp detection method as claimed in claim 5, wherein said feature enhancement processing method is as follows:
extracting k intermediate layers from a convolutional neural network, wherein k is 5;
secondly, when feature fusion is carried out on the feature layer Si, wherein i is 0-k-1, firstly, smoothing and dimensionality reduction operation is carried out on the feature layer Si +1 by using a convolution layer with a convolution kernel of 1x 1;
thirdly, performing upsampling by using the transposed convolution with the convolution kernel of 2x2 and the step length of 2, and adding the result and the characteristic layer Si to obtain an enhanced characteristic Ei, wherein the formula is as follows:
Ei＝Si+deconv(smooth(Si+1))；
where deconv () is a transposed convolution and smooth () is a convolution operation of 1x 1.
7. The deep convolutional network-based large intestine endoscope polyp detection method according to claim 6, characterized in that: the horizon lifting module comprises three branches, each branch is used for processing the channel number of 1/3 and comprises 1,2 and 3 sparse convolutions with convolution kernels of 3x3 and step length of 1 and sparse convolution of 3; and after the picture characteristic layer information enters the visual field lifting module, outputting the convolution result of each branch in the cascade connection of the channel level.
8. The deep convolutional network-based large intestine endoscope polyp detection method as claimed in claim 1, wherein the decoding process in S5 is as follows:
for a bounding box regression feature layer, the output of a feature point x, y is predicted to be
Figure FDA0002738892910000021
Wherein the content of the first and second substances,
Figure FDA0002738892910000022
Figure FDA0002738892910000023
decoded to obtain the predicted object position cx，cy，w，h]Wherein c isx，cyThe central point x, y value of the object frame, w, h are the width and height of the object frame, and then the object class information is combinedDeriving polyp classification and location information [ s, cx, xy, w, h]；
Where s is the object class.
9. The deep convolutional network based large intestine endoscope polyp detection method as claimed in claim 7, wherein the training method of the convolutional neural network is as follows:
firstly, pre-training is carried out on ImageNet by a backbone network, and then training is carried out by a training image data set containing polyp types and position information to obtain a convolutional neural network;
preprocessing a color endoscope image;
inputting the preprocessed color endoscope image into a convolution neural network to extract image characteristic layer information;
fourthly, performing feature enhancement and visual field promotion on the image feature information extracted by the convolutional neural network;
giving different labels to the feature points of different feature layers according to the size of the target polyp;
suppose a polyp target bounding box b ═ cx，cy，w，h)，(cx，cy) Is the center point of the bounding box, and (w, h) is width and height; it is allocated to the scaling step size slCharacteristic layer K oflThe feature layer contains m2A feature point, where m is the feature layer side length, set
Figure FDA0002738892910000031
Indicating the position of the feature point in the original image, where xi＝s1×i，yj＝sl×j；
The positive sample region is
Figure FDA0002738892910000032
The non-positive sample is
Figure FDA0002738892910000033
Wherein
Figure FDA0002738892910000034
Figure FDA0002738892910000035
Area size according to epsilonp＝0.75，εnScaling is performed as 1.25;
when the feature point is in the area of the positive sample, determining as the positive sample; when the feature point is between the regions of the positive sample and the non-positive sample, determining to ignore the sample; when the feature point is outside the region of the non-positive sample, determining as a negative sample;
sixthly, polyp target distribution of the characteristic points and polyp target screening of the current layer are carried out;
when positive and negative samples are judged, assuming that an object b belongs to a feature layer i, i belongs to [0, k-1], and d is a value of the distance scaling size between the current feature layer and the optimal feature layer, wherein the value is as follows:
δ1＝max(Lcosin(λ，d，k)，0)；
Figure FDA0002738892910000041
the new non-positive samples for the current layer are therefore:
Figure FDA0002738892910000042
wherein, deltalThe calculated scaling non-positive sample proportion;
and for different polyp targets, different distance heat force values can be generated for different feature layers to assist the determination of the polyp target at the current position by the feature points, and the specific formula is as follows:
Figure FDA0002738892910000043
wherein:
Figure FDA0002738892910000044
Figure FDA0002738892910000045
min (x, y) is the minimum value of x and y, and max (x, y) is the maximum value of x and y in the same way;
wherein left, right, top and bottom are respectively expressed as the distance from the feature point to the left frame, the right frame, the upper frame and the lower frame of the sample frame;
using binary cross entry as a loss function in the training phase of the generated thermodynamic diagram, and performing point multiplication on the generated thermodynamic diagram and the object class predicted value obtained in the step 8;
in the training stage, different weights are generated by using a formula when classification loss functions are calculated for different characteristic layers for different polyp targets; different distances are given different weights in calculating the loss by a Gaussian distribution formula
Figure FDA0002738892910000046
The concrete formula is as follows:
Figure FDA0002738892910000047
Figure FDA0002738892910000048
wherein alpha is 1, cbIs the target object position, cpIs the position of the characteristic point;
ninthly, calculating a bounding box regression loss L by using a smooth L1 loss function as a fitting loss function for the position sizes of the feature points in the training phaselocAnd encoding the feature points of the bounding box by using a function:
Figure FDA0002738892910000051
Figure FDA0002738892910000052
Figure FDA0002738892910000053
wherein the content of the first and second substances,
Figure FDA0002738892910000054
as a regression target, β is 0.45;
the r training phase uses the following loss function for the classification of feature points:
LF(pt)＝-αt(1-pt)γlog(pt)；
Figure FDA0002738892910000055
wherein L isF(pt) Is focal length. L isCESorting the loss function for cross entry, LclsIs the final classification loss function.
CN202011143365.1A 2020-10-23 2020-10-23 Large intestine endoscope polyp detection method based on deep convolutional network Pending CN112200795A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011143365.1A CN112200795A (en) 2020-10-23 2020-10-23 Large intestine endoscope polyp detection method based on deep convolutional network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011143365.1A CN112200795A (en) 2020-10-23 2020-10-23 Large intestine endoscope polyp detection method based on deep convolutional network

Publications (1)

Publication Number Publication Date
CN112200795A true CN112200795A (en) 2021-01-08

Family

ID=74010930

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011143365.1A Pending CN112200795A (en) 2020-10-23 2020-10-23 Large intestine endoscope polyp detection method based on deep convolutional network

Country Status (1)

Country Link
CN (1) CN112200795A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967209A (en) * 2021-04-23 2021-06-15 上海大学 Endoscope image blood vessel texture enhancement method based on multiple sampling
CN113486990A (en) * 2021-09-06 2021-10-08 北京字节跳动网络技术有限公司 Training method of endoscope image classification model, image classification method and device
CN113837989A (en) * 2021-05-25 2021-12-24 苏州慧维智能医疗科技有限公司 Large intestine endoscope polyp detection and pathological classification method based on anchor-free frame
CN114419366A (en) * 2021-12-30 2022-04-29 福州大学 Method and system for quickly identifying adulteration of pepper powder based on deep learning

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110189264A (en) * 2019-05-05 2019-08-30 深圳市华星光电技术有限公司 Image processing method
CN110363751A (en) * 2019-07-01 2019-10-22 浙江大学 A kind of big enteroscope polyp detection method based on generation collaborative network
CN111027547A (en) * 2019-12-06 2020-04-17 南京大学 Automatic detection method for multi-scale polymorphic target in two-dimensional image
CN111383214A (en) * 2020-03-10 2020-07-07 苏州慧维智能医疗科技有限公司 Real-time endoscope enteroscope polyp detection system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110189264A (en) * 2019-05-05 2019-08-30 深圳市华星光电技术有限公司 Image processing method
CN110363751A (en) * 2019-07-01 2019-10-22 浙江大学 A kind of big enteroscope polyp detection method based on generation collaborative network
CN111027547A (en) * 2019-12-06 2020-04-17 南京大学 Automatic detection method for multi-scale polymorphic target in two-dimensional image
CN111383214A (en) * 2020-03-10 2020-07-07 苏州慧维智能医疗科技有限公司 Real-time endoscope enteroscope polyp detection system

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
DECHUN WANG等: "AFP-Net: Realtime Anchor-Free Polyp Detection in Colonoscopy", 《2019 IEEE 31ST INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE (ICTAI)》 *
H. LAW等: "Cornernet: Detecting objects as paired keypoints", 《CORR》 *
PENGFEI ZHANG等: "An efficient spatial-temporal polyp detection framework for colonoscopy video", 《2019 IEEE 31ST INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE(ICTAI)》 *
周立旺等: "多阶段优化的小目标聚焦检测", 《图学学报》 *
季江舟: "弱监督学习在计算机视觉中的应用研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 *
陈明辉等: "《遥感技术在城市发展与规划中的综合应用研究与实践》", 30 November 2017 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967209A (en) * 2021-04-23 2021-06-15 上海大学 Endoscope image blood vessel texture enhancement method based on multiple sampling
CN113837989A (en) * 2021-05-25 2021-12-24 苏州慧维智能医疗科技有限公司 Large intestine endoscope polyp detection and pathological classification method based on anchor-free frame
WO2022247486A1 (en) * 2021-05-25 2022-12-01 苏州慧维智能医疗科技有限公司 Anchor-free polyp colonoscopy and pathological classification method
US11954857B2 (en) 2021-05-25 2024-04-09 Highwise Co, Ltd. Method for detection and pathological classification of polyps via colonoscopy based on anchor-free technique
CN113486990A (en) * 2021-09-06 2021-10-08 北京字节跳动网络技术有限公司 Training method of endoscope image classification model, image classification method and device
CN113486990B (en) * 2021-09-06 2021-12-21 北京字节跳动网络技术有限公司 Training method of endoscope image classification model, image classification method and device
CN114419366A (en) * 2021-12-30 2022-04-29 福州大学 Method and system for quickly identifying adulteration of pepper powder based on deep learning

Similar Documents

Publication Publication Date Title
Ali et al. Structural crack detection using deep convolutional neural networks
Li et al. Image-based concrete crack detection using convolutional neural network and exhaustive search technique
CN112200795A (en) Large intestine endoscope polyp detection method based on deep convolutional network
Li et al. Deep learning based gastric cancer identification
CN109284670B (en) Pedestrian detection method and device based on multi-scale attention mechanism
WO2023077404A1 (en) Defect detection method, apparatus and system
Xu et al. Automated analysis and classification of melanocytic tumor on skin whole slide images
CN113592845A (en) Defect detection method and device for battery coating and storage medium
CN110751154B (en) Complex environment multi-shape text detection method based on pixel-level segmentation
AU3585495A (en) Biological analysis system self-calibration apparatus
Wan et al. Ceramic tile surface defect detection based on deep learning
Ngugi et al. A new approach to learning and recognizing leaf diseases from individual lesions using convolutional neural networks
Vani et al. Detection and Classification of Invasive Ductal Carcinoma using Artificial Intelligence
Zhu et al. HMFCA-Net: Hierarchical multi-frequency based Channel attention net for mobile phone surface defect detection
CN116596875A (en) Wafer defect detection method and device, electronic equipment and storage medium
Lin et al. An antagonistic training algorithm for TFT-LCD module mura defect detection
CN115017931A (en) Method and system for extracting QR codes in batches in real time
Kukreja Segmentation and Contour Detection for handwritten mathematical expressions using OpenCV
Parhizkar et al. Car detection and damage segmentation in the real scene using a deep learning approach
Naiemi et al. Scene text detection using enhanced extremal region and convolutional neural network
Gong et al. FRCNN-AA-CIF: An automatic detection model of colon polyps based on attention awareness and context information fusion
Al-Huda et al. Asymmetric dual-decoder-U-Net for pavement crack semantic segmentation
Avots et al. A new kernel development algorithm for edge detection using singular value ratios
Chowdhury et al. Scene text detection using sparse stroke information and MLP
CN115861226A (en) Method for intelligently identifying surface defects by using deep neural network based on characteristic value gradient change

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210108