WO2022271129A1

WO2022271129A1 - Deep learning-based decision support system for real-time automatic polyp detection

Info

Publication number: WO2022271129A1
Application number: PCT/TR2022/050592
Authority: WO
Inventors: Derviş KARABOĞA; Bahriye AKAY; Alper BAŞTÜRK; Özkan Ufuk NALBANTOĞLU; Ahmet KARAMAN; İshak PAÇAL
Original assignee: T.C. Erciyes Universitesi
Priority date: 2021-06-24
Filing date: 2022-06-15
Publication date: 2022-12-29

Abstract

The invention relates to a computer-aided diagnostic system and method for real-time automatic polyp detection by deep learning methods.

Description

DEEP LEARNING-BASED DECISION SUPPORT SYSTEM FOR REAL-TIME

AUTOMATIC POLYP DETECTION

TECHNICAL FIELD

STATE OF THE ART

In the state of the art, colorectal cancer is the third most common cancer diagnosed in both men and women in Turkey and the United States. A National Cancer Institute (NIH) statistic shows that about 53,200 people will die of colorectal cancer in the US in 2020, with 104,610 new cases of colon cancer and 43,340 new cases of rectal cancer being diagnosed. While the risk of cancer in men is approximately one in 22, this rate is one in 24 in women.

Colorectal (CRC) cancer begins as a small, benign glandular tissue growth in the inner lining of the colon, called adenomatous polyps (adenomas). These structures, polyps, can turn into a malignant tumours with high potential over time. These structures, which turn into malignant tumours, cause colon cancer. Over time, it may spread to various tissues and organs depending on the type of cancer.

Early detection and prevention of colorectal cancer are usually done through regular screening. Polyps reaching a certain structure and size can be detected by colonoscopy scanning, which is considered the gold standard. The detected polyps can be easily removed by the specialist without turning into cancer. During the scan, specialist physicians perform a real-time scan with a colonoscopy device, a long flexible tube with a small camera mounted. This device allows regular scanning from the rectum to the colon. The only disadvantage of the colonoscopy method is that it may pose a risk of tearing in the rectum wall or colon in some cases. Previous approaches to polyp detection were generally based on manual features such as texture, colour and shape. Recently, significant success has been achieved in automatic polyp detection with deep learning methods for the diagnosis of CRC, and time and cost savings have been achieved by using it as an adjunct to the specialist physician.

Polyps of different sizes, shapes and textures make it difficult to detect. In addition, the images have a noisy background that includes bleeding and endoluminal folds. This reduces the accuracy of the detection process. For this, the deep learning model to be taught should be trained with an effective and sufficient dataset.

In the early diagnosis of cancers, computer-aided algorithms have been used since the 1960s. Recently, it has made it possible for deep learning methods to play an important role in the early diagnosis of cancers, as they offer high performances in medical images and data. Real time and high-precision systems emerged with the developing deep learning algorithms and equipment. These achievements are the ability to learn that underlies deep learning algorithms.

A specialist physician can transfer their experience over a long period to a deep learning algorithm within hours. In addition, the experience of many specialists in different fields can be presented in these algorithms. This is to label the data to be given with minimum error as a result of joint studies by specialist physicians and computer scientists. Thus, it is possible to use computer-aided algorithms more effectively and powerfully in the field of medicine.

In object detection, deep learning algorithms are used in many areas such as medicine, defence, autonomous tools and finance due to their high performance. Therefore, it is necessary to look at the latest object detection algorithms in the literature if a detection process is to be performed in general. After these algorithms are found, they can effectively be used after meeting the application requirements in the desired area locally.

Considering the studies in the literature, deep learning architectures are used extensively in the medical field. Often, the main disruptions are that publicly available colorectal cancer datasets contain very little data and thus, trained architectures are inadequate. For example, the ENDOANGEL system is used as an auxiliary system for specialist physicians. However, the fact that it is inadequate in real-time detection and does not include an advanced object detection algorithm makes it weak. In a similar study, the YOLOv3 model was trained with a large data set. The average performance of the system where 50,000 data is used for education is around 92 percent. A more advanced version of YOLOv3, the YOLOv4 architecture is faster and has higher performance. None of the studies that detect intestinal diseases with deep learning methods uses the artificial bee colony algorithm in hyperparameter optimization. In addition, these applications have disadvantages in making real-time detection. Most algorithms have also been inadequate in accuracy and precision.

The studies included in the state of the art are as follows;

In the study [7] in the references, the endoscope image processing method, endoscope image processing device, electronic apparatus and storage medium are included. In this study, popular object detection algorithms such as YOLOv3, RetinaNet and Faster R-CNN were used. The system, which allows you to select these 3 models in its works, can only perform semi-real-time detection with YOLOv3 and RetinaNET. Non-synchronous parallel frame diagram is not recommended for real-time detection. While the conventional solution (Detector solution) in their study gives an 84.7 ms response time, the system they propose; the unconjugated parallel frame diagram responds at 24.4 ms. Generally, the minimum speed value allowed for real-time detection should be 30 FPS (frame per second) or greater. YOLOv4 offers 82 FPS for 416x416 image resolution with an RTX2080ti graphics card. It is very successful in real-time detection with a value of 53 FPS for 608x608 image resolution. As a delay, YOLOv4 is much faster than YOLOv3 with less than 10ms response for 416x416 resolution.

US2018/0225820 proposes methods, systems and media for monitoring the colonoscopic video quality and detecting polyps. In this study, polyp detection was performed by using CNN architecture and various image processing methods together. The metrics obtained and the operating speed are not sufficient for real-time detection.

The study [8] in the references is an intelligent auxiliary system for polyp detection and an old YOLO version was used in the relevant study. The performance of the proposed system is low (FLO.915) and lags in terms of speed. In the article [1] in the references, a system that can detect the location of polyps in real-time with 96% accuracy in colonoscopy scanning is proposed. However, in this study, a special limited data set was used and it has low performance and speed since it uses the first versions of YOLO.

The article [2] in the references deals with real-time polyp detection, localization, and segmentation in colonoscopy using deep learning, and the YOLOv4 version was trained and tested on the Kvasir dataset. The limited data set was studied and hyperparameter optimization was not performed using the artificial bee colony algorithm as in the proposed invention.

In the study [3] in the references, the first version of the YOLO algorithm was used, and the proposed system is insufficient in terms of speed and metric values.

SSD and YOLOv3 algorithms were used in the study [4] in the references, but they are insufficient in terms of performance metrics and speed.

The ResYOLO version was used in the study [5] in the references. This version is far behind YOLOv4 in terms of speed and performance.

The model proposed in the study [6] in the references was trained with a large data set (50,000 samples). The performance obtained in this study using the YOLOv3 object detection algorithm is lower than the performance of YOLOv4 and remains slow in terms of speed.

BRIEF DESCRIPTION OF THE INVENTION

The said invention relates to a computer-aided diagnostic system and method for real-time automatic polyp detection by deep learning methods to eliminate the above-mentioned disadvantages and to bring new advantages to the related technical field.

Especially due to the rapid development of systems in computer science and the fact that the previous systems are becoming out of date day by day, the structure to be presented is more successful than the current approaches by providing real-time detection, high accuracy and precision.

This algorithm uses data enhancement techniques such as scaling, mosaic, cropping, flipping, perspective, confusion, and colour space transformation to improve educational success both before and during training. However, to increase the performance of the model, the optimization process is continued until the maximum number of cycles determined by the user is reached by using the Artificial Bee Colony algorithm [9] to maximize the FI value of hyperparameters such as data increase parameters and learning rate, momentum, weight decay, and anchor.

In this aspect, the proposed approach brings innovation to the current technique and allows access to a more successful model. Pre-processes performed during or before the training allow the functional structure that will enable the models to learn collectively after the training to show higher performance.

The performance of the system can be improved by training the weights in the proposed system on the new datasets. In case of any wrong polyp detection, with retrospective training, this polyp sets an example for later situations, allowing the algorithm to learn this error and to perform more on new different data. In this way, a continuous education model can be created and easily adapted to new updates and data thanks to its flexible structure.

The invention contributes to the help of physicians in many different ways with over 95 percent precision, accuracy, and recall rate of automatic polyp detection in colonoscopy images. In detecting the small polyps that physicians suffer from, it has the potential to perform many procedures such as warning in case of polyps that are overlooked depending on the colonoscopy device. In addition, if sufficient data is provided, it is possible that this system, which is capable of continuous learning, can exceed the detection level of physicians. Such a system plays a major role in detecting polyps, which are the precursors of colon cancer, more successfully. Using the YOLOv4 algorithm together with an advanced optimization algorithm such as the artificial bee colony algorithm distinguishes the invention from similar approaches available. The real-time polyp detection and performance rate offered by the proposed method again cause it to differentiate from other existing studies. Although the studies in the state of the art work in real-time, the invention in question is more successful than the others with a delay of less than 10 ms and an image resolution of 82 FPS (416x416 on the RTX2080ti graphics card).

The system proposed in the invention has been trained in public data and has reached metric values over 95 percent with a data set and a special data set that have never participated in training before. As a result, the structure in the proposed system is ahead of the studies in the state of the art, both in terms of speed and metric values.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures and descriptions of the figures are listed below for a better understanding of the invention.

Figure 1 Flowchart of the method

DETAILED DESCRIPTION OF THE INVENTION

In this detailed description, the novelty of the invention is explained with examples that do not have any limiting effect only for a better understanding of the subject.

The said invention relates to a computer-aided diagnostic system and method for real-time automatic polyp detection by deep learning methods.

The proposed system is based on structures such as data pre-processing, object detection algorithm, hyperparameter optimization, ensemble learning and testing.

Data pre-processing consists of removing unwanted parts in the image and increasing the data set size by using image processing methods on the existing data set. Several data pre processing are available to move colonoscopy images to a deep learning environment. The most important of these is that there is enough labelled data for the training dataset. For this purpose, software was developed for the specialist physician to mark polyp regions on the videos recorded from the colonoscopy device. Polyps marked by a specialist physician are compressed in JPG format and recorded in the database. This data is then used for the training and testing of the model.

Data augmentation is a pre-processing where the data is insufficient or unbalanced. Although YOLOv4 performs data augmentation (scaling, mosaic, cropping, turning, perspective, confusion, colour space transformation) in its structure, it may be insufficient in special areas. Therefore, the number of data was increased by using different data augmentation methods (saturation, free rotation, derivation from neighbouring frames in the video, mosaic, variants in different histograms, zoom, translation, and colour scale change).

As an object detection algorithm, the YOLOv4 algorithm [10] was used to determine the positions of polyp regions. The most important feature that distinguishes the YOLOv4 algorithm from other algorithms is its ability to detect in real-time, that is, the frames per second (FPS) value is high. In the proposed system, the YOLOv4 algorithm was used together with optimization algorithms.

As is known, a good optimization of hyperparameters significantly affects the performance of the current deep learning model. With YOLOv4, the model is optimized in the best way with the artificial bee colony algorithm.

With the use of different deep learning backbones, the weights obtained from the deep learning algorithms trained are put into a community learning process to increase the performance, and while the performance is further increased, speed is not compromised.

The system in the invention consists of a data set unit, educational unit, test unit and CAD diagnostic unit.

Based on the information above, the invention is a method that provides real-time automatic polyp detection with deep learning methods, characterized in that it comprises the following process steps:

• In the data set unit; - Collecting the data labelled with a desktop or cloud-based computer program (application) to record the data classes from a real-time colonoscopy device and the data contained in existing databases,

- Keeping the received data in image pre-processing (softening filtering, Gaussian filter, median filter, data enhancement techniques (saturation, scaling, cropping, image mixing, free rotation, derivation from neighbouring frames in the video, mosaic, variants in different histograms, zoom, rotation, colour scale change)),

• In the training unit;

- In the hyperparameter optimization unit; applying the artificial bee colony algorithm and storage of the best parameters (determination of the average precision and loss function of the parameters used in the learning rate, momentum, weight reduction, pre-training epoch number, box, loss function decrease, anchor parameters and data increase methods (harmonic average of precision and recall: FI) by worker, observer and explorer bee steps in a way that will optimize the maximum number of cycles determined by the user (determining the parameters that will give the performance metric measurements with the highest numerical value)),

- Re-feeding the parameters corresponding to the lowest value (best loss) obtained as a result of hyperparameter optimization and obtained until reaching the maximum number of cycles determined by the user and producing the highest FI values in the same process to the YOLOv4 model,

- Using the architecture (CSPNet+DarkNET53=CSPDarkNET53) [10], which consists of a combination of a Cross Stage Partial neural network and a 53-layered YOLOv4 architecture (DarkNET53) for the Backbone in the convolutional neural network (CNN) architecture YOLO's feature extraction step by feeding with self -regulated non-monotonic (Mish) as activation function, on the other hand, using the structure (SPP+PANET) [ll]consisting of the combination Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PANET) for neck,

- Using YOLOv3 Head [12] for the head in the detection section determining the coordinates of the polyp positions,

- Testing after the number of epochs (one round of training on the data set) given by the designer as a pre-defined,

- Storing the best models: storing the models that give the highest values according to performance metrics (Precision, Recall, Harmonic Average of Precision and Recall (FI), Accurate Prediction Rate (ACC)), • In the test unit;

- Loading models (model 1, model 2, model 3... model n) whose weights are stored at periods determined by the designer during the training,

- Testing each model with datasets not used during the training,

- Comparing the models according to performance metrics (Precision, Recall, FI, ACC),

- Examination of whether it has reached a targeted limit value based on performance metrics (Precision, Recall, FI, ACC),

- If the metrics examined in the previous step reach numerically smaller metric values than the performance metric values reached by the current technique, applying community learning that can produce numerically larger metric values than the performance metric values reached by the individual models by blending the information of the models with different precision and recall values using the model merging (Merge-NMS [13]) method,

- If the model performance reaches numerically higher values than the performance metric values reached by the current technique, determining the new model as a successful model and storing it on disk,

- Switching to Cad (Computer-Aided Diagnosis) diagnostic unit,

• In Cad diagnostic unit;

- Uploading the model formed before or after community learning and testing the data,

- Obtaining the approval of the physician,

- Performing the polyp detection,

- Storing the polyp regions obtained by real-time polyp detection.

REFERENCES

[1] G. ETrban, P. Tripathi, T. Alkayali, M. Mittal, F. Jalali, W. Karnes, P. Baldi, Deep

Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy, Gastroenterology. 155 (2018) 1069-1078. e8. https://doi.Org/10.1053/j.gastro.2018.06.037.

[2] 2. D. Jha, S. Ali, H.D. Johansen, D.D. Johansen, J. Rittscher, M.A. Riegler, P. Halvorsen, Real-Time Polyp Detection, Localisation and Segmentation in Colonoscopy Using Deep Learning, (2020) 1-12. http://arxiv.org/abs/2011.07631. [3] 3. Y. Zheng, R. Zhang, R. Yu, Y. Jiang, T.W.C. Mak, S.H. Wong, J.Y.W. Lau, C.C.Y. Poon, Localisation of Colorectal Polyps by Convolutional Neural Network Features Learnt from White Light and Narrow Band Endoscopic Images of Multiple Databases, Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS. 2018-July (2018) 4142-4145. https://doi.org/10.1109/EMBC.2018.8513337.

[4] M. Liu, J. Jiang, Z. Wang, Colonic Polyp Detection in Endoscopic Videos with Single Shot Detection Based Deep Convolutional Neural Network, IEEE Access. 7 (2019) 45058- 45066. https://doi.org/10.1109/ACCESS.2019.2921027.

[5] Zhang, Ruikai & Zheng, Yali & Poon, Carmen & Shen, Dinggang & Lau, James. (2018). Polyp Detection during Colonoscopy using a Regression-based Convolutional Neural Network with a Tracker. Pattern Recognition. 83. 10.1016/j.patcog.2018.05.026.

[6] 6. M. Misawa, S. eiKudo, Y. Mori, K. Hotta, K. Ohtsuka, T. Matsuda, S. Saito, T. Kudo, T. Baba, F. Ishida, H. Itoh, M. Oda, K. Mori, Development of a computer-aided detection

[9] Karaboga, D. (2010). Artificial bee colony algorithm scholarpedia, 5(3), 6915.

[10] Bochkovskiy, A., Wang, C-Y, Liao, H-Y, YOLOv4: Optimal Speed and Accuracy of Object Detection, 2020.

[11] K. He, X. Zhang, S. Ren and J. Sun, "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1904-1916, 1 Sept. 2015, doi: 10.1109/TPAML2015.2389824.

[12] Redmon, Joseph & Farhadi, Ali. (2018). YOLOv3: An Incremental Improvement., arXiv preprint arXiv : 1804.02767

[13] https://zenodo.Org/record/4679653#.YIW9HZAzYuU system for colonoscopy and a publicly accessible large colonoscopy video database (with video), Gastrointest. Endosc. (2020) 1-11. https://doi.Org/10.1016/j.gie.2020.07.060.

[7] Zijian et al., Endoscope image processing method, endoscope image processing device, electronic apparatus, and storage medium, 2020

[8] Xiaonan et al. Intelligent auxiliary system for intestinal polyp detection and identification, 2019.

Claims

1. A real-time automatic polyp detection method, characterized in that it comprises the following process steps:

• In the data set unit;

- Collecting the data labelled with a desktop or cloud-based computer program to record the data classes from a real-time colonoscopy device and the data contained in existing databases,

- Subjecting the received data to an image pre-processing,

• In the training unit,

- In the hyperparameter optimization unit, applying the artificial bee colony algorithm and determining and storing the parameters that will provide the performance metric measurements with the highest numerical value until reaching the maximum number of cycles determined by the user with worker, scout and explorer bee steps,

- Feeding the parameters, which are producing the harmonic average values of the best loss and the highest precision and recall in the same process corresponding to the lowest value obtained as a result of hyperparameter optimization and obtained until reaching the maximum number of cycles determined by the user, back into the YOLOv4 model,

- Using the structure consisting of spatial pyramid pooling and path aggregation network combination for the neck as YOLO's feature extraction step by feeding with self-regulated non-monotonic as activation function of the architecture consisting of cross-stage partial neural network and 53-layered YOLOv4 architecture for backbone in convolutional neural network architecture,

- Using the YOLOv3 head for the head in the detection section which determines the coordinates of polyp positions,

- Testing after the number of epochs pre-defined by the designer,

- Storing the models that give the highest values according to the performance metrics,

• In the test unit,

- Loading the models whose weights are stored at periods determined by the designer during the training,

- Testing each model with data sets which are not used during the training,

- Comparing the models according to performance metrics, - Examining whether it has reached a targeted limit value based on performance metrics,

- If the metrics examined in the previous step reach numerically smaller metric values than the performance metric values reached by the current technique, applying community learning that can produce numerically larger metric values than the performance metric values reached by individual models by blending the information of models with different precision and recall values by using the model combination method,

- Switching to a Computer-Aided Diagnostics unit,

• In the Computer-Aided Diagnostics unit;

- Obtaining the approval of the physician,

- Performing the polyp detection,

- Storing the polyp regions obtained by real-time polyp detection.

2. A method according to claim 1, characterized in that the pre-processing mentioned in the method are softening filtering, Gaussian filter, median filter, saturation from data increment techniques, scaling, cropping, image mixing, free rotation, derivation from neighbouring frames in the video, mosaic, variants in different histograms, zoom, rotation, colour scale change.

3. A method according to claim 1, characterized in that the parameters mentioned in the training unit are the average precision and harmonic average of recall of the parameters used in the learning rate, momentum, weight reduction, pre-training epoch number, box, loss function decrease, anchor parameters and data increase methods.

4. A method according to claim 1, characterized in that the performance metrics mentioned in the training unit are the precision, recall, harmonic average of precision and recall, and the correct prediction rate.

5. A method according to claim 1, characterized in that the performance metrics mentioned in the test unit are the precision, recall, harmonic average of precision and recall, and the correct prediction rate.