KR20220140924A - Deep-learning based colonoscopy image analysis method and using the same - Google Patents

Abstract

The present invention relates to a deep-learning-based colonoscopy image analysis method with increased accuracy of lesion diagnosis and an image analysis system using the same. The deep-learning-based colonoscopy image analysis system, according to one embodiment of the present invention, comprises: an endoscopy computer which receives an image taken by a colonoscopy device; a server which obtains the image transmitted to the endoscope computer through an application downloaded to the endoscope computer in a hooking manner, has a diagnosis algorithm that performs correction of the obtained image and diagnosis based on the corrected image through deep learning, and transmits diagnosis results derived through the corrected image and the diagnosis algorithm to the endoscope computer; and a display device which receives the corrected image and the diagnosis results from the endoscope computer and output the received corrected image and diagnosis results.

Description

Deep-learning-based colonoscopy image analysis method and image analysis system using the same

The present invention relates to a deep learning-based colonoscopy image analysis method with increased accuracy of lesion diagnosis and an image analysis system using the same.

Cells, which are the smallest units of the human body, divide, grow, and die according to the intracellular regulatory function when normal, and maintain cell number balance. However, for various reasons, a condition in which abnormal cells whose proliferation and inhibition are not controlled proliferate excessively, invade surrounding tissues, and destroy the structure and function of organs is called cancer.

In particular, the incidence of colorectal cancer is rapidly increasing every year due to the development of westernized dietary habits and diagnostic technology. More than 95% of colorectal cancers start with polyps (adenomas), small tumors in the colon. Therefore, early detection and removal of polyps through colonoscopy can prevent colorectal cancer.

However, there is a big difference in the diagnosis rate of polyps in colonoscopy depending on the skill level of the doctor. If artificial intelligence assists during colonoscopy, it is expected that the accuracy of polyp diagnosis will be greatly improved. A number of papers have reported accuracy exceeding the polyp diagnosis rate of experienced doctors using deep learning artificial intelligence programs.

Colorectal polyps are divided into neoplastic polyps (adenomas), which can progress to malignancy, and non-neoplastic polyps, which are unlikely to progress to malignancy. If a deep learning program can accurately distinguish between oncological polyps and non-neoplastic polyps, unnecessary polypectomy and associated complications can be avoided.

In addition, it is important to accurately measure the size of a polyp because the risk of developing colorectal cancer varies according to the size of the polyp. In particular, polyps with a size of 10 mm or more have a high risk of developing colorectal cancer, so it is necessary to accurately measure the polyp size.

However, in the existing method, the doctor visually measures the size of the polyp compared to the biopsy forceps of known thickness. Therefore, the accuracy could vary depending on the skill level of the reader or the location of the polyp.

In addition, there is a light reflection problem that is commonly encountered during endoscopy. Even when the photographed image is unclear due to light reflection caused by the mucus inside the large intestine, an image reading error may occur, which tends to increase the doctor's fatigue. .

Recently, a colonoscopy system with improved accuracy has been developed, including image correction functions (high-resolution endoscopes, narrowband endoscopes) and artificial intelligence endoscopes that automatically diagnose polyps. However, most of these colonoscopy are still expensive, and their use is limited because it is a burden to purchase in small and medium-sized hospitals.

In addition, in most cases, the computer used for the colonoscopy equipment does not have sufficient capacity to perform functions other than image storage and processing of the colonoscopy. Replacing the computer to use other programs to improve image fidelity can also be costly.

According to an embodiment of the present invention, there is provided a deep learning-based colonoscopy image analysis method and an image analysis system using the same for processing images taken in real time using window hooking and window overlay technology.

In addition, a deep learning-based colonoscopy image analysis method for correcting a light-reflected image using previously learned information and memorizing newly learned contents, and an image analysis system using the same are provided.

In addition, a deep learning-based colonoscopy image analysis method that provides an accurate size of a polyp using previously learned marker information and an image analysis system using the same are provided.

In addition, a deep learning-based colonoscopy image analysis method in which lesion information by a plurality of users is stored as big data and an image analysis system using the same are provided.

The deep learning-based colonoscopy image analysis system according to an embodiment of the present invention includes an endoscope computer that receives an image captured by a colonoscopy device, and the image transferred to the endoscope computer through an application downloaded to the endoscope computer. is equipped with a diagnostic algorithm that acquires the obtained image in a hooking method and performs a diagnosis based on the correction of the acquired image and the corrected image based on deep learning, and the corrected image and the diagnosis result derived through the diagnosis algorithm and a server for transmitting to the endoscope computer and a display device for receiving and outputting the corrected image and the diagnosis result from the endoscope computer.

In addition, the application is driven in a window overlay method.

In addition, information learned through deep learning is stored in an information storage unit provided in the server, and the information is shared with users of the application.

In addition, information learned through deep learning is stored in a cloud database and the information is shared with users of the application.

In addition, the image transmitted to the server is subjected to color correction and light reflection correction based on the deep-learning information.

In addition, it is determined that the image photographed by the endoscope device has light reflection for pixels each having an RGB level of 170 or more, and the server executes light reflection correction through deep learning.

In addition, the image transmitted to the server includes an image of a polyp, and a border line matching the shape of the polyp is formed through deep learning and displayed on the display device.

A deep learning-based colonoscopy image analysis method according to an embodiment of the present invention comprises the steps of: an image acquired by an endoscope device is hooked in an endoscope computer and transferred to a server;

Correcting the image delivered to the server delivered based on deep learning, diagnosing and learning about the lesion on the image, transmitting the corrected image and information about the diagnosed lesion to the endoscope computer, and the corrected image and the diagnosed lesion and displaying information on the lesion on a display device.

In addition, the method further includes the step of storing the corrected image and the information on the diagnosed lesion in a storage space open to the application user.

The method further includes forming a border line along the shape of a polyp present on the image transmitted to the server and displaying it on a display device.

According to the deep learning-based colonoscopy image analysis method and the image analysis system using the same according to an embodiment of the present invention, it is possible to operate without a manufacturer's API (Application Programming Interface) by fetching and processing images in a window hooking method.

In addition, it is possible to improve the reliability of diagnosis by correcting the image due to light reflection and accurately measuring the size of the polyp.

In addition, it is possible to implement an endoscopic apparatus with improved diagnostic accuracy at a low cost without the need to replace expensive equipment.

In addition, the information on the lesion collected by a plurality of users is converted into big data and opened so that all users can use it, so that the user can be provided with more accurate information about the lesion.

1 is a diagram illustrating a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.
2 is a diagram for explaining an image analysis method of a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.
3 is a view for explaining an image classification method of a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.
4 is a diagram illustrating a state in which the shape of a polyp is displayed by a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.

Hereinafter, a deep learning-based colonoscopy image analysis method and an image analysis system using the same according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.

Referring to FIG. 1 , a deep learning-based colonoscopy image analysis system 1 according to an embodiment of the present invention includes a server 10 storing deep learning learning information, a colonoscope entering the patient's body, and It consists of a connected endoscope computer 20 and a display device 30 for outputting the photographed image.

The server 10 includes an information storage unit 100 in which all information stored in advance in a supervised learning method and self-learning information through deep learning are stored, and a control unit 101 that performs a calculation operation using the received information. do. The server 10 may transmit/receive information about a photographed image to and from the endoscope computer 20 in which an application 50 to be described later is activated.

The endoscope computer 20 may be connected to the server 10 through wireless communication to transmit/receive information related to the photographed image.

The display device 30 may be connected to the endoscope computer 20 in a wired communication or wireless communication method to receive and output image information from the endoscope computer 20 .

When the colonoscope is drawn into the large intestine to photograph the interior of the large intestine, the photographed image information is transmitted to the endoscope computer, and the endoscope computer transmits the image information to the display device 30 . The user may check whether there is a lesion inside the large intestine while looking at the image displayed on the display device 30 .

In this case, the image taken through the colonoscopy may be distorted due to light reflection caused by the mucus inside the large intestine. In addition, even if a lesion exists due to a curved or wrinkled shape of the large intestine, the image may not be clearly displayed. Such image distortion prevents the user from making an accurate diagnosis of the lesion.

Therefore, for accurate diagnosis, the user connects and uses a set-top box for correcting an ambiguous image to the endoscope computer 20, or uses an endoscope computer with an image correction function. However, an endoscope computer with high-end image correction function is expensive, and it is also expensive to purchase an additional set-top box. Therefore, small and medium-sized hospitals, especially many hospitals in developing countries with relatively backward medical environments, may be reluctant to use high-end equipment because it is burdensome to incur additional costs.

The deep learning-based colonoscopy image analysis system 1 according to an embodiment of the present invention implements a high-accuracy diagnostic system at a low cost, and corrects the image in the endoscope computer 20 to enable accurate diagnosis of lesions. You can download and use an application (50, tentative name: ALEXIS1) that operates functions such as.

As an example, the application 50 may be directly sold to doctors and hospitals in a B2C (B2C) method. A user such as a doctor or a hospital may download the application 50 distributed by the developer to the endoscope computer 20 and activate the application 50 by inputting the pin number provided after purchasing the application 50 .

As another example, the application 50 is sold to existing companies that manufacture and sell the colonoscopy device in a B2B method, and the user may select and use it as an option of the colonoscopy device when purchasing the product.

A method for a user to purchase and activate the application 50 is not limited to the above description.

When the information on the photographed image is transmitted to the endoscope computer 20, the application 50 retrieves and processes the image in a window hooking and window overlay method.

Window hooking refers to commands, methods, techniques, or actions that intercept or intercept function calls, messages, and events occurring between software components in various computer programs such as operating systems or software.

The window overlay means loading another division of the program into a storage area in which a segment of the program, which became unnecessary at that point during program execution, was loaded. As a method that can be used to run additional programs when the computer capacity is limited, when a specific routine is not needed, the segment related to the routine is moved to an external memory, replaced with the necessary routine, and the same block in the internal memory is reused can do.

Therefore, when an additional program is to be driven and the specifications of the computer are not sufficient, the added program can be smoothly driven through the hooking method and the window overlay method.

In the present invention, an image captured by the colonoscopy device may be retrieved from the application 50 through a hooking method, the image may be corrected, and the corrected result may be output through the display device 30 . Since the driving of the application 50 is performed in a window overlay method, the operation of the application 50 can be smoothly performed even when the specifications of the endoscope computer 20 are not sufficient.

The application 50 has compatibility so that it can be downloaded to the existing endoscope computer 20 as well. When the application 50 is downloaded to the endoscope computer 20 and activated, the endoscope computer 20 provides information about the image taken by the application 50 and the application 50 performs deep learning and image processing. After analyzing the information on the image, the analyzed information is output through the display device 30 so that the user can check it.

The application 50 may perform correction for image distortion such as light reflection on the image taken through the colonoscopy device. Hereinafter, an image analysis method of the deep learning-based colonoscopy image analysis system 1 will be described.

2 is a diagram for explaining an image analysis method of a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention, and FIG. 3 is a deep learning-based colonoscopy image analysis according to an embodiment of the present invention. It is a diagram for explaining the image classification method of the system.

2 and 3, the deep learning-based colonoscopy image analysis system 1 according to an embodiment of the present invention performs image correction and classification of lesions based on the deep-learning information, and provides image information. can learn

The deep learning-based colonoscopy image analysis system 1 according to an embodiment of the present invention uses a deep learning technology to read a colon disease from a photographed image. Deep learning is a form of artificial intelligence developed from artificial neural networks (ANNs).

As shown in FIG. 3 , the artificial neural network may include an input layer, a hidden layer, and an output layer.

Each layer includes a plurality of nodes, and each layer is connected to the next layer. Nodes are connected to each other with weights.

An output from an arbitrary node belonging to the first hidden layer becomes an input value to at least one node belonging to the second hidden layer. In this case, the input of each node may be a value to which a weight is applied to the output of the node of the previous layer. A weight means the strength of a connection between nodes. The deep learning process can also be viewed as a process of finding an appropriate weight.

In order to better understand deep learning, looking at the well-known face recognition process, the computer distinguishes between bright and dark pixels from the input image according to the brightness of the pixels, distinguishes simple shapes such as borders and edges, and then Distinguish between form and object. Finally, the computer can figure out the shape that defines the human face. In this way, the feature specification (regulation of the human face shape) is finally obtained from the output layer through a plurality of hidden layers.

The information storage unit 100 of the server 1 may store the aspect of the lesion, data for learning the deep neural network, weights constituting the structure of the deep neural network, and bias.

When the interior of the large intestine is photographed through the endoscope device and information about the image is obtained (S1), the information on the image is corrected by performing a deep learning operation by the application 50 (S2). Data for the captured image is input data, and data for the image corrected through multiple hidden layers is output data. The corrected output data image from which the noise is removed is displayed on the display device 30 .

In this case, image correction includes distortion correction due to light reflection and color correction. Image correction may be performed on the entire photographed image or only on a region selected by the user. For example, in the case of light reflection correction, it is possible to perform light reflection correction by considering pixels each having an RGB level of 170 or more as light reflection. A conventional method may be applied to the light reflection correction and color correction methods, and thus detailed descriptions thereof will be omitted.

The case of the corrected image is learned through deep learning (S3), and the learned information is stored in the information storage unit 100 of the server 10.

The user may designate a specific region of interest on the image displayed on the display unit 30 ( S4 ), and observe enlarged. For example, areas suspected of being abnormal can be observed by magnifying them because they must be examined closely. The enlarged image designated as a region of interest is displayed after light reflection correction and color correction are performed again.

Information on the lesion existing in the region of interest is transmitted to the server 1 through the application 50 to become input data, and information on the lesion is extracted through the deep neural network (S5). Here, the information about the lesion may be about the type of polyp. The information about the extracted polyp includes whether the polyp present in the region of interest is abnormal that can develop into cancer, the possibility of developing into a malignant tumor, the size, depth, and location of the polyp.

For lesions, based on the deep-learning information, the information can be extracted in the same way as 'adenomatous polyps of 5 mm or less located in the descending colon have a 90% chance of developing into cancer and a 10% chance of being normal'. The information extracted on the lesion is output through the display device 30 , and the user can make a more accurate diagnosis of the lesion with the help of the information output through the display device 30 .

The information on the lesion updated through deep learning is learned and stored in the information storage unit 100 of the server 1 (S7). The information storage unit 100 may be included in the physical server 1 or may be provided in the form of a cloud database. When the updated information is stored in the cloud database, the server 10 may execute an operation of deep learning using the information stored in the cloud database.

The learned information about the lesion is converted into big data and stored in the information storage unit 100 , and any user of the application 50 can access the information. That is, the user can use not only the case in which the colonoscopy was performed by the user, but also information about the case performed by a third party using the application 50, so that the information on the lesion can be further improved based on more data. can be provided accurately.

In the above description, a method for obtaining information on a region of interest by a user directly designating a region of interest has been described. It is also possible for the analysis system to discover itself using deep learning information and deliver the information to the user.

4 is a diagram illustrating a state in which the shape of a polyp is displayed by a deep learning-based colonoscopy image analysis system according to an embodiment of the present invention.

Referring to FIG. 4 , the deep learning-based colonoscopy image analysis system 1 according to an embodiment of the present invention may form a border line around the shape of the polyp so that the size and shape of the polyp can be easily identified. Before forming the edge line, the above-described color correction and light reflection correction may be performed preferentially.

When the border line of the polyp is formed based on the deep learning information, the user can easily grasp information on the shape and size of the polyp.

In general, the possibility of developing colorectal cancer may vary depending on the shape and size of the polyp, so it is important to accurately identify the shape and size of the polyp. According to an embodiment of the present invention, the shape and size of the polyp can be clearly identified through the formation of the border line of the polyp, and further, classification information according to the nature of the polyp is provided based on deep learning, so that the user's It is possible to make an accurate diagnosis of a lesion without relying on experience.

According to the deep learning-based colonoscopy image analysis system and image analysis method according to an embodiment of the present invention, there is no need to replace the existing colonoscopy device or endoscope computer, and there is no need to purchase additional equipment for image correction. , it is advantageous in terms of cost because it is possible to obtain high-level image analysis information by downloading the application.

Since high-level image correction and analysis are performed at a relatively low cost and accurate diagnosis is possible regardless of the user's experience level, high-quality medical services can be easily provided even in small and medium-sized hospitals and developing countries, and the stratification of medical services is deepened. can be prevented from becoming

In addition, not only the user himself but also the information about the lesion deep-learned by a third party is stored in the data pool, so all users can use the information stored in the data pool, so many users see more cases in more cases. The more the invention is used, the more accurate information about the lesion can be provided to the user.

1: Deep learning-based colonoscopy image analysis system
10: Server
20: endoscope computer
30: display device
50: application
100: information storage unit
101: control unit

Claims (10)

Endoscopy computer receiving the image taken by the colonoscopy device;
It is equipped with a diagnostic algorithm that acquires the image transmitted to the endoscope computer through an application downloaded to the endoscope computer in a hooking method, and performs correction of the acquired image and a diagnosis based on the corrected image based on deep learning and a server that transmits the corrected image and a diagnosis result derived through the diagnosis algorithm to the endoscope computer; and
A deep learning-based colonoscopy image analysis system including a; a display device for receiving and outputting the corrected image and the diagnosis result from the endoscope computer. According to claim 1,
The application is a deep learning-based colonoscopy image analysis system, characterized in that driven in a window overlay method. According to claim 1,
The information learned through deep learning is stored in the information storage unit provided in the server, and the information is shared among users of the application. According to claim 1,
A deep learning-based colonoscopy image analysis system in which information learned through deep learning is stored in a cloud database and the information is shared between users of the application. According to claim 1,
The image transmitted to the server is a deep learning-based colonoscopy image analysis system, characterized in that it undergoes a process of color correction and light reflection correction based on the deep-learned information. According to claim 1,
A deep learning-based colonoscopy image analysis system that determines that the image taken by the endoscope device has light reflection for pixels having an RGB level of 170 or more, and the server executes light reflection correction through deep learning. According to claim 1,
The image transmitted to the server includes an image of a polyp, and a border line matching the shape of the polyp is formed through deep learning and displayed on the display device. An image acquired by the endoscope device is hooked in the endoscope computer and transferred to a server;
calibrating the image delivered to the server delivered based on deep learning and diagnosing and learning the lesion on the image;
transmitting the corrected image and information on the diagnosed lesion to the endoscope computer; and
Displaying the corrected image and information on the diagnosed lesion on a display device; Deep learning-based colonoscopy image analysis method comprising a. 9. The method of claim 8,
The information on the corrected image and the diagnosed lesion is a deep learning-based colonoscopy image analysis method further comprising the step of storing the information in a storage space open to the application user. 9. The method of claim 8,
Deep learning-based colonoscopy image analysis method further comprising the step of forming a border line along the shape of the polyp present on the image transmitted to the server and displaying it on a display device.

Images