WO2021261727A1

WO2021261727A1 - Capsule endoscopy image reading system and method

Info

Publication number: WO2021261727A1
Application number: PCT/KR2021/004735
Authority: WO
Inventors: 이한희; 이승철; 황윤섭
Original assignee: 가톨릭대학교 산학협력단; 포항공과대학교 산학협력단
Priority date: 2020-06-25
Filing date: 2021-04-15
Publication date: 2021-12-30
Also published as: KR20220000437A; KR102359984B1

Abstract

Disclosed in the present specification are a capsule endoscopy image reading system and method, the system comprising: a preprocessing unit for preprocessing a capsule endoscopy image captured by means of capsule endoscopy; a convolution neural network (CNN) using the preprocessed capsule endoscopy image as an input to determine whether a lesion is present in the capsule endoscopy image; and a gradient class activation map (grad-CAM) acquisition unit for acquiring a grad-CAM for the capsule endoscopy image.

Description

Capsule endoscopy image reading system and method

The present invention relates to a capsule endoscopic image reading system and method.

Capsule endoscope is a device used for diagnosing digestive diseases by swallowing a pill-shaped capsule with the mouth, taking pictures of health conditions of the esophagus, stomach, and small intestine, and analyzing and reading these images. The small intestine is located in the middle of the stomach and large intestine, and is about 6 meters long. When taking a small intestine with a capsule endoscope, it takes more than 10 hours and records more than 50,000 images, but there are limitations in time and accuracy for a doctor to read it directly.

Embodiments provide a capsule endoscope image reading system and method that can reduce a doctor's reading time and increase accuracy for a large amount of endoscopic images taken with a capsule endoscope.

In addition, embodiments provide a capsule endoscopy image reading system and method by which a doctor can intuitively check the location of a lesion by visualizing a location of a lesion located on a capsule endoscopy image without separate labeling.

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

One embodiment, a pre-processing unit for pre-processing the capsule endoscope image taken by the capsule endoscope; a convolutional neural network (CNN) that determines whether a lesion exists in a capsule endoscopy image by inputting the preprocessed capsule endoscopy image; and a grad-CAM acquisition unit for acquiring a gradient class activation map (grad-CAM) for the capsule endoscopy image, wherein the convolutional neural network includes an input layer for receiving the preprocessed capsule endoscope image; one or more convolutional layers for extracting features of the preprocessed capsule endoscopy image input through the input layer; one or more maximal pooling layers for subsampling features for a capsule endoscopy image; and an output layer that outputs a probability value indicating the presence or absence of a lesion with respect to the capsule endoscopy image, wherein the grad-CAM acquisition unit acquires the grad-CAM from the layer determined to have the highest lesion location detection ability among the convolutional layer and the maximum pooling layer. A capsule endoscopy image reading system is provided.

Another embodiment is a capsule endoscope image reading method using a capsule endoscope image reading system, a pre-processing step of pre-processing the capsule endoscope image taken by the capsule endoscope; an input step of receiving a pre-processed capsule endoscope image; a processing step of repeatedly executing a processing operation of extracting features for the pre-processed capsule endoscopy image input in the input step and subsampling the extracted features; a grad-CAM acquisition step of acquiring a grad-CAM (gradient class activation map) based on the result of the processing step; and outputting a probability value indicating whether a lesion exists in the capsule endoscopy image.

According to the capsule endoscope image reading system and method according to the embodiments, it is possible to reduce a doctor's reading time and increase accuracy for a large amount of endoscopic images taken with the capsule endoscope.

In addition, according to the capsule endoscopy image reading system and method according to the embodiments, the position of the lesion located on the capsule endoscope image is visualized without separate labeling, so that the doctor can intuitively check the position of the lesion.

1 is a block diagram showing an example of a capsule endoscope image reading system according to the present invention.

2 is a block diagram illustrating an example of a preprocessor of a capsule endoscope image reading system according to the present invention.

FIG. 3 is a diagram illustrating an example of removing noise from an image through the preprocessor of FIG. 2 .

4 is a diagram illustrating an example of augmenting an image through the preprocessor of FIG. 2 .

5 is a diagram illustrating a convolutional neural network of a capsule endoscope image reading system according to the present invention.

6 is a view showing an example of grad-CAM generated by the capsule endoscopy image reading system according to the present invention.

7 is a diagram illustrating an example in which the capsule endoscopy image reading system according to the present invention determines a layer from which grad-CAM is acquired.

8 is a view showing an example of the structure of a video clip generated by the capsule endoscope image reading system according to the present invention.

9 is a diagram illustrating an example of a frame included in a video clip generated by the capsule endoscope image reading system according to the present invention.

10 is a diagram illustrating an example of a frame included in two video clips generated by the capsule endoscope image reading system according to the present invention.

11 is a diagram illustrating a new video clip in which the two video clips of FIG. 10 are merged;

12 is a view showing a capsule endoscope image set applied to the capsule endoscope image reading system according to the present invention.

13 is a flowchart of a capsule endoscopy image reading method according to the present invention.

14 is a flowchart illustrating details of a pre-processing step of a capsule endoscopy image reading method according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, it includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

The terms "about", "substantially", etc. to the extent used throughout the specification are used in or close to the numerical value when manufacturing and material tolerances inherent in the stated meaning are presented, and are intended to enhance the understanding of the present invention. To help, precise or absolute figures are used to prevent unfair use by unconscionable infringers of the stated disclosure. As used throughout the specification of the present invention, the term "step of (to)" or "step of" does not mean "step for".

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal, apparatus, or device in the present specification may be performed instead of by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , the capsule endoscopy image reading system 100 includes a preprocessor 110 , a convolutional neural network 120 , and a gradient class activation map (grad-CAM) acquisition unit 130 . can do.

The preprocessor 110 may preprocess the capsule endoscope image captured by the capsule endoscope.

The convolutional neural network 120 may determine whether a lesion exists in the capsule endoscope image, that is, whether a lesion exists in the capsule endoscope image, by receiving the capsule endoscope image preprocessed by the preprocessor 110 as an input. The capsule endoscopy image reading system processes a large amount of capsule endoscopy images through the convolutional neural network 120, so that the presence of a lesion is read faster than a doctor reading the capsule endoscopy images one by one with the naked eye to determine the presence of a lesion. make it possible

At this time, the convolutional neural network 120 is an input layer 121 that receives a preprocessed capsule endoscope image from the preprocessor 110 , and extracts features of the preprocessed capsule endoscope image input through the input layer 121 . Outputs one or more convolution layers 122, one or more Max Pooling Layers 123 for sub-sampling features of a capsule endoscopy image, and a probability value indicating whether a lesion exists in the capsule endoscopy image. It may include an output layer 124 that

The grad-CAM acquisition unit 130 may acquire the grad-CAM for the capsule endoscopy image.

In this case, the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection capability among the above-described convolutional layer 122 and the maximum pooling layer 123 .

In addition, the capsule endoscopy image reading system 100 may further include a video clip generation unit 140 in addition to the preprocessor 110 , the convolutional neural network 120 , and the grad-CAM acquisition unit 130 described above. The video clip generator 140 may generate a video clip corresponding to the capsule endoscope image based on the capsule endoscope image.

Referring to FIG. 2 , the pre-processing unit 110 may include a noise removing unit 111 and an image enhancing unit 112 .

The noise removing unit 111 may remove noise from the capsule endoscope image input to the preprocessing unit 110 .

The image augmentation unit 112 may generate a plurality of augmented images by performing at least one of rotation and vertical inversion on the noise-removed capsule endoscope image.

Hereinafter, operations of the noise removing unit 111 and the image enhancing unit 112 in FIGS. 3 and 4 will be described in detail.

The noise removing unit 111 may remove noise including letters, numbers, and symbols recorded in the capsule endoscope image from the capsule endoscope image. This is because letters, numbers, symbols, etc. are irrelevant to determining whether a lesion exists, and the convolutional neural network 120 may interfere with determining whether a lesion exists in the capsule endoscopy image.

Referring to FIG. 3 , the noise removing unit 111 deletes letters, numbers, symbols, etc. indicating the time, date, and photographing equipment, etc. on the edge of the capsule endoscopy image of 576 * 576 * 3 to 512 * 512 * 3 Capsule endoscopy images of

Referring to FIG. 4 , the image augmentation unit 112 may generate eight augmented images by performing at least one of rotation (90 degrees/180 degrees/270 degrees) and vertical inversion on the noise-removed capsule endoscope image. have. Meanwhile, FIG. 4 describes a case in which both rotation (90 degrees/180 degrees/270 degrees) and vertical inversion are performed. However, in the present invention, the image augmentation unit 112 may perform all of the rotation and up/down inversion, or may generate less than 8 augmented images by performing only some of the rotation and up/down inversion.

The reason why the image augmentation unit 112 generates the augmented image by performing at least one of rotation and vertical inversion is to prevent the characteristics of the lesion present in the original capsule endoscopy image for the augmented image from being damaged in the image processing process. to be.

Capsule endoscopy images generally have a shape that shows images of the small intestine in a circular shape in a black background.

This is because, in the original capsule endoscopy image, if a pixel representing a lesion is distorted or damaged, the distorted pixel may influence the convolutional neural network to determine whether a lesion exists.

Referring to FIG. 5 , the convolutional neural network 120 may extract a feature of the capsule endoscope image preprocessed through the input layer 121 through the convolution layer 122 . In addition, the convolutional neural network 120 may subsample the feature extracted through the convolution layer 122 through the maximum pooling layer 123 . Meanwhile, the convolutional neural network 120 may repeat the process of subsampling a feature extracted through any one convolutional layer through any one maximum pooling layer and then inputting the result back to another convolutional layer.

In addition, the convolutional neural network 120 uses the result generated using the convolutional layer 122 and the maximum pooling layer 123 to generate a probability value indicating whether a lesion exists in the capsule endoscopy image through the output layer 124 . can be printed out.

The output layer 124 analyzes the result value generated using the convolution layer 122 and the maximum pooling layer 123 using one or more fully connected layers, and applies various transform functions (eg softmax) to it. It can be converted into a probability value indicating the presence or absence of a lesion using

In this case, the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection capability among the convolutional layer 122 and the maximum pooling layer 123 . The doctor can check with high accuracy which part of the capsule endoscopy image where the lesion was judged to have had an influence on the judgment of the presence of the lesion through grad-CAM.

In general, a CAM (Class Activation Map) is used to analyze which part of a capsule endoscopy image has a great influence on determining whether a convolutional neural network is a lesion. The CAM is a map that visualizes the result of calculating the sum of the weights of the feature map using the weight just before the layer that predicts the probability value. can be found If the CAM is superimposed on the capsule endoscopy image, it is possible to easily determine the area where the lesion has occurred in the capsule endoscopy image.

However, in order to obtain CAM, the process of performing tuning using a global average pooling (GAP) layer is essential. Therefore, in order to include the global average pooling layer in the convolutional neural network, modifications are made to the structure of the convolutional neural network, which may reduce the convolutional neural network's ability to detect lesions.

Accordingly, in the present invention, grad-CAM is acquired using the grad-CAM acquisition unit 130 to obtain a CAM result without depending on the global average pooling (GAP) layer and without modifying the structure of the convolutional neural network. Therefore, the use of grad-CAM does not impose any constraints on the structure of the convolutional neural network, so the ability to detect the presence of lesions and the ability to track the location of lesions can be improved.

As an example, the grad-CAM may obtain an importance weight by the following equation by using the gradient of the above-described convolutional layer or the maximum pooling layer and result information passing the layer.

At this time,

is class,

is the result information passing through the layer,

is the value corresponding to the result information (i,j) to be observed,

denotes the effect of y on A, i.e. the gradient.

the aforementioned

Wow

grad-CAM can be calculated using the following equation.

On the other hand, the reason for using ReLU in Equation 2 is to reflect only the effect of the positive (plus) on the class.

Referring to FIG. 6 , the grad-CAM for the capsule endoscopy image on the left is displayed on the right. Looking at the grad-CAM, it can be seen that the middle part marked with a different color from the outer part had a great influence on the determination of the presence of a lesion, and it can be seen that the probability of lesion presence in the capsule endoscopy image is about 71.64%. As such, since the location of the lesion is intuitively displayed on the grad-CAM for the capsule endoscopy image, a viewer (e.g. a doctor) reading the capsule endoscope image can check the location of the lesion in the capsule endoscope image without separate labeling.

As described above, the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection ability among one or more convolutional layers 122 and one or more maximum pooling layers 123 . . In this case, the highest lesion location detection ability means that the region that has an important influence on the grad-CAM to determine the lesion existence probability and the region where the lesion exists in the actual capsule endoscopy image match the most.

Referring to FIG. 7 , as an example, the grad-CAM acquisition unit 130 includes a first convolutional layer (CONV_1), a second convolutional layer (CONV_2), and a third convolutional layer included in the convolutional layer 122 for a capsule endoscopy image. (CONV_3), the fourth convolutional layer (CONV_4), the first maximum pooling layer (MAXP_1) included in the maximum pooling layer 123, the second maximum pooling layer (MAXP_2), and the third maximum pooling layer (MAXP_3) Based on grad-CAM, it is possible to determine the layer with the highest lesion localization ability.

In this case, the grad-CAM acquisition unit 130 may acquire the grad-CAM for one or more test capsule endoscopy images in order to determine the layer having the highest lesion location detection capability. If there are a plurality of capsule endoscopy images for testing, the grad-CAM acquisition unit 130 calculates, for example, the average of the lesion location detection capability of grad-CAM for each test capsule endoscope image for each layer for each layer of the lesion of each layer. It is possible to determine the location detection capability.

In FIG. 7 , the grad-CAM acquisition unit 130 may determine that the grad-CAM acquired from the first maximum pooling layer MAXP_1 with respect to the capsule endoscopy image for testing has the highest lesion location detection capability. In this case, the grad-CAM acquisition unit 130 may then acquire the grad-CAM for the capsule endoscopy image from the first maximum pooling layer MAXP_1 among the convolutional layer 122 and the maximum pooling layer 123 .

Referring to FIG. 8 , when generating a video clip corresponding to a capsule endoscopy image, the video clip generating unit 140 may generate a video clip based on a video image in which a lesion exists.

In the present invention, the presence or absence of a lesion is expressed through a probability value, so the video clip generator 140 calculates a probability value indicating the presence of a lesion using the convolutional neural network 120 described above for the capsule endoscopy image. A video clip corresponding to the capsule endoscope image may be generated based on a capsule endoscope image in which a probability value indicating the presence of a lesion in the capsule endoscope image is equal to or greater than a threshold value (eg 0.8).

Referring to FIG. 9 , the video clip generator 140 adds frames as much as the maximum reference value (eg 5) before and after the capsule endoscopy image in which the lesion is determined to exist (that is, the probability value is greater than or equal to the threshold value) to the capsule endoscopy image. A corresponding video clip can be created. For example, when the reference value is 5, the video clip generating unit 140 may add up to 5 frames before and after the capsule endoscopy image determined to exist of the lesion to the video clip. If the number of frames to be played is less than 5 (eg the 4th frame after the start), all appendable frames can be added to the video clip.

In FIG. 9 , a case in which it is determined that a lesion exists in the capsule endoscopy image of frame N will be described as an example. In this case, assuming that the reference value is A, the video clip generator 140 may generate a video clip by combining not only the N-th frame, but also A frames before the N-th frame and A frames after the N-th frame.

The reason that the video clip includes not only the capsule endoscopy image determined to have a lesion, but also the capsule endoscopy image before and after it is included in the video clip. This is in order to confirm the continuous change to the endoscopic image or vice versa through the video clip. A video clip consisting only of images predicted to have a lesion is unsuitable for reading because the front and back frames appear to be cut off to the viewer. In this case, the aforementioned reference value may be arbitrarily determined at a level that does not cause inconvenience to the viewer.

In FIG. 10 , it is assumed that a lesion has occurred in the capsule endoscopy images of frame M and frame N. At this time, according to the method described with reference to FIG. 9 , the video clip generating unit 140 generates video clip 1 which is a video clip based on the capsule endoscope image of frame M, and a video clip that is based on the capsule endoscope image of frame N. You can create video clip 2.

At this time, in FIG. 10 , frame (M+A), which is the last frame of video clip 1, is a frame earlier than frame (N-A), which is the start frame of video clip 2. That is, video clip 1 and video clip 2 have frames overlapping each other.

When there is a frame overlapping the video clip 1 and the video clip 2, the video clip generator 140 merges the video clip 1 and the video clip 2 instead of separately generating the video clip 1 and the video clip 2 to form one video. You can create clips.

Referring to FIG. 11 , the video clip generator 140 may generate the video clip 3 by merging the aforementioned video clip 1 and video clip 2 . The video clip 3 may include a frame (N+A), which is the most recently generated frame, from a frame (M-A) that is the earliest generated frame among frames included in the video clip 1 or the video clip 2 . Through this, the video clip generating unit 140 may express a change between frame M and frame N, which is a capsule endoscopy image in which a lesion exists, through one video clip.

Referring to FIG. 12 , the number of images included in a training image set, which is a set of capsule endoscopy images used for learning the convolutional neural network 120 of the capsule endoscope image reading system, and the convolutional neural network 120 ), the ratio of the number of images included in the test image set, which is a set of capsule endoscopy images used for the test, may be determined as a preset ratio value. As an example, (the number of images in the training image set):(the number of images in the test image set) may be 7:3 or 8:2.

The above-mentioned ratio value may be selected as a value that can increase the accuracy of the lesion existence probability the most. This ratio value may be set to a fixed value in the capsule endoscopy image reading system.

In this case, the ratio of the number of images of the test image set to the number of images of the training image set may decrease as the number of the entire capsule endoscopy image increases. For example, if the total number of capsule endoscopy images is 10000 (number of images in the training image set):(number of images in the test image set) = 7:3, when the number of total capsule endoscopy images is 20000 (the number of images in the training image set) number of images in):(number of images in the test image set) = 8:2.

This means that as the number of total capsule endoscopy images increases, the ratio of the number of images in the test image set decreases, but the ratio of the number of images in the test image set satisfies a certain level or more, whereas as the number of images in the training image set increases, the convolutional neural This is because it is desirable to increase the number of images in the training image set as much as possible because the accuracy of the network 120 increases.

Meanwhile, the aforementioned ratio value may be equally applied to each of a capsule endoscopy image in which a lesion exists and a capsule endoscopy image in which a lesion does not exist. For example, if (the number of training image sets):(the number of test image sets) = 8:2 for a capsule endoscopy image in which a lesion exists, even for a capsule endoscopy image in which a lesion does not exist (the number of training image sets) ):(number of test image sets) = 8:2.

Meanwhile, when receiving the training image set, the input layer 121 of the convolutional neural network 120 may receive the same number of images with lesions and images without lesions. This is because, in the learning process of the convolutional neural network 120, when an image corresponding to either an image in which a lesion exists or an image without a lesion is excessively input, a bias is likely to be applied to the excessively inputted image. because it is high

Referring to FIG. 13 , the capsule endoscope image reading method 1300 may include a pre-processing step ( S1310 ) of pre-processing the capsule endoscope image captured by the capsule endoscope.

And the capsule endoscope image reading method may include an input step (S1320) of receiving the capsule endoscope image preprocessed in step S1310.

In addition, the capsule endoscope image reading method may include a processing step (S1330) of extracting features for the preprocessed capsule endoscope image input in step S1320 and repeatedly executing a processing operation of subsampling the extracted features.

And the capsule endoscopy image reading method may include a grad-CAM acquisition step (S1340) of acquiring a grad-CAM (gradient class activation map) based on the result of step S1330.

In addition, the capsule endoscopy image reading method may include outputting a probability value indicating whether a lesion exists in the capsule endoscope image ( S1350 ).

Meanwhile, the capsule endoscope image reading method 1300 may be performed through the capsule endoscope image reading system 100 described above.

Referring to FIG. 14 , the pre-processing step ( S1310 ) may include a noise removing step ( S1410 ) of removing noise from the capsule endoscopy image.

In addition, the preprocessing step (S1310) may include an image augmentation step (S1420) of generating a plurality of augmented images by performing at least one of rotation and vertical inversion on the capsule endoscope image from which the noise has been removed in step S1410.

Meanwhile, the pre-processing step S1310 may be performed by the pre-processing unit 110 of the above-described capsule endoscope image reading system.

The capsule endoscopy image reading system and method described in the embodiments of the present invention can reduce the reading time of a doctor and increase the accuracy of a large amount of endoscopic images taken with the capsule endoscope.

The deep learning model described in the embodiments of the present invention may be a model in which artificial neural networks are stacked in multi-layered layers. That is, the deep learning model automatically learns the characteristics of the input value by learning a large amount of data from a deep neural network consisting of a multi-layered network, and through this, the network is trained to minimize the error in the objective function, that is, the prediction accuracy. is the model of

In the present invention, a case in which the deep learning model is a convolutional neural network (CNN) has been described as an example, but the present invention is not limited thereto, and various deep learning models that can be used now or in the future may be used.

A deep learning model can be implemented through a deep learning framework. The deep learning framework provides functions commonly used when developing deep learning models in the form of a library and plays a role in supporting the good use of system software or hardware platforms. In this embodiment, the deep learning model may be implemented using any deep learning framework that is currently public or will be released in the future.

The capsule endoscope image reading system described above may be implemented by a computing device including at least a portion of a processor, a memory, a user input device, and a presentation device. A memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and/or data, etc. coded to perform specific tasks when executed by a processor. The processor may read and execute computer-readable software, applications, program modules, routines, instructions, and/or data stored in the memory and/or the like. The user input device may be a means for allowing the user to input a command to the processor to execute a specific task or to input data required for the execution of the specific task. The user input device may include a physical or virtual keyboard or keypad, key button, mouse, joystick, trackball, touch-sensitive input means, or a microphone. The presentation device may include a display, a printer, a speaker, or a vibrator.

Computing devices may include various devices such as smartphones, tablets, laptops, desktops, servers, clients, and the like. The computing device may be a single stand-alone device, or may include a plurality of computing devices operating in a distributed environment comprising a plurality of computing devices cooperating with each other through a communication network.

In addition, the capsule endoscopy image reading method described above includes a processor, and when executed by the processor, computer readable software, applications, program modules, routines, and instructions coded to perform an image diagnosis method using a deep learning model , and/or may be executed by a computing device having a memory storing data structures, and/or the like.

The above-described embodiments may be implemented through various means. For example, the present embodiments may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the image diagnosis method using the deep learning model according to the present embodiments includes one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), It may be implemented by Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers or microprocessors, and the like.

For example, the capsule endoscopy image reading method according to the embodiments may be implemented using an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented with semiconductor elements. In this case, the semiconductor device may be currently used semiconductor devices, for example, SRAM, DRAM, NAND, or the like, or may be next-generation semiconductor devices, RRAM, STT MRAM, PRAM, or the like, or a combination thereof.

When the capsule endoscope image reading method according to the embodiments is implemented using an artificial intelligence semiconductor device, the result (weight) of learning a deep learning model with software is transcribed into a synaptic mimic device arranged in an array or learned in an artificial intelligence semiconductor device may proceed.

In the case of implementation by firmware or software, the capsule endoscope image reading method according to the present embodiments may be implemented in the form of an apparatus, procedure, or function for performing the functions or operations described above. The software code may be stored in the memory unit and driven by the processor. The memory unit may be located inside or outside the processor, and may transmit and receive data to and from the processor by various known means.

Also, as described above, terms such as "system", "processor", "controller", "component", "module", "interface", "model", or "unit" generally refer to computer-related entities hardware, hardware and software. may mean a combination of, software, or running software. For example, the aforementioned component may be, but is not limited to, a process run by a processor, a processor, a controller, a controlling processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a controller or processor and a controller or processor can be a component. One or more components may reside within a process and/or thread of execution, and the components may be located on one device (eg, a system, computing device, etc.) or distributed across two or more devices.

On the other hand, another embodiment provides a computer program stored in a computer recording medium for performing the above-described capsule endoscope image reading method. Another embodiment also provides a computer-readable recording medium in which a program for realizing the above-described capsule endoscope image reading method is recorded.

The program recorded on the recording medium can be read by a computer, installed, and executed to execute the above-described steps.

In this way, in order for the computer to read the program recorded in the recording medium and execute the functions implemented as the program, the above-described program is C, C++, which the processor (CPU) of the computer can read through the device interface of the computer. , JAVA, and may include code coded in a computer language such as machine language.

Such codes may include function codes related to functions defining the above-mentioned functions, etc., and may include control codes related to an execution procedure necessary for the processor of the computer to execute the above-mentioned functions according to a predetermined procedure.

In addition, this code may further include additional information necessary for the processor of the computer to execute the above functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. .

In addition, when the computer's processor needs to communicate with any other computer or server located remotely in order to execute the above-mentioned functions, the code is transmitted to the computer's processor using the computer's communication module. It may further include a communication-related code for how to communicate with another computer or server, and what kind of information or media to transmit and receive during communication.

The computer-readable recording medium in which the program as described above is recorded is, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage device, etc., and also carrier wave (eg, , transmission through the Internet) may be implemented in the form of.

In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes can be stored and executed in a distributed manner.

In addition, in consideration of the system environment of a computer that reads a recording medium and executes a program by reading a recording medium, a functional program for implementing the present invention and related codes and code segments, programmers in the technical field to which the present invention belongs may be easily inferred or changed by

The capsule endoscopy image reading method described with reference to FIG. 10 may also be implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above-described capsule endoscope image reading method may be executed by an application basically installed in the terminal (which may include a program included in a platform or operating system basically mounted in the terminal), and the user may use an application store server, an application or It may be executed by an application (ie, a program) directly installed in the master terminal through an application providing server such as a web server related to the corresponding service. In this sense, the above-described capsule endoscopy image reading method may be implemented as an application (ie, a program) installed basically in a terminal or directly installed by a user, and may be recorded in a computer-readable recording medium such as a terminal.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

CROSS-REFERENCE TO RELATED APPLICATIONCROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority under section 119(a) of the US Patent Act (35 USC § 119(a)) with respect to Patent Application No. 10-2020-0077757 filed in Korea on June 25, 2020, and All contents are incorporated into this patent application by reference. In addition, if this patent application claims priority for countries other than the United States for the same reason as above, all contents thereof are incorporated into this patent application by reference.

Claims

A pre-processing unit for pre-processing the capsule endoscope image taken by the capsule endoscope;

a convolutional neural network (CNN) that receives the pre-processed capsule endoscopy image as an input and determines whether a lesion exists in the capsule endoscope image; and

and a grad-CAM acquisition unit for acquiring a grad-CAM (Gradient Class Activation Map) for the capsule endoscopy image,

The convolutional neural network is

an input layer receiving the pre-processed capsule endoscope image;

one or more convolutional layers for extracting features of the preprocessed capsule endoscopy image input through the input layer;

one or more maximal pooling layers for sub-sampling features of the capsule endoscopy image; and

and an output layer for outputting a probability value indicating whether a lesion exists with respect to the capsule endoscopy image,

The grad-CAM acquisition unit,

A capsule endoscopy image reading system for acquiring grad-CAM from the layer determined to have the highest lesion location detection ability among the convolutional layer and the maximum pooling layer.
According to claim 1,

The preprocessor is

a noise removing unit for removing noise from the capsule endoscope image; and

and an image enhancer configured to generate a plurality of augmented images by performing at least one of rotation and vertical inversion on the capsule endoscope image from which the noise has been removed.
3. The method of claim 2,

The noise removing unit,

Capsule endoscope image reading system for removing noise including letters, numbers, and symbols recorded in the capsule endoscope image.
According to claim 1,

The capsule endoscope image reading system further comprising a video clip generation unit generating a video clip corresponding to the capsule endoscope image based on the capsule endoscope image in which a probability value indicating the presence of a lesion in the capsule endoscope image is equal to or greater than a threshold value.
5. The method of claim 4,

The video clip generation unit,

A capsule endoscope image reading system for generating a video clip corresponding to the capsule endoscope image by adding frames as many as a maximum reference value before and after the capsule endoscope image.
6. The method of claim 5,

The video clip generation unit,

When there are two different video clips with overlapping frames, the capsule endoscope image reading system merges the two video clips.
According to claim 1,

Among the capsule endoscopy images, for each of an image with a lesion and an image without a lesion, the number of images in the training image set used for learning the convolutional neural network and the test image set used for testing the convolutional neural network. A capsule endoscope image reading system in which the ratio of the number of images is determined by a preset ratio value.
8. The method of claim 7,

The input layer is

A capsule endoscopy image reading system for receiving the same number of images with lesions and images without lesions with respect to the learning image set.
A capsule endoscope image reading method using a capsule endoscope image reading system,

A pre-processing step of pre-processing the capsule endoscope image taken by the capsule endoscope;

an input step of receiving the pre-processed capsule endoscope image;

a processing step of extracting features for the pre-processed capsule endoscope image input in the input step and repeatedly executing a processing operation of subsampling the extracted features;

a grad-CAM obtaining step of obtaining a grad-CAM (gradient class activation map) based on the result of the processing step; and

and an output step of outputting a probability value indicating whether a lesion exists in the capsule endoscopy image.
10. The method of claim 9,

The pre-processing step is

a noise removing step of removing noise from the capsule endoscopy image; and

and an image augmentation step of generating a plurality of augmented images by performing at least one of rotation and vertical inversion on the capsule endoscope image from which the noise has been removed.
11. The method of claim 10,

The noise removal step is

A capsule endoscope image reading method for removing noise including letters, numbers, and symbols recorded in the capsule endoscope image.
10. The method of claim 9,

and a video clip generation step of generating a video clip corresponding to the capsule endoscope image based on a capsule endoscope image in which a probability value indicating whether a lesion exists in the capsule endoscope image is equal to or greater than a threshold value.
13. The method of claim 12,

The video clip creation step comprises:

A capsule endoscopy image reading method for generating a video clip corresponding to the capsule endoscope image by adding frames as many as a maximum reference value before and after the capsule endoscope image.
14. The method of claim 13,

The video clip creation step comprises:

When there are two different video clips having overlapping frames, the capsule endoscopy image reading method is for merging the two video clips.
10. The method of claim 9,

Among the capsule endoscopy images, for each of an image with a lesion and an image without a lesion, the number of images in the training image set used for learning the convolutional neural network and the test image set used for testing the convolutional neural network. A capsule endoscopy image reading method in which the ratio of the number of images is determined by a preset ratio value.
16. The method of claim 15,

The input step is

Capsule endoscopy image reading method for receiving the same number of images with lesions and images without lesions for the learning image set.