WO2021261727A1 - Système et procédé de lecture d'images d'endoscopie par capsule - Google Patents

Système et procédé de lecture d'images d'endoscopie par capsule Download PDF

Info

Publication number
WO2021261727A1
WO2021261727A1 PCT/KR2021/004735 KR2021004735W WO2021261727A1 WO 2021261727 A1 WO2021261727 A1 WO 2021261727A1 KR 2021004735 W KR2021004735 W KR 2021004735W WO 2021261727 A1 WO2021261727 A1 WO 2021261727A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
capsule endoscope
capsule
endoscope image
images
Prior art date
Application number
PCT/KR2021/004735
Other languages
English (en)
Korean (ko)
Inventor
이한희
이승철
황윤섭
Original Assignee
가톨릭대학교 산학협력단
포항공과대학교 산학협력단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 가톨릭대학교 산학협력단, 포항공과대학교 산학협력단 filed Critical 가톨릭대학교 산학협력단
Publication of WO2021261727A1 publication Critical patent/WO2021261727A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00064—Constructional details of the endoscope body
    • A61B1/00108—Constructional details of the endoscope body characterised by self-sufficient functionality for stand-alone use
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/041—Capsule endoscopes for imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2576/00—Medical imaging apparatus involving image processing or analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • the present invention relates to a capsule endoscopic image reading system and method.
  • Capsule endoscope is a device used for diagnosing digestive diseases by swallowing a pill-shaped capsule with the mouth, taking pictures of health conditions of the esophagus, stomach, and small intestine, and analyzing and reading these images.
  • the small intestine is located in the middle of the stomach and large intestine, and is about 6 meters long. When taking a small intestine with a capsule endoscope, it takes more than 10 hours and records more than 50,000 images, but there are limitations in time and accuracy for a doctor to read it directly.
  • Embodiments provide a capsule endoscope image reading system and method that can reduce a doctor's reading time and increase accuracy for a large amount of endoscopic images taken with a capsule endoscope.
  • embodiments provide a capsule endoscopy image reading system and method by which a doctor can intuitively check the location of a lesion by visualizing a location of a lesion located on a capsule endoscopy image without separate labeling.
  • the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.
  • a pre-processing unit for pre-processing the capsule endoscope image taken by the capsule endoscope; a convolutional neural network (CNN) that determines whether a lesion exists in a capsule endoscopy image by inputting the preprocessed capsule endoscopy image; and a grad-CAM acquisition unit for acquiring a gradient class activation map (grad-CAM) for the capsule endoscopy image, wherein the convolutional neural network includes an input layer for receiving the preprocessed capsule endoscope image; one or more convolutional layers for extracting features of the preprocessed capsule endoscopy image input through the input layer; one or more maximal pooling layers for subsampling features for a capsule endoscopy image; and an output layer that outputs a probability value indicating the presence or absence of a lesion with respect to the capsule endoscopy image, wherein the grad-CAM acquisition unit acquires the grad-CAM from the layer determined to have the highest lesion location detection ability among the convolutional layer and the maximum pool
  • Another embodiment is a capsule endoscope image reading method using a capsule endoscope image reading system, a pre-processing step of pre-processing the capsule endoscope image taken by the capsule endoscope; an input step of receiving a pre-processed capsule endoscope image; a processing step of repeatedly executing a processing operation of extracting features for the pre-processed capsule endoscopy image input in the input step and subsampling the extracted features; a grad-CAM acquisition step of acquiring a grad-CAM (gradient class activation map) based on the result of the processing step; and outputting a probability value indicating whether a lesion exists in the capsule endoscopy image.
  • a pre-processing step of pre-processing the capsule endoscope image taken by the capsule endoscope
  • an input step of receiving a pre-processed capsule endoscope image
  • a processing step of repeatedly executing a processing operation of extracting features for the pre-processed capsule endoscopy image input in the input step and subs
  • the capsule endoscope image reading system and method according to the embodiments it is possible to reduce a doctor's reading time and increase accuracy for a large amount of endoscopic images taken with the capsule endoscope.
  • the position of the lesion located on the capsule endoscope image is visualized without separate labeling, so that the doctor can intuitively check the position of the lesion.
  • FIG. 1 is a block diagram showing an example of a capsule endoscope image reading system according to the present invention.
  • FIG. 2 is a block diagram illustrating an example of a preprocessor of a capsule endoscope image reading system according to the present invention.
  • FIG. 3 is a diagram illustrating an example of removing noise from an image through the preprocessor of FIG. 2 .
  • FIG. 4 is a diagram illustrating an example of augmenting an image through the preprocessor of FIG. 2 .
  • FIG. 5 is a diagram illustrating a convolutional neural network of a capsule endoscope image reading system according to the present invention.
  • FIG. 6 is a view showing an example of grad-CAM generated by the capsule endoscopy image reading system according to the present invention.
  • FIG. 7 is a diagram illustrating an example in which the capsule endoscopy image reading system according to the present invention determines a layer from which grad-CAM is acquired.
  • FIG. 8 is a view showing an example of the structure of a video clip generated by the capsule endoscope image reading system according to the present invention.
  • FIG. 9 is a diagram illustrating an example of a frame included in a video clip generated by the capsule endoscope image reading system according to the present invention.
  • FIG. 10 is a diagram illustrating an example of a frame included in two video clips generated by the capsule endoscope image reading system according to the present invention.
  • FIG. 11 is a diagram illustrating a new video clip in which the two video clips of FIG. 10 are merged;
  • FIG. 12 is a view showing a capsule endoscope image set applied to the capsule endoscope image reading system according to the present invention.
  • FIG. 13 is a flowchart of a capsule endoscopy image reading method according to the present invention.
  • FIG. 14 is a flowchart illustrating details of a pre-processing step of a capsule endoscopy image reading method according to the present invention.
  • a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both.
  • one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.
  • Some of the operations or functions described as being performed by the terminal, apparatus, or device in the present specification may be performed instead of by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.
  • mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as
  • FIG. 1 is a block diagram showing an example of a capsule endoscope image reading system according to the present invention.
  • the capsule endoscopy image reading system 100 includes a preprocessor 110 , a convolutional neural network 120 , and a gradient class activation map (grad-CAM) acquisition unit 130 . can do.
  • the preprocessor 110 may preprocess the capsule endoscope image captured by the capsule endoscope.
  • the convolutional neural network 120 may determine whether a lesion exists in the capsule endoscope image, that is, whether a lesion exists in the capsule endoscope image, by receiving the capsule endoscope image preprocessed by the preprocessor 110 as an input.
  • the capsule endoscopy image reading system processes a large amount of capsule endoscopy images through the convolutional neural network 120, so that the presence of a lesion is read faster than a doctor reading the capsule endoscopy images one by one with the naked eye to determine the presence of a lesion. make it possible
  • the convolutional neural network 120 is an input layer 121 that receives a preprocessed capsule endoscope image from the preprocessor 110 , and extracts features of the preprocessed capsule endoscope image input through the input layer 121 .
  • the grad-CAM acquisition unit 130 may acquire the grad-CAM for the capsule endoscopy image.
  • the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection capability among the above-described convolutional layer 122 and the maximum pooling layer 123 .
  • the capsule endoscopy image reading system 100 may further include a video clip generation unit 140 in addition to the preprocessor 110 , the convolutional neural network 120 , and the grad-CAM acquisition unit 130 described above.
  • the video clip generator 140 may generate a video clip corresponding to the capsule endoscope image based on the capsule endoscope image.
  • FIG. 2 is a block diagram illustrating an example of a preprocessor of a capsule endoscope image reading system according to the present invention.
  • the pre-processing unit 110 may include a noise removing unit 111 and an image enhancing unit 112 .
  • the noise removing unit 111 may remove noise from the capsule endoscope image input to the preprocessing unit 110 .
  • the image augmentation unit 112 may generate a plurality of augmented images by performing at least one of rotation and vertical inversion on the noise-removed capsule endoscope image.
  • FIG. 3 is a diagram illustrating an example of removing noise from an image through the preprocessor of FIG. 2 .
  • the noise removing unit 111 may remove noise including letters, numbers, and symbols recorded in the capsule endoscope image from the capsule endoscope image. This is because letters, numbers, symbols, etc. are irrelevant to determining whether a lesion exists, and the convolutional neural network 120 may interfere with determining whether a lesion exists in the capsule endoscopy image.
  • the noise removing unit 111 deletes letters, numbers, symbols, etc. indicating the time, date, and photographing equipment, etc. on the edge of the capsule endoscopy image of 576 * 576 * 3 to 512 * 512 * 3 Capsule endoscopy images of
  • FIG. 4 is a diagram illustrating an example of augmenting an image through the preprocessor of FIG. 2 .
  • the image augmentation unit 112 may generate eight augmented images by performing at least one of rotation (90 degrees/180 degrees/270 degrees) and vertical inversion on the noise-removed capsule endoscope image. have. Meanwhile, FIG. 4 describes a case in which both rotation (90 degrees/180 degrees/270 degrees) and vertical inversion are performed. However, in the present invention, the image augmentation unit 112 may perform all of the rotation and up/down inversion, or may generate less than 8 augmented images by performing only some of the rotation and up/down inversion.
  • the reason why the image augmentation unit 112 generates the augmented image by performing at least one of rotation and vertical inversion is to prevent the characteristics of the lesion present in the original capsule endoscopy image for the augmented image from being damaged in the image processing process. to be.
  • Capsule endoscopy images generally have a shape that shows images of the small intestine in a circular shape in a black background.
  • FIG. 5 is a diagram illustrating a convolutional neural network of a capsule endoscope image reading system according to the present invention.
  • the convolutional neural network 120 may extract a feature of the capsule endoscope image preprocessed through the input layer 121 through the convolution layer 122 .
  • the convolutional neural network 120 may subsample the feature extracted through the convolution layer 122 through the maximum pooling layer 123 .
  • the convolutional neural network 120 may repeat the process of subsampling a feature extracted through any one convolutional layer through any one maximum pooling layer and then inputting the result back to another convolutional layer.
  • the convolutional neural network 120 uses the result generated using the convolutional layer 122 and the maximum pooling layer 123 to generate a probability value indicating whether a lesion exists in the capsule endoscopy image through the output layer 124 . can be printed out.
  • the output layer 124 analyzes the result value generated using the convolution layer 122 and the maximum pooling layer 123 using one or more fully connected layers, and applies various transform functions (eg softmax) to it. It can be converted into a probability value indicating the presence or absence of a lesion using various transform functions (eg softmax)
  • the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection capability among the convolutional layer 122 and the maximum pooling layer 123 .
  • the doctor can check with high accuracy which part of the capsule endoscopy image where the lesion was judged to have had an influence on the judgment of the presence of the lesion through grad-CAM.
  • a CAM Class Activation Map
  • the CAM is a map that visualizes the result of calculating the sum of the weights of the feature map using the weight just before the layer that predicts the probability value. can be found If the CAM is superimposed on the capsule endoscopy image, it is possible to easily determine the area where the lesion has occurred in the capsule endoscopy image.
  • grad-CAM is acquired using the grad-CAM acquisition unit 130 to obtain a CAM result without depending on the global average pooling (GAP) layer and without modifying the structure of the convolutional neural network. Therefore, the use of grad-CAM does not impose any constraints on the structure of the convolutional neural network, so the ability to detect the presence of lesions and the ability to track the location of lesions can be improved.
  • GAP global average pooling
  • the grad-CAM may obtain an importance weight by the following equation by using the gradient of the above-described convolutional layer or the maximum pooling layer and result information passing the layer.
  • is class is the result information passing through the layer, is the value corresponding to the result information (i,j) to be observed, denotes the effect of y on A, i.e. the gradient.
  • FIG. 6 is a view showing an example of grad-CAM generated by the capsule endoscopy image reading system according to the present invention.
  • the grad-CAM for the capsule endoscopy image on the left is displayed on the right.
  • the middle part marked with a different color from the outer part had a great influence on the determination of the presence of a lesion, and it can be seen that the probability of lesion presence in the capsule endoscopy image is about 71.64%.
  • a viewer e.g. a doctor reading the capsule endoscope image can check the location of the lesion in the capsule endoscope image without separate labeling.
  • FIG. 7 is a diagram illustrating an example in which the capsule endoscopy image reading system according to the present invention determines a layer from which grad-CAM is acquired.
  • the grad-CAM acquisition unit 130 may acquire the grad-CAM from the layer determined to have the highest lesion location detection ability among one or more convolutional layers 122 and one or more maximum pooling layers 123 .
  • the highest lesion location detection ability means that the region that has an important influence on the grad-CAM to determine the lesion existence probability and the region where the lesion exists in the actual capsule endoscopy image match the most.
  • the grad-CAM acquisition unit 130 includes a first convolutional layer (CONV_1), a second convolutional layer (CONV_2), and a third convolutional layer included in the convolutional layer 122 for a capsule endoscopy image. (CONV_3), the fourth convolutional layer (CONV_4), the first maximum pooling layer (MAXP_1) included in the maximum pooling layer 123, the second maximum pooling layer (MAXP_2), and the third maximum pooling layer (MAXP_3) Based on grad-CAM, it is possible to determine the layer with the highest lesion localization ability.
  • the grad-CAM acquisition unit 130 may acquire the grad-CAM for one or more test capsule endoscopy images in order to determine the layer having the highest lesion location detection capability. If there are a plurality of capsule endoscopy images for testing, the grad-CAM acquisition unit 130 calculates, for example, the average of the lesion location detection capability of grad-CAM for each test capsule endoscope image for each layer for each layer of the lesion of each layer. It is possible to determine the location detection capability.
  • the grad-CAM acquisition unit 130 may determine that the grad-CAM acquired from the first maximum pooling layer MAXP_1 with respect to the capsule endoscopy image for testing has the highest lesion location detection capability. In this case, the grad-CAM acquisition unit 130 may then acquire the grad-CAM for the capsule endoscopy image from the first maximum pooling layer MAXP_1 among the convolutional layer 122 and the maximum pooling layer 123 .
  • FIG. 8 is a view showing an example of the structure of a video clip generated by the capsule endoscope image reading system according to the present invention.
  • the video clip generating unit 140 may generate a video clip based on a video image in which a lesion exists.
  • the presence or absence of a lesion is expressed through a probability value, so the video clip generator 140 calculates a probability value indicating the presence of a lesion using the convolutional neural network 120 described above for the capsule endoscopy image.
  • a video clip corresponding to the capsule endoscope image may be generated based on a capsule endoscope image in which a probability value indicating the presence of a lesion in the capsule endoscope image is equal to or greater than a threshold value (eg 0.8).
  • FIG. 9 is a diagram illustrating an example of a frame included in a video clip generated by the capsule endoscope image reading system according to the present invention.
  • the video clip generator 140 adds frames as much as the maximum reference value (eg 5) before and after the capsule endoscopy image in which the lesion is determined to exist (that is, the probability value is greater than or equal to the threshold value) to the capsule endoscopy image.
  • a corresponding video clip can be created.
  • the video clip generating unit 140 may add up to 5 frames before and after the capsule endoscopy image determined to exist of the lesion to the video clip. If the number of frames to be played is less than 5 (eg the 4th frame after the start), all appendable frames can be added to the video clip.
  • the video clip generator 140 may generate a video clip by combining not only the N-th frame, but also A frames before the N-th frame and A frames after the N-th frame.
  • the video clip includes not only the capsule endoscopy image determined to have a lesion, but also the capsule endoscopy image before and after it is included in the video clip. This is in order to confirm the continuous change to the endoscopic image or vice versa through the video clip.
  • a video clip consisting only of images predicted to have a lesion is unsuitable for reading because the front and back frames appear to be cut off to the viewer.
  • the aforementioned reference value may be arbitrarily determined at a level that does not cause inconvenience to the viewer.
  • FIG. 10 is a diagram illustrating an example of a frame included in two video clips generated by the capsule endoscope image reading system according to the present invention.
  • the video clip generating unit 140 generates video clip 1 which is a video clip based on the capsule endoscope image of frame M, and a video clip that is based on the capsule endoscope image of frame N. You can create video clip 2.
  • frame (M+A), which is the last frame of video clip 1 is a frame earlier than frame (N-A), which is the start frame of video clip 2. That is, video clip 1 and video clip 2 have frames overlapping each other.
  • the video clip generator 140 merges the video clip 1 and the video clip 2 instead of separately generating the video clip 1 and the video clip 2 to form one video. You can create clips.
  • FIG. 11 is a diagram illustrating a new video clip in which the two video clips of FIG. 10 are merged;
  • the video clip generator 140 may generate the video clip 3 by merging the aforementioned video clip 1 and video clip 2 .
  • the video clip 3 may include a frame (N+A), which is the most recently generated frame, from a frame (M-A) that is the earliest generated frame among frames included in the video clip 1 or the video clip 2 .
  • the video clip generating unit 140 may express a change between frame M and frame N, which is a capsule endoscopy image in which a lesion exists, through one video clip.
  • FIG. 12 is a view showing a capsule endoscope image set applied to the capsule endoscope image reading system according to the present invention.
  • the number of images included in a training image set which is a set of capsule endoscopy images used for learning the convolutional neural network 120 of the capsule endoscope image reading system, and the convolutional neural network 120
  • the ratio of the number of images included in the test image set may be determined as a preset ratio value.
  • (the number of images in the training image set):(the number of images in the test image set) may be 7:3 or 8:2.
  • the above-mentioned ratio value may be selected as a value that can increase the accuracy of the lesion existence probability the most. This ratio value may be set to a fixed value in the capsule endoscopy image reading system.
  • the input layer 121 of the convolutional neural network 120 may receive the same number of images with lesions and images without lesions. This is because, in the learning process of the convolutional neural network 120, when an image corresponding to either an image in which a lesion exists or an image without a lesion is excessively input, a bias is likely to be applied to the excessively inputted image. because it is high
  • FIG. 13 is a flowchart of a capsule endoscopy image reading method according to the present invention.
  • the capsule endoscope image reading method 1300 may include a pre-processing step ( S1310 ) of pre-processing the capsule endoscope image captured by the capsule endoscope.
  • the capsule endoscope image reading method may include an input step (S1320) of receiving the capsule endoscope image preprocessed in step S1310.
  • the capsule endoscope image reading method may include a processing step (S1330) of extracting features for the preprocessed capsule endoscope image input in step S1320 and repeatedly executing a processing operation of subsampling the extracted features.
  • the capsule endoscopy image reading method may include a grad-CAM acquisition step (S1340) of acquiring a grad-CAM (gradient class activation map) based on the result of step S1330.
  • the capsule endoscopy image reading method may include outputting a probability value indicating whether a lesion exists in the capsule endoscope image ( S1350 ).
  • the capsule endoscope image reading method 1300 may be performed through the capsule endoscope image reading system 100 described above.
  • FIG. 14 is a flowchart illustrating details of a pre-processing step of a capsule endoscopy image reading method according to the present invention.
  • the pre-processing step ( S1310 ) may include a noise removing step ( S1410 ) of removing noise from the capsule endoscopy image.
  • the preprocessing step (S1310) may include an image augmentation step (S1420) of generating a plurality of augmented images by performing at least one of rotation and vertical inversion on the capsule endoscope image from which the noise has been removed in step S1410.
  • the pre-processing step S1310 may be performed by the pre-processing unit 110 of the above-described capsule endoscope image reading system.
  • the capsule endoscopy image reading system and method described in the embodiments of the present invention can reduce the reading time of a doctor and increase the accuracy of a large amount of endoscopic images taken with the capsule endoscope.
  • the deep learning model described in the embodiments of the present invention may be a model in which artificial neural networks are stacked in multi-layered layers. That is, the deep learning model automatically learns the characteristics of the input value by learning a large amount of data from a deep neural network consisting of a multi-layered network, and through this, the network is trained to minimize the error in the objective function, that is, the prediction accuracy. is the model of
  • the deep learning model is a convolutional neural network (CNN)
  • CNN convolutional neural network
  • a deep learning model can be implemented through a deep learning framework.
  • the deep learning framework provides functions commonly used when developing deep learning models in the form of a library and plays a role in supporting the good use of system software or hardware platforms.
  • the deep learning model may be implemented using any deep learning framework that is currently public or will be released in the future.
  • the capsule endoscope image reading system described above may be implemented by a computing device including at least a portion of a processor, a memory, a user input device, and a presentation device.
  • a memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and/or data, etc. coded to perform specific tasks when executed by a processor.
  • the processor may read and execute computer-readable software, applications, program modules, routines, instructions, and/or data stored in the memory and/or the like.
  • the user input device may be a means for allowing the user to input a command to the processor to execute a specific task or to input data required for the execution of the specific task.
  • the user input device may include a physical or virtual keyboard or keypad, key button, mouse, joystick, trackball, touch-sensitive input means, or a microphone.
  • the presentation device may include a display, a printer, a speaker, or a vibrator.
  • Computing devices may include various devices such as smartphones, tablets, laptops, desktops, servers, clients, and the like.
  • the computing device may be a single stand-alone device, or may include a plurality of computing devices operating in a distributed environment comprising a plurality of computing devices cooperating with each other through a communication network.
  • the capsule endoscopy image reading method described above includes a processor, and when executed by the processor, computer readable software, applications, program modules, routines, and instructions coded to perform an image diagnosis method using a deep learning model , and/or may be executed by a computing device having a memory storing data structures, and/or the like.
  • the above-described embodiments may be implemented through various means.
  • the present embodiments may be implemented by hardware, firmware, software, or a combination thereof.
  • the image diagnosis method using the deep learning model includes one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), It may be implemented by Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers or microprocessors, and the like.
  • ASICs Application Specific Integrated Circuits
  • DSPs Digital Signal Processors
  • DSPDs Digital Signal Processing Devices
  • PLDs Programmable Logic Devices
  • FPGAs Field Programmable Gate Arrays
  • processors controllers, microcontrollers or microprocessors, and the like.
  • the capsule endoscopy image reading method may be implemented using an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented with semiconductor elements.
  • the semiconductor device may be currently used semiconductor devices, for example, SRAM, DRAM, NAND, or the like, or may be next-generation semiconductor devices, RRAM, STT MRAM, PRAM, or the like, or a combination thereof.
  • the capsule endoscope image reading method according to the embodiments is implemented using an artificial intelligence semiconductor device
  • the result (weight) of learning a deep learning model with software is transcribed into a synaptic mimic device arranged in an array or learned in an artificial intelligence semiconductor device may proceed.
  • the capsule endoscope image reading method may be implemented in the form of an apparatus, procedure, or function for performing the functions or operations described above.
  • the software code may be stored in the memory unit and driven by the processor.
  • the memory unit may be located inside or outside the processor, and may transmit and receive data to and from the processor by various known means.
  • terms such as “system”, “processor”, “controller”, “component”, “module”, “interface”, “model”, or “unit” generally refer to computer-related entities hardware, hardware and software. may mean a combination of, software, or running software.
  • the aforementioned component may be, but is not limited to, a process run by a processor, a processor, a controller, a controlling processor, an object, a thread of execution, a program, and/or a computer.
  • an application running on a controller or processor and a controller or processor can be a component.
  • One or more components may reside within a process and/or thread of execution, and the components may be located on one device (eg, a system, computing device, etc.) or distributed across two or more devices.
  • another embodiment provides a computer program stored in a computer recording medium for performing the above-described capsule endoscope image reading method.
  • Another embodiment also provides a computer-readable recording medium in which a program for realizing the above-described capsule endoscope image reading method is recorded.
  • the program recorded on the recording medium can be read by a computer, installed, and executed to execute the above-described steps.
  • the above-described program is C, C++, which the processor (CPU) of the computer can read through the device interface of the computer.
  • JAVA and may include code coded in a computer language such as machine language.
  • Such codes may include function codes related to functions defining the above-mentioned functions, etc., and may include control codes related to an execution procedure necessary for the processor of the computer to execute the above-mentioned functions according to a predetermined procedure.
  • this code may further include additional information necessary for the processor of the computer to execute the above functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. .
  • the code is transmitted to the computer's processor using the computer's communication module. It may further include a communication-related code for how to communicate with another computer or server, and what kind of information or media to transmit and receive during communication.
  • the computer-readable recording medium in which the program as described above is recorded is, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage device, etc., and also carrier wave (eg, , transmission through the Internet) may be implemented in the form of.
  • the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes can be stored and executed in a distributed manner.
  • the capsule endoscopy image reading method described with reference to FIG. 10 may also be implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer.
  • Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • the above-described capsule endoscope image reading method may be executed by an application basically installed in the terminal (which may include a program included in a platform or operating system basically mounted in the terminal), and the user may use an application store server, an application or It may be executed by an application (ie, a program) directly installed in the master terminal through an application providing server such as a web server related to the corresponding service.
  • the above-described capsule endoscopy image reading method may be implemented as an application (ie, a program) installed basically in a terminal or directly installed by a user, and may be recorded in a computer-readable recording medium such as a terminal.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Biomedical Technology (AREA)
  • Physics & Mathematics (AREA)
  • Pathology (AREA)
  • Molecular Biology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Optics & Photonics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Veterinary Medicine (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Endoscopes (AREA)

Abstract

La présente invention concerne un système et un procédé de lecture d'image d'endoscopie par capsule, le système comprenant : une unité de prétraitement pour prétraiter une image d'endoscopie par capsule capturée à l'aide d'une endoscopie par capsule ; un réseau de neurones convolutifs (CNN) à l'aide de l'image d'endoscopie par capsule prétraitée en tant qu'entrée pour déterminer si une lésion est présente dans l'image d'endoscopie par capsule ; et une unité d'acquisition de carte d'activation de classe pondérée par le gradient (grad-CAM) pour acquérir une grad-CAM pour l'image d'endoscopie par capsule.
PCT/KR2021/004735 2020-06-25 2021-04-15 Système et procédé de lecture d'images d'endoscopie par capsule WO2021261727A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020200077757A KR102359984B1 (ko) 2020-06-25 2020-06-25 캡슐 내시경 영상 판독 시스템 및 방법
KR10-2020-0077757 2020-06-25

Publications (1)

Publication Number Publication Date
WO2021261727A1 true WO2021261727A1 (fr) 2021-12-30

Family

ID=79281497

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/004735 WO2021261727A1 (fr) 2020-06-25 2021-04-15 Système et procédé de lecture d'images d'endoscopie par capsule

Country Status (2)

Country Link
KR (1) KR102359984B1 (fr)
WO (1) WO2021261727A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116542883A (zh) * 2023-07-07 2023-08-04 四川大学华西医院 一种磁控胶囊胃镜影像病灶粘膜增强系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240071188A (ko) 2022-11-15 2024-05-22 동의대학교 산학협력단 인공신경망을 이용한 캡슐 내시경 병변 검사 지원 장치 및 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101889725B1 (ko) * 2018-07-04 2018-08-20 주식회사 루닛 악성 종양 진단 방법 및 장치
KR20180136857A (ko) * 2017-06-14 2018-12-26 한국전자통신연구원 병변 영역을 판단하는 캡슐 내시경 및 수신 장치
KR20190090150A (ko) * 2018-01-24 2019-08-01 주식회사 인트로메딕 캡슐 내시경 영상의 서술자 생성 방법 및 장치, 서술자 기반 캡슐 내시경 영상 검색 방법 및 캡슐 내시경 모니터링 디바이스
KR20190128292A (ko) * 2018-05-08 2019-11-18 서울대학교산학협력단 녹내장 조기진단과 의심영역 표시 방법 및 이를 위한 시스템
KR20200070062A (ko) * 2018-12-07 2020-06-17 주식회사 포인바이오닉스 인공신경망을 이용하여 캡슐형 내시경 영상에서 병변을 감지하는 시스템 및 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20180136857A (ko) * 2017-06-14 2018-12-26 한국전자통신연구원 병변 영역을 판단하는 캡슐 내시경 및 수신 장치
KR20190090150A (ko) * 2018-01-24 2019-08-01 주식회사 인트로메딕 캡슐 내시경 영상의 서술자 생성 방법 및 장치, 서술자 기반 캡슐 내시경 영상 검색 방법 및 캡슐 내시경 모니터링 디바이스
KR20190128292A (ko) * 2018-05-08 2019-11-18 서울대학교산학협력단 녹내장 조기진단과 의심영역 표시 방법 및 이를 위한 시스템
KR101889725B1 (ko) * 2018-07-04 2018-08-20 주식회사 루닛 악성 종양 진단 방법 및 장치
KR20200070062A (ko) * 2018-12-07 2020-06-17 주식회사 포인바이오닉스 인공신경망을 이용하여 캡슐형 내시경 영상에서 병변을 감지하는 시스템 및 방법

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116542883A (zh) * 2023-07-07 2023-08-04 四川大学华西医院 一种磁控胶囊胃镜影像病灶粘膜增强系统
CN116542883B (zh) * 2023-07-07 2023-09-05 四川大学华西医院 一种磁控胶囊胃镜影像病灶粘膜增强系统

Also Published As

Publication number Publication date
KR20220000437A (ko) 2022-01-04
KR102359984B1 (ko) 2022-02-08

Similar Documents

Publication Publication Date Title
WO2021261727A1 (fr) Système et procédé de lecture d'images d'endoscopie par capsule
WO2017213398A1 (fr) Modèle d'apprentissage pour détection de région faciale saillante
WO2018212494A1 (fr) Procédé et dispositif d'identification d'objets
WO2020130309A1 (fr) Dispositif de masquage d'image et procédé de masquage d'image
WO2019132168A1 (fr) Système d'apprentissage de données d'images chirurgicales
WO2020122432A1 (fr) Dispositif électronique et procédé d'affichage d'une image tridimensionnelle de celui-ci
WO2019235828A1 (fr) Système de diagnostic de maladie à deux faces et méthode associée
WO2022131642A1 (fr) Appareil et procédé pour déterminer la gravité d'une maladie sur la base d'images médicales
WO2021230534A1 (fr) Appareil de prédiction de lésion orbitaire et périorbitaire et procédé de prédiction associé
WO2022146050A1 (fr) Procédé et système d'entraînement d'intelligence artificielle fédéré pour le diagnostic de la dépression
WO2021101045A1 (fr) Appareil électronique et procédé de commande associé
WO2021246770A1 (fr) Procédé et système de lecture automatique d'images radiographiques en temps réel sur la base d'intelligence artificielle
KR20220078495A (ko) 캡슐 내시경 영상 기반 소장 병변 판독 방법, 장치 및 프로그램
WO2023075508A1 (fr) Dispositif électronique et procédé de commande associé
WO2024019337A1 (fr) Procédé et appareil d'amélioration de vidéo
WO2022119364A1 (fr) Procédé, dispositif et programme de déchiffrement de lésion de l'intestin grêle basé sur une image d'endoscopie par capsule
WO2022231200A1 (fr) Procédé d'entraînement pour l'entraînement d'un réseau de neurones artificiels pour déterminer une zone de lésion du cancer du sein, et système informatique le réalisant
WO2022004970A1 (fr) Appareil et procédé d'entraînement de points clés basés sur un réseau de neurones artificiels
WO2021096136A1 (fr) Dispositif électronique et son procédé de commande
WO2021015490A2 (fr) Procédé et dispositif de détection d'une zone spécifique d'une image
WO2021015489A2 (fr) Procédé et dispositif d'analyse d'une zone d'image singulière à l'aide d'un codeur
WO2022158694A1 (fr) Procédé permettant de traiter une image de tissu pathologique et appareil associé
WO2019168280A1 (fr) Procédé et dispositif permettant de déchiffrer une lésion à partir d'une image d'endoscope à capsule à l'aide d'un réseau neuronal
WO2023149649A1 (fr) Dispositif électronique et procédé destinés à améliorer la qualité d'image
WO2023182796A1 (fr) Dispositif d'intelligence artificielle permettant de détecter des produits défectueux sur la base d'images de produit et procédé associé

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21829218

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21829218

Country of ref document: EP

Kind code of ref document: A1