WO2022119364A1

WO2022119364A1 - Capsule endoscopy image-based small intestine lesion deciphering method, device, and program

Info

Publication number: WO2022119364A1
Application number: PCT/KR2021/018171
Authority: WO
Inventors: 이종혁; 이연주
Original assignee: 주식회사 서르; 부산대학교 산학협력단
Priority date: 2020-12-03
Filing date: 2021-12-03
Publication date: 2022-06-09

Abstract

The present invention relates to a capsule endoscopy image-based small intestine lesion deciphering method, device, and program whereby small intestine lesions can be deciphered on the basis of image data acquired from a capsule endoscopy. The method comprises the steps of: acquiring capsule endoscopy image data; inputting the capsule endoscopy image data to a pre-trained first neural network model to identify a first transition portion between the stomach and small intestine and a second transition portion between the small intestine and large intestine; pre-processing small intestine image data between the identified first transition portion and second transition portion by separating the small intestine image data from the capsule endoscopy image data; deciphering small intestine lesions by inputting the pre-processed small intestine image data to a pre-trained second neural network model; and generating result information on the small intestine image data on the basis of the state of the deciphered small intestine lesions.

Description

Capsule endoscopy image-based small intestine lesion reading method, device and program

The present invention relates to a method for reading a small intestine lesion, and more particularly, to a capsule endoscopy image-based small intestine lesion reading method, apparatus, and program capable of reading a small intestine lesion based on image data obtained from a capsule endoscope.

Recently, the case of using a capsule endoscope to examine the digestive organs in the human body is increasing.

Unlike the conventional endoscope, the capsule endoscope has the advantage that when the patient swallows the capsule, the image is taken while passing through organs such as the stomach, small intestine, and large intestine, so that the patient does not have any objection to the endoscope.

However, since it takes about 10 hours for the capsule to take an endoscopic image, and tens of thousands of frames of images taken during this time, from the point of view of the medical staff who diagnose the patient by checking the capsule endoscopic image, the small intestine There was a problem in that it was not easy to accurately diagnose the lesion of

In order to solve this problem, an artificial intelligence model that can read directly from the capsule endoscopy image was used, but this method may cause an overfitting problem due to insufficient data or improperly collected data, It is impossible to check which part the intelligent model has seen and judged, so the utilization may be reduced.

Therefore, in the future, there is a need to develop a method for reading small intestine lesions using an artificial intelligence model to assist doctors in diagnosing small intestine lesions based on capsule endoscopy images.

One object of the present invention to solve the above problems is to identify the first transitional part of the stomach and small intestine and the second transitional part of the small intestine and the large intestine from the capsule endoscopy image data, segmenting the small intestine image data, and preprocessing the small intestine image data To provide a method, apparatus and program for reading small intestine lesions that can increase the accuracy of small intestine lesion reading in a minimum amount of time by inputting .

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a method for reading a small intestine lesion performed by a computing device according to an embodiment of the present invention includes: acquiring capsule endoscopy image data; inputting the capsule endoscopy image data into a pre-trained first neural network model to identify a first transitional portion of the stomach and small intestine and a second transitional portion of the small intestine and large intestine; dividing and pre-processing small intestine image data between the identified first and second transitional portions from the capsule endoscopy image data; reading the small intestine lesion by inputting the pre-processed small intestine image data into a pre-trained second neural network model; and generating result information on the small intestine image data based on the read small intestine lesion state.

In an embodiment, in the identifying step, when identifying the first transitional portion of the stomach and small intestine, a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to the structural characteristics of the first transitional portion, and a color of the mucous membrane When the combination condition of at least two or more of the third conditions corresponding to the change is satisfied, the stomach and small intestine are recognized as the first transitional part.

In an embodiment, the pre-processing step comprises segmenting small intestine image data based on the identified first transitional portion and second transitional portion from the capsule endoscopy image data, and quantifying the ratio of the mucosal region of the small intestine image data. do.

In an embodiment, in the pre-processing step, when quantifying the image ratio, a weight is applied to the ratio of the mucosal region seen in the frame unit of the small intestine image data and the ratio of the mucosal region seen in the moving picture based on the imaging conditions of the capsule endoscope. characterized by quantification.

In an embodiment, the imaging conditions of the capsule endoscope include at least one of a progress speed of the capsule endoscope and a imaging speed of the capsule endoscope.

In an embodiment, in the reading step, when the small intestine lesion is read, if the small intestine lesion is an area in the small intestine image data, the small intestine lesion is read by reproducing the image reproduction speed as a reference speed, and the small intestine lesion does not appear. If it is an area that is not, the small intestine lesion is read by reproducing the image reproduction speed faster than the reference speed.

In an embodiment, the generating step is characterized in that the result information for visualizing the small intestine lesion reading result is generated based on the read small intestine lesion state. At this time, when generating result information to visualize the small intestine lesion reading result,

(where VS is the visualization scale, M is the total number of frames of capsule endoscopy image data, size (image) is the pixel size of the image, Ci is the set of clean area pixels, Di is the set of dark area pixels, Fi is the float/bubble area pixels It is characterized in that the result information for visualizing the small intestine lesion reading result is generated by a formula consisting of a set).

On the other hand, when the computer program stored in the computer-readable storage medium according to an embodiment of the present invention is executed by one or more processors, the following operations for reading small intestine lesions are performed, and the operations are: the operation of acquiring the capsule endoscope image data and inputting the capsule endoscopy image data into a pre-trained first neural network model to identify the first transition part of the stomach and small intestine and the second transition part of the small intestine and the large intestine, the identified from the capsule endoscope image data The operation of dividing and preprocessing small intestine image data between the first transition unit and the second transition unit, the operation of inputting the preprocessed small intestine image data into a pre-trained second neural network model to read the small intestine lesion, and the read small intestine lesion and generating result information on the small intestine image data based on the state.

In addition, a computing device according to an embodiment of the present invention is a computing device for providing a method for reading a small intestine lesion, including a processor including one or more cores, and a memory, wherein the processor acquires capsule endoscopy image data and , input the capsule endoscopy image data into the pre-trained first neural network model to identify the first transitional portion of the stomach and small intestine and the second transitional portion between the small and large intestine, and the identified first transitional portion and the second transitional portion from the capsule endoscopy image data The small intestine image data is divided and pre-processed between the two transition parts, the small intestine lesion is read by inputting the pre-processed small intestine image data to a pre-trained second neural network model, and the small intestine image is based on the read small intestine lesion state. It is characterized in that the result information about the data is generated.

A computer program providing a method for reading a small intestine lesion according to another embodiment of the present invention for solving the above-described problems is stored in a medium in combination with a computer that is hardware to perform any one of the methods described above.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium for recording a computer program for executing the method may be further provided.

As described above, according to the present invention, the small intestine image data is segmented by identifying the first transitional part of the stomach and small intestine and the second transitional part of the small intestine and the large intestine from the capsule endoscopy image data, and the preprocessed small intestine image data is applied to a neural network model that has been pre-learned. By reading the small intestine lesion by input, it is possible to increase the accuracy of the small intestine lesion reading in a minimum time.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a block diagram of a computing device that performs an operation for providing a method for reading a small intestine lesion, according to an embodiment of the present invention.

2 and 3 are diagrams for explaining a process of identifying the transitional portion of the stomach and small intestine from the capsule endoscopy image data.

4 is a diagram illustrating a boundary region between the stomach and small intestine and a boundary region between the small intestine and the large intestine in the entire capsule endoscopy image, respectively.

5 and 6 are diagrams for explaining a process of reading a small intestine lesion through a mucosal region of a small intestine image having floating matter and bubbles.

7 is a diagram showing result information of visualizing a small intestine lesion reading, according to an embodiment of the present invention.

8 is a flowchart illustrating a method for reading a small intestine lesion, according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Before the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that, since the description of the term is for the purpose of helping the understanding of the present specification, it is not used in the meaning of limiting the technical idea of the present invention unless explicitly described as limiting the present invention.

In this specification, neural network, artificial neural network, and network function may often be used interchangeably.

The term "image" or "image data" as used throughout the present description and claims refers to multidimensional data composed of discrete image elements (eg, pixels in a two-dimensional image), in other words, ( For example, a term that refers to a visible object (eg, displayed on a video screen) or a digital representation of the object (eg, a file corresponding to a pixel output of a CT, MRI detector, etc.).

For example, “image” or “imaging” can be defined as computed tomography (CT), magnetic resonance imaging (MRI), capsule endoscopic imaging, ultrasound or any other medical treatment known in the art. It may be a medical image of the subject collected by the imaging system. The image does not necessarily have to be provided in a medical context, but may be provided in a non-medical context, for example, there may be X-ray imaging for security screening.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a generic term for various standards used for digital image expression and communication in medical devices. The DICOM standard is published by a joint committee formed by the American Society of Radiological Medicine (ACR) and the National Electrical Engineers Association (NEMA).

In addition, throughout the detailed description and claims of the present invention, 'Picture Archiving and Communication System (PACS)' is a term that refers to a system that stores, processes, and transmits in accordance with the DICOM standard, capsule endoscope, X Medical imaging images acquired using digital medical imaging equipment such as line, CT, and MRI are stored in DICOM format and can be transmitted to terminals inside and outside the hospital through a network, to which reading results and medical records can be added. .

Also, throughout this specification, a neural network, a neural network network, and a network function may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may be generally referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network is configured to include at least two or more nodes. Nodes (or neurons) constituting the neural networks may be interconnected by one or more “links”.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In an embodiment of the present invention, the computing device 100 may include other components for performing the computing environment of the computing device 100 , and only some of the disclosed components may configure the computing device 100 .

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

In the present invention, the processor 110 obtains capsule endoscopy image data, inputs the capsule endoscopy image data to a pre-trained first neural network model, and identifies the first transitional portion of the stomach and small intestine and the second transitional portion of the small intestine and large intestine The small intestine image data is divided and pre-processed between the first transitional part and the second transitional part identified from the capsule endoscopy image data, and the small intestine lesion is read by inputting the preprocessed small intestine image data into the pre-trained second neural network model, In addition, result information on the small intestine image data may be generated based on the read small intestine lesion state.

Here, the capsule endoscopy image data may include image data of the oral cavity, esophagus, stomach, small intestine, and large intestine inside the human body.

Next, the processor 110 reads from the first frame of the capsule endoscope image data in the posterior frame direction to identify the first transition portion of the stomach and small intestine, and reads from the last frame of the capsule endoscopy image data in the anterior frame direction to the small intestine and the small intestine. A second transitional part of the large intestine can be identified.

For example, when identifying the first transitional portion of the stomach and small intestine, the processor 110 may identify the first transitional portion of the stomach and small intestine based on a change in the shape of the mucosal surface.

Here, when the processor 110 identifies the first transition portion of the stomach and small intestine, if the roughness of the shape of the mucosal surface of the current read data increases, it may be recognized as the first transition portion of the stomach and small intestine.

That is, when the processor 110 identifies the first transition part of the stomach and small intestine, if the shape of the mucosal surface of the current read data is a smooth shape, it is recognized as the stomach area, and if the shape of the mucosal surface of the current read data is a carpet shape, it is recognized as the small intestine area. can recognize

As another example, when identifying the first transitional portion of the stomach and small intestine, the processor 110 may identify the first transitional portion of the stomach and small intestine based on structural characteristics of the first transitional portion.

Here, when the processor 110 identifies the first transitional portion of the stomach and small intestine, if the current read data has a characteristic structure of a pyloric shape, it may be recognized as the first transitional portion of the stomach and small intestine.

As another example, when identifying the first transitional portion of the stomach and small intestine, the processor 110 may identify the first transitional portion of the stomach and small intestine based on a color change of the mucous membrane.

Here, when the processor 110 identifies the first transitional portion of the stomach and small intestine, if the mucous membrane color of the current read data changes to a specific color, the processor 110 may identify the first transitional portion of the stomach and small intestine.

That is, when identifying the first transition portion of the stomach and small intestine, the processor 110 may identify the first transition portion of the stomach and small intestine if the color of the mucous membrane of the current read data changes to yellow.

In addition, when the processor 110 identifies the first transition portion of the stomach and small intestine, a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to a structural characteristic of the first transition portion, and a color change of the mucosa When the combination condition of at least two or more of the third conditions is satisfied, it may be recognized as the first transitional part of the stomach and small intestine.

Also, when the processor 110 identifies the second transition between the small and large intestines, the processor 110 may also identify the second transition between the small and large intestines based on a change in diameter.

Here, when the processor 110 identifies the second transition portion between the small intestine and the large intestine, if a change in the size of the diameter of the current read data appears, it may be recognized as the second transition portion between the small intestine and the large intestine.

Next, the processor 110 may segment the small intestine image data based on the first transitional portion and the second transitional portion identified from the capsule endoscopy image data, and quantify the ratio of the mucosal region of the small intestine image data.

Here, when quantifying the image ratio, the processor 110 may quantify by applying a weight to the ratio of the mucosal region seen in a frame unit of the small intestine image data and the ratio of the mucosal region seen in the video based on the imaging conditions of the capsule endoscope. .

As an example, the imaging condition of the capsule endoscope may include at least one of the progress speed of the capsule endoscope and the imaging speed of the capsule endoscope, which is only an example and is not limited thereto.

That is, the processor 110 gives a low weight to several image frames photographed because the progress speed of the capsule endoscope moves slowly, and gives a high weight to the image frames photographed because the progress speed of the capsule endoscope moves quickly In this way, it can be corrected similarly to the captured image by moving at the same speed.

Also, when quantifying the image ratio, the processor 110 may quantify the ratio of the mucosal region of the small intestine image data by inputting the small intestine image data and the imaging information of the capsule endoscope into the third neural network model learned in advance.

Then, when reading the small intestine lesion, the processor 110 reads the small intestine lesion by reproducing the image reproduction speed as a reference speed if the small intestine lesion is an area in the small intestine image data, and if the small intestine lesion does not appear, the image reproduction speed The small intestine lesion can be read by regenerating faster than the reference rate.

Here, when reading the small intestine lesion, the processor 110 reads the small intestine lesion by reproducing the image reproduction speed as the reference speed if the small intestine lesion is a region with a high probability of appearing in the small intestine image data, and the probability that the small intestine lesion does not appear is increased. In a high area, the small intestine lesion can be read by reproducing the image faster than the reference speed.

In addition, the processor 110 may generate result information for visualizing the small intestine lesion reading result based on the read small intestine lesion state.

Here, the processor 110, when generating the result information to visualize the small intestine lesion reading result,

(where VS is the visualization scale, M is the total number of frames of capsule endoscopy image data, size (image) is the pixel size of the image, Ci is the set of clean area pixels, Di is the set of dark area pixels, Fi is the float/bubble area pixels set), it is possible to generate result information for visualizing the result of reading the small intestine lesion.

Also, the processor 110 may classify a class for the small intestine lesion based on the read small intestine lesion state, and generate result information for visualizing the small intestine lesion reading result corresponding to the classified class.

Also, the above-described neural network model may be a deep neural network. Throughout this specification, the terms neural network, network function, and neural network may be used interchangeably. A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify the latent structures of data. In other words, it can identify the potential structure of photos, texts, videos, voices, and music (e.g., what objects are in the photos, what the text and emotions are, what the texts and emotions are, etc.) . Deep neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), and a deep belief network (DBN). , Q network, U network, Siamese network, and the like.

A convolutional neural network is a kind of deep neural network, and includes a neural network including a convolutional layer. Convolutional neural networks are a type of multilayer perceptrons designed to use minimal preprocessing. A CNN may be composed of one or several convolutional layers and artificial neural network layers combined therewith. CNN may additionally utilize weights and pooling layers. Thanks to this structure, CNN can fully utilize the input data of the two-dimensional structure. A convolutional neural network may be used to recognize an object in an image. A convolutional neural network may process image data by representing it as a matrix having a dimension. For example, in the case of RGB (red-green-blue) encoded image data, each R, G, and B color may be represented as a two-dimensional (eg, two-dimensional image) matrix. That is, the color value of each pixel of the image data may be a component of the matrix, and the size of the matrix may be the same as the size of the image. Accordingly, the image data can be represented by three two-dimensional matrices (three-dimensional data array).

In a convolutional neural network, a convolutional process (input/output of a convolutional layer) can be performed by moving the convolutional filter and multiplying the convolutional filter and matrix components at each position of the image. The convolutional filter may be composed of an n*n matrix. A convolutional filter may be composed of a fixed type filter that is generally smaller than the total number of pixels in an image. That is, when an m*m image is input as a convolutional layer (eg, a convolutional layer having a size of n*n convolutional filters), a matrix representing n*n pixels including each pixel of the image This convolutional filter can be multiplied with a component (ie, a product of each component of a matrix). A component matching the convolutional filter may be extracted from the image by multiplication with the convolutional filter. For example, a 3*3 convolutional filter for extracting upper and lower linear components from an image can be configured as [[0,1,0], [0,1,0], [0,1,0]] can When the 3*3 convolutional filter for extracting the upper and lower linear components from the image is applied to the input image, the upper and lower linear components matching the convolutional filter may be extracted and output from the image. The convolutional layer may apply a convolutional filter to each matrix (ie, R, G, B colors in the case of R, G, and B coded images) for each channel representing the image. The convolutional layer may extract a feature matching the convolutional filter from the input image by applying the convolutional filter to the input image. The filter value of the convolutional filter (ie, the value of each component of the matrix) may be updated by backpropagation in the learning process of the convolutional neural network.

A subsampling layer may be connected to the output of the convolutional layer to simplify the output of the convolutional layer, thereby reducing memory usage and computation amount. For example, if you input the output of the convolutional layer to a pooling layer with a 2*2 max pooling filter, you can compress the image by outputting the maximum value included in each patch for every 2*2 patch in each pixel of the image. can The above-described pooling may be a method of outputting a minimum value from a patch or an average value of a patch, and any pooling method may be included in the present invention.

A convolutional neural network may include one or more convolutional layers and subsampling layers. The convolutional neural network may extract features from an image by repeatedly performing a convolutional process and a subsampling process (eg, the aforementioned max pooling, etc.). Through iterative convolutional and subsampling processes, neural networks can extract global features from images.

An output of the convolutional layer or the subsampling layer may be input to a fully connected layer. A fully connected layer is a layer in which all neurons in one layer and all neurons in neighboring layers are connected. The fully connected layer may refer to a structure in which all nodes of each layer are connected to all nodes of other layers in a neural network.

According to an embodiment of the present invention, the processor 110 may be composed of one or more cores, and a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU) ), data analysis such as a tensor processing unit (TPU), and a processor for deep learning. The processor 110 may read a computer program stored in the memory 130 to perform data processing for machine learning according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 110 may perform an operation for learning the neural network. The processor 110 performs learning of the neural network such as processing of input data for learning in deep learning (DL), extraction of features from input data, calculation of errors, and weight update of the neural network using backpropagation. calculations can be performed for At least one of a CPU, a GPGPU, and a TPU of the processor 110 may process learning of a network function. For example, the CPU and the GPGPU can process learning of a network function and data classification using the network function. Also, in an embodiment of the present invention, learning of a network function and data classification using the network function may be processed by using the processors of a plurality of computing devices together. In addition, the computer program executed in the computing device according to an embodiment of the present invention may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present invention, the memory 130 may store a computer program for performing small intestine lesion reading and providing a small intestine lesion reading result, and the stored computer program may be read and driven by the processor 120 . have. The memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present invention, the memory 130 includes a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, a memory card type). SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Memory (PROM) read-only memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The description of the above-described memory is only an example, and is not limited thereto.

The network unit 150 according to an embodiment of the present invention may transmit and receive information on a result of reading a small intestine lesion, etc. with another computing device, a server, and the like. In addition, the network unit 150 may enable communication between a plurality of computing devices so that operations for reading a pneumonia or learning a model are performed distributedly in each of the plurality of computing devices. The network unit 150 may enable communication between a plurality of computing devices to perform distributed processing of operations for reading small intestine lesions or learning a model using a network function.

The network unit 150 according to an embodiment of the present invention may operate based on any type of wired/wireless communication technology currently used and implemented, such as short-distance (short-range), long-distance, wired and wireless, and other networks. can also be used in

The computing device 100 of the present invention may further include an output unit and an input unit.

The output unit according to an embodiment of the present invention may display a user interface (UI) for providing a result of reading a small intestine lesion. The output unit may output any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

In an embodiment of the present invention, the output unit is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) , it may include at least one of a flexible display (flexible display), a three-dimensional display (3D display). Some of these display modules may be configured as a transparent type or a light transmission type so that the outside can be seen through them. This may be referred to as a transparent display module, and a representative example of the transparent display module is a transparent OLED (TOLED).

The input unit according to an embodiment of the present invention may receive a user input. The input unit may include a key and/or buttons on a user interface for receiving a user input, or a physical key and/or buttons. A computer program for controlling the display according to embodiments of the present invention may be executed according to a user input through the input unit.

The input unit according to embodiments of the present invention may receive a signal by sensing a user's button manipulation or touch input, or may receive a user's voice or motion through a camera or a microphone and convert it into an input signal. For this, speech recognition technology or motion recognition technology may be used.

The input unit according to embodiments of the present invention may be implemented as an external input device connected to the computing device 100 . For example, the input device may be at least one of a touch pad, a touch pen, a keyboard, and a mouse for receiving a user input, but this is only an example and is not limited thereto.

The input unit according to an embodiment of the present invention may recognize a user touch input. The input unit according to an embodiment of the present invention may have the same configuration as the output unit. The input unit may be configured as a touch screen configured to receive a user's selection input. For the touch screen, any one of a contact capacitive method, an infrared light sensing method, a surface ultrasonic wave (SAW) method, a piezoelectric method, and a resistive film method may be used. The detailed description of the touch screen described above is only an example according to an embodiment of the present invention, and various touch screen panels may be employed in the computing device 100 . The input unit configured as a touch screen may include a touch sensor. The touch sensor may be configured to convert a change in pressure applied to a specific portion of the input unit or capacitance generated at a specific portion of the input unit into an electrical input signal. The touch sensor may be configured to detect not only the touched position and area, but also the pressure at the time of the touch. When there is a touch input to the touch sensor, a signal(s) corresponding thereto is sent to the touch controller. The touch controller may process the signal(s) and then send the corresponding data to the processor 110 . Accordingly, the processor 110 can recognize which area of the input unit has been touched, and the like.

In an embodiment of the present invention, the server may include other components for performing a server environment of the server. The server may include any type of device. The server is a digital device, and may be a digital device equipped with a processor and having a computing capability, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone.

A server (not shown) that performs an operation for providing a user interface displaying a pneumonia reading result according to an embodiment of the present invention to a user terminal may include a network unit, a processor, and a memory.

The server may generate a user interface according to embodiments of the present invention. The server may be a computing system that provides information to a client (eg, a user terminal) through a network. The server may transmit the generated user interface to the user terminal. In this case, the user terminal may be any type of computing device 100 that can access the server. The processor of the server may transmit the user interface to the user terminal through the network unit. The server according to embodiments of the present invention may be, for example, a cloud server. The server may be a web server that processes a service. The above-described types of servers are merely examples and are not limited thereto.

As described above, in the present invention, the small intestine image data is segmented by identifying the first transition portion of the stomach and small intestine and the second transition portion of the small intestine and the large intestine from the capsule endoscopy image data, and the pre-processed small intestine image data is input to the pre-trained neural network model. Thus, by reading the small intestine lesion, it is possible to increase the accuracy of the small intestine lesion reading in a minimum time.

As shown in FIGS. 2 and 3 , in the present invention, by inputting capsule endoscopy image data into a pre-trained neural network model, it is possible to identify the first junction of the stomach and small intestine and the second junction of the small intestine and the large intestine.

As shown in FIG. 2 , in the present invention, when identifying the first transitional portion of the stomach and small intestine, it is possible to identify the first transitional portion of the stomach and small intestine based on a change in the shape of the mucosal surface.

Here, in the present invention, when the first transition of the stomach and small intestine is identified, if the roughness of the shape of the mucosal surface of the current read data increases, it can be recognized as the first transition of the stomach and small intestine.

That is, according to the present invention, when the first transition part of the stomach and small intestine is identified, if the shape of the mucosal surface of the current read data (left) is smooth, it is recognized as the stomach region, and the shape of the mucosal surface (right) of the current read data is the carpet shape. This can be recognized as the small intestine area.

That is, in the small intestine region, the mucosal surface may have a carpet-like shape due to villi.

Also, as shown in FIG. 3 , when the present invention identifies the first transitional portion of the stomach and small intestine, the first transitional portion of the stomach and small intestine may be identified based on the structural characteristics of the first transitional portion.

Here, in the present invention, when the first transitional portion of the stomach and small intestine is identified, if the current read data has a characteristic structure of a pylorus shape, it can be recognized as the first transitional portion of the stomach and small intestine.

Also, in the present invention, when identifying the first transitional portion of the stomach and small intestine, the first transitional portion of the stomach and small intestine may be identified based on the color change of the mucous membrane.

Here, in the present invention, when the first transitional portion of the stomach and small intestine is identified, if the color of the mucous membrane of the current read data is changed to a specific color, the first transitional portion of the stomach and small intestine can be identified.

That is, according to the present invention, when the first transitional portion of the stomach and small intestine is identified, if the color of the mucous membrane of the current read data changes to yellow, the first transitional portion of the stomach and small intestine can be identified.

In addition, the present invention provides a first condition corresponding to a change in the shape of a mucosal surface, a second condition corresponding to a structural characteristic of the first transition, and a third condition corresponding to a color change of the mucosa, when identifying the first transitional region of the stomach and small intestine When a combination condition of at least two or more of the conditions is satisfied, it may be recognized as the first transitional part of the stomach and small intestine.

As shown in FIG. 4 , in the present invention, the boundary region between the stomach and small intestine and the boundary region between the small intestine and the large intestine can be identified by inputting the entire image data of the capsule endoscope into a pre-trained neural network model.

The present invention identifies the boundary region of the stomach and small intestine based on a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to a structural feature, and a third condition corresponding to a color change of the mucosa, and changes in diameter and size Based on this, the boundary region between the small and large intestine can be identified.

And, in the present invention, if the boundary region of the stomach and small intestine and the boundary region of the small intestine and the large intestine are identified from the entire capsule endoscopy image, the small intestine image between the boundary region of the stomach and small intestine and the boundary region of the small and large intestine can be divided and pre-processed. have.

Therefore, in the present invention, when reading a small intestine lesion, an image of an unnecessary area is removed and only an image of a small intestine area necessary for reading a small intestine lesion is divided, thereby minimizing the time to read a small intestine lesion and providing accurate and fast reading result information. can create

5 and 6 are diagrams for explaining a process of reading a small intestine lesion through a mucosal region of a small intestine image with floats and bubbles, and FIG. 7 is a visualization of the small intestine lesion reading according to an embodiment of the present invention. It is a drawing showing the result information.

5 and 6 , according to the present invention, the small intestine lesion can be read by inputting the preprocessed small intestine image data into the pre-trained neural network model.

Here, in the small intestine image, the lesions may exist in a mixture of bubbles and floaters, or they may exist individually.

In the present invention, when reading small intestine lesions, classifying the classes into normal, unreadable and lesions, reading the lesions by including bubbles and floats in the normal category, quantifying the foam and floating regions as unreadable regions, results in information may provide.

In addition, in the present invention, when the small intestine lesion is read, when the input image is divided into a black color clean area, green color dark area, and blue color float/bubble area, a blue color float/bubble area and a green color area The small intestine lesion can be read through the black-colored clean area while excluding the area including the dark area.

In addition, in the present invention, if the small intestine lesion appears in the small intestine image, the image reproduction speed is reproduced at the reference speed to read the small intestine lesion, and if the small intestine lesion does not appear, the image reproduction speed is reproduced faster than the reference speed to reproduce the small intestine lesion. can be read quickly and accurately in the minimum time.

Here, the present invention reads the small intestine lesion by reproducing the image reproduction speed as the reference speed when there is a high probability that small intestine lesions appear in the area of the small intestine image. It can also regenerate faster to read small bowel lesions.

And, as shown in FIG. 7 , the present invention may generate result information for visualizing the small intestine lesion reading result based on the read small intestine lesion state.

According to the present invention, the input image may be generated by visualizing result information into a GT (Ground Truth) image including a black color clean area, a green color dark area, and a blue color float/bubble area and a prediction image.

Here, the present invention, when generating the result information to visualize the small intestine lesion reading result,

In addition, the present invention classifies a class (normal, unreadable, lesion) for a small intestine lesion based on the read small intestine lesion state, and generates result information for visualizing the small intestine lesion reading result corresponding to the classified class. connect.

As shown in FIG. 8 , in the present invention, capsule endoscopy image data may be acquired ( S10 ).

And, according to the present invention, by inputting the capsule endoscopy image data into the pre-trained first neural network model, the first transition of the stomach and small intestine and the second transition of the small intestine and the large intestine can be identified (S20).

Here, the present invention identifies the first transitional portion of the stomach and small intestine by reading from the first frame of the capsule endoscopy image data in the posterior frame direction, and reading from the last frame of the capsule endoscopy image data in the anterior frame direction. A second transition may be identified.

In addition, the present invention can identify the first junction of the stomach and small intestine based on the change in the shape of the mucosal surface.

Optionally, the present invention may identify the first transitional portion of the stomach and small intestine based on structural features of the first transitional portion.

Alternatively, the present invention may identify the first transitional portion of the stomach and small intestine based on the color change of the mucosa.

In addition, the present invention provides a combination of at least two or more of a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to the structural characteristics of the first transition part, and a third condition corresponding to a color change of the mucosa. It can also be recognized as the first transitional part of the stomach and small intestine.

In addition, the present invention may identify the second transition between the small and large intestines based on the change in diameter and size.

Next, according to the present invention, the small intestine image data between the first and second transition parts identified from the capsule endoscopy image data may be divided and pre-processed (S30).

Here, according to the present invention, small intestine image data may be segmented based on the first transitional portion and the second transitional portion identified from the capsule endoscopy image data, and the ratio of the mucosal region of the small intestine image data may be quantified.

That is, the present invention can be quantified by applying a weight to the ratio of the mucosal region seen in each frame of the small intestine image data and the ratio of the mucosal region seen in the video based on the imaging conditions of the capsule endoscope.

As an example, the imaging condition of the capsule endoscope may include at least one of a progress speed of the capsule endoscope and a imaging speed of the capsule endoscope, which is only an example, and is not limited thereto.

Also, according to the present invention, when quantifying the image ratio, it is possible to quantify the ratio of the mucosal region of the small intestine image data by inputting the image data of the small intestine and the imaging information of the capsule endoscope into the third neural network model that has been learned in advance.

Next, according to the present invention, the small intestine lesion can be read by inputting the pre-processed small intestine image data to the pre-trained second neural network model (S40).

Here, in the present invention, if the small intestine lesion appears in the small intestine image data, the image reproduction speed is reproduced at the reference speed to read the small intestine lesion, and if the small intestine lesion does not appear, the image reproduction speed is reproduced faster than the reference speed. The lesion can be read.

In addition, in the present invention, if the small intestine lesion is a region with a high probability of appearing in the small intestine image data, the image reproduction speed is reproduced as the reference speed to read the small intestine lesion. It regenerates faster and can read small bowel lesions.

And, according to the present invention, result information on the small intestine image data may be generated based on the read small intestine lesion state (S50).

Here, according to the present invention, it is possible to generate result information for visualizing the small intestine lesion reading result based on the read small intestine lesion state.

In addition, the present invention may classify a class for a small intestine lesion based on the read small intestine lesion state, and generate result information for visualizing the small intestine lesion reading result according to the classified class.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The above-described program is C, C++, JAVA, machine language, etc. that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program It may include code (Code) coded in the computer language of Such code may include functional code related to a function defining functions necessary for executing the methods, etc., and includes an execution procedure related control code necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected to a network, and a computer-readable code may be stored in a distributed manner.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains know that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims

A method performed by a computing device, comprising:

acquiring capsule endoscopy image data;

inputting the capsule endoscopy image data into a pre-trained first neural network model to identify a first transitional portion of the stomach and small intestine and a second transitional portion of the small and large intestine;

dividing and pre-processing small intestine image data between the identified first and second transitional portions from the capsule endoscopy image data;

reading the small intestine lesion by inputting the pre-processed small intestine image data into a pre-trained second neural network model; and

and generating result information on the small intestine image data based on the read small intestine lesion state.
According to claim 1,

The identification step is

at least two of a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to a structural characteristic of the first transition, and a third condition corresponding to a color change of the mucosa, when identifying the first transition portion of the stomach and small intestine A method for reading a small intestine lesion, characterized in that when the combination condition is satisfied, it is recognized as the first transition part of the stomach and small intestine.
According to claim 1,

The pre-processing step is

Small intestine lesion reading method, characterized in that the small intestine image data is segmented based on the identified first transitional portion and second transitional portion from the capsule endoscopy image data, and the ratio of the mucosal region of the small intestine image data is quantified.
4. The method of claim 3,

The pre-processing step is

When quantifying the image ratio, the small intestine lesion reading method, characterized in that the quantification is performed by applying a weight to the ratio of the mucosal area seen in the frame unit of the small intestine image data and the ratio of the mucosal area seen in the video based on the imaging conditions of the capsule endoscope. .
5. The method of claim 4,

The photographing conditions of the capsule endoscope are,

The small intestine lesion reading method, characterized in that it includes at least one of the progress speed of the capsule endoscope and the imaging speed of the capsule endoscope.
According to claim 1,

The reading step is

When reading the small intestine lesion, if the small intestine lesion is an area in the small intestine image data, the image reproduction speed is reproduced as the reference speed to read the small intestine lesion, and if the small intestine lesion does not appear, the image reproduction speed is used as the reference A method for reading a small intestine lesion, characterized in that the small intestine lesion is read by regenerating faster than the speed.
According to claim 1,

The generating step is

Small intestine lesion reading method, characterized in that based on the read small intestine lesion state, the result information for visualizing the small intestine lesion reading result is generated.
8. The method of claim 7,

When generating result information to visualize the small intestine lesion reading result,

(where VS is the visualization scale, M is the total number of frames of the capsule endoscopy image data, size (image) is the pixel size of the image, Ci is the set of clean area pixels, Di is the set of dark area pixels, Fi is the float/bubble area pixels A method for reading a small intestine lesion, characterized in that the result information for visualizing the small intestine lesion reading result is generated by a formula consisting of a set.
A computer program stored in a computer readable storage medium, wherein the computer program, when executed on one or more processors, performs the following operations for reading a small intestine lesion, the operations comprising:

acquiring capsule endoscopy image data;

inputting the capsule endoscopy image data into a pre-trained first neural network model to identify a first transitional portion of the stomach and small intestine and a second transitional portion of the small intestine and large intestine;

dividing and pre-processing small intestine image data between the identified first and second transition units from the capsule endoscopy image data;

reading the small intestine lesion by inputting the pre-processed small intestine image data into a pre-trained second neural network model; and

and generating result information on the small intestine image data based on the read state of the small intestine lesion.
a processor including one or more cores; and

Memory;

including,

The processor is

Acquire capsule endoscopy image data,

By inputting the capsule endoscopy image data into a pre-trained first neural network model, a first transitional portion of the stomach and small intestine and a second transitional portion of the small and large intestine are identified,

Pre-processing by dividing the small intestine image data between the identified first and second transitions from the capsule endoscopy image data,

Reading the small intestine lesion by inputting the pre-processed small intestine image data into the pre-trained second neural network model, and

Computing device, characterized in that generating result information on the small intestine image data based on the read small intestine lesion state.
11. The method of claim 10,

The processor is

at least two of a first condition corresponding to a change in the shape of the mucosal surface, a second condition corresponding to a structural characteristic of the first transition, and a third condition corresponding to a color change of the mucosa, when identifying the first transition portion of the stomach and small intestine Computing device, characterized in that when the combination of more than one condition is satisfied, it is recognized as the first transition part of the stomach and small intestine.
11. The method of claim 10,

The processor is

The computing device according to claim 1, wherein the small intestine image data is segmented based on the identified first transitional portion and the second transitional portion from the capsule endoscopy image data, and a ratio of a mucosal region of the small intestine image data is quantified.
13. The method of claim 12,

The processor is

Computing device, characterized in that when quantifying the image ratio, quantification is performed by applying a weight to the ratio of the mucosal region seen in a frame unit of the small intestine image data and the ratio of the mucosal region seen in the video based on the imaging conditions of the capsule endoscope.
14. The method of claim 13,

The photographing conditions of the capsule endoscope are,

Computing device comprising at least one of a moving speed of the capsule endoscope and a photographing speed of the capsule endoscope.
11. The method of claim 10,

The processor is

When reading the small intestine lesion, if the small intestine lesion is an area in the small intestine image data, the image reproduction speed is reproduced as the reference speed to read the small intestine lesion, and if the small intestine lesion does not appear, the image reproduction speed is used as the reference A computing device characterized in that it reads the small intestine lesion by regenerating faster than the speed.
11. The method of claim 10,

The processor is

Based on the read small intestine lesion state, the computing device, characterized in that generating result information for visualizing the small intestine lesion reading result.
17. The method of claim 16,

The processor is

When generating result information to visualize the small intestine lesion reading result,

(where VS is the visualization scale, M is the total number of frames of the capsule endoscopy image data, size (image) is the pixel size of the image, Ci is the set of clean area pixels, Di is the set of dark area pixels, Fi is the float/bubble area pixels A computing device, characterized in that for generating result information for visualizing the result of reading the small intestine lesion by a formula consisting of