KR20200070062A - System and method for detecting lesion in capsule endoscopic image using artificial neural network - Google Patents

Abstract

Provided is a system for detecting a lesion from a capsule-endoscopic image using an artificial neural network, to provide basic information for enabling a diagnostician to efficiently make a final diagnosis. According to the present invention, the system comprises: a capsule endoscopy capturing a plurality of images of human organs and acquiring a circular image inside the body; a preprocessing unit correcting the circular image captured by the capsule endoscopy; a convolutional neural network unit extracting features from each of the plurality of preprocessed images; a recurrent neural network extracting a time-series pattern feature of the plurality of extracted features; a lesion section detection unit comparing the value of the time-series pattern feature with a threshold; a display unit displaying an image corresponding to the value of the time-series pattern feature, which exceeds the threshold, among the plurality of captured images; and a storage unit storing the plurality of images, the plurality of extracted features, and a time-series pattern extracted from the plurality of features.

Description

System and method for detecting lesions in an encapsulated endoscope image using an artificial neural network {SYSTEM AND METHOD FOR DETECTING LESION IN CAPSULE ENDOSCOPIC IMAGE USING ARTIFICIAL NEURAL NETWORK}

The present disclosure relates to a system and method for detecting lesions in an encapsulated endoscopic image using an artificial neural network, and more specifically, to detect lesions in body images collected through a capsule endoscope using a convolutional neural network and a circulatory neural network. It relates to a system and method for sensing.

Generally, an endoscope can be used to diagnose a condition inside the body or to detect a lesion. As an endoscopic examination method for acquiring an image inside the body, a method in which a soft tube with a camera is inserted into a digestive organ through a patient's mouth or anus is photographed. However, such an endoscopic examination method has limitations in the internal observation of the digestive organs, which are narrow and long, such as the small intestine, and complicatedly bent. The capsule type endoscope developed to solve this problem is an ultra-small endoscope with a pill-shaped diameter of about 9 to 11 mm and a length of about 24 mm to 26 mm, and the patient's camera, like a pill, is swallowed, and the endoscope's camera is placed in the stomach, small intestine, and/or large intestine. While collecting images inside the organ, such as, it is transmitted to an external receiver. Diagnosers can directly observe the internal state of organs by observing the body image transmitted by the capsule endoscope through a video screen and/or a computer monitor. In addition, since the capsule-type endoscope communicates wirelessly with an external device and its size is very small, the foreign body sensation and pain felt by the existing endoscopy can be resolved.

In general, a capsule endoscope takes about 8 to 10 hours to pass through various digestive organs, and on average, 60,000 images are taken while passing through the body. Therefore, the diagnostician needs a long treatment time in order to observe and diagnose the vast amount of images taken by the capsule endoscope. Since the long treatment time of such a diagnoser is a cause of increasing the medical expenses for the examinee, there is a problem that it cannot be popularized due to the high cost despite the advantage of being able to diagnose easily and accurately without the objection of the examinee.

In addition, if a disease or lesion (intestinal stenosis, bleeding, cancer) is found when a diagnosis is observed by a diagnostician through an image taken by a capsule endoscope, the diagnostician accurately identifies the site of the disease and performs treatment (immediate action or procedure) There is a need. However, since the diagnostician cannot accurately know the position of the capsule endoscope with respect to the time when the disease is imaged, there is a hassle that it is necessary to perform a conventional digestive endoscope or a separate detailed examination to accurately identify the disease site for treatment.

According to the conventional method for detecting a lesion using a capsule type endoscope, since the diagnosis and diagnosis of the lesion are performed while the diagnosis is directly observed by the diagnosis of the image inside the body collected through the capsule type endoscope, it takes a lot of time and money, and It can cause excessive work load. In addition, even an experienced diagnostician may perform a subjective judgment that relies on personal knowledge or experience, which may cause a problem in which different diagnostic results are derived from the same internal body image. Therefore, there is a need to overcome the problems of lesion detection using an existing capsule type endoscope, and to provide an accurate lesion detection result to a patient by efficiently performing an image analysis by a diagnostician.

Embodiments disclosed herein provide a system and method for acquiring an image of a patient's body through a capsule endoscope and analyzing the acquired image using a convolutional neural network and a circulatory neural network to detect lesions. In addition, embodiments of the present disclosure are to provide a system and method for a diagnostician to efficiently provide a final diagnosis based on a lesion image by displaying an image of the lesion detected using a convolutional neural network and a circulatory neural network on a display unit do.

A system for detecting lesions in a capsule type endoscope image using an artificial neural network according to an embodiment of the present disclosure extracts features of each of a plurality of images photographed by a capsule type endoscope, and time series of the extracted multiple features Extracting the pattern feature, comparing the value of the time-series pattern feature with a threshold value, and displaying an image corresponding to the value of the time-series pattern feature exceeding the threshold value among the captured multiple images, and extracting the plurality of images A plurality of features and time-series patterns extracted from the plurality of features are stored.

According to another embodiment of the present disclosure, a method of detecting a lesion in a capsule-type endoscope image using an artificial neural network includes: taking a plurality of images of a human organ using a capsule-type endoscope; Extracting features, extracting time-series pattern features of the plurality of extracted features, comparing values of time-series pattern features with threshold values, and among a plurality of captured images, time-series patterns exceeding a threshold value And displaying an image corresponding to the value of the feature, storing a plurality of images, a plurality of extracted features, and a time-series pattern extracted from the plurality of features.

According to various embodiments of the present disclosure, since an image inside the body collected through the capsule endoscope is analyzed using a convolutional neural network and a circulatory neural network, the lesion is detected, and images detected by the lesion are displayed. It can provide basic information to make the final diagnosis efficiently.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will become apparent to those skilled in the art from the description of the claims.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, where similar reference numerals indicate similar elements, but are not limited thereto.
1 is a schematic diagram of a system for detecting lesions in an capsule type endoscope image using an artificial neural network according to an embodiment of the present disclosure.
2 is a structural diagram of an capsule endoscope according to an embodiment of the present disclosure.
3 is an image of the body collected through the capsule endoscope according to an embodiment of the present disclosure.
4 is an image of a result of executing a pre-processing and feature extraction process on an internal body image collected through a capsule endoscope according to an embodiment of the present disclosure.
5 is a structural diagram showing the configuration of a convolutional neural network according to an embodiment of the present disclosure.
6 is a structural diagram showing the configuration of a circulating neural network according to an embodiment of the present disclosure.
7 is a graph showing a lesion section when a result value of a circulating neural network according to an embodiment of the present disclosure exceeds a threshold value.
8 is a flowchart illustrating a method of detecting a lesion in an capsule type endoscope image using an artificial neural network according to an embodiment of the present disclosure.
9 is an exemplary view showing a screen of an image display unit and a lesion section including a lesion detected according to an embodiment of the present disclosure.
10 is an exemplary view showing a screen of a display unit in which a threshold is automatically set through disease selection according to an embodiment of the present disclosure.
11 is an exemplary view showing a screen of a display unit that changes the number of images output through threshold adjustment according to an embodiment of the present disclosure.

Hereinafter, with reference to the accompanying drawings, specific details for the practice of the present disclosure will be described in detail. However, in the following description, when there is a risk of unnecessarily obscuring the subject matter of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. In addition, in the following description of the embodiments, it may be omitted to describe the same or corresponding elements overlapping. However, although descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

A system for detecting a lesion in a capsule-type endoscope image using an artificial neural network according to an embodiment of the present disclosure converts an image of a human organ taken through the capsule-type endoscope into a correction process and/or a flat image through a pre-processing unit , Convolutional neural network is used for feature extraction of the transformed image. In addition, the system uses a cyclic neural network to extract time-series pattern features of features of an image extracted using a convolutional neural network, and compares the values of the extracted time-series pattern features with threshold values having certain values to determine lesions. Detection can provide an image of the lesion to the diagnostician.

1 is a schematic diagram of a system 100 for detecting lesions in an encapsulated endoscope image using an artificial neural network according to an embodiment of the present disclosure. The system 100 illustrated in FIG. 1 may transmit an image in the human body collected through the capsule endoscope 110 to the lesion detection device 120 using wireless communication. In the present disclosure, the "capsule type endoscope", when the patient swallows like a pill, passes through the esophagus and passes through the internal organs such as the stomach, small intestine, large intestine, etc., and transmits an image of the body, thereby transmitting an image of the body taken by a diagnostician such as a doctor Refers to the smallest endoscope device that can be observed on a video screen or a computer monitor to diagnose the condition or lesion inside the organ or body cavity. In the case of a general endoscope including a flexible tube with a camera attached, many patients avoid the endoscopy due to pain and discomfort felt while the endoscope is inserted into the body. The capsule endoscope was developed to compensate for the shortcomings of this general endoscope and to be used for diagnosis of diseases such as the longest small intestine among digestive organs. A typical capsule endoscope can take about 1.4 to 2.8 images per second and transmit it to an external diagnostic device through a wireless communication device. Considering that a typical capsule type endoscope stays in the body is about 8 to 10 hours, a total of 60,000 to 80,000 high-sensitivity image information can be transmitted to an external diagnostic device and stored. When the capsule endoscope is discharged out of the body and the test is over, the diagnostician can observe the image information stored in the external diagnosis device through the monitor.

As shown in FIG. 1, the lesion detection device 120 includes a communication unit 122, a pre-processing unit 132, a convolutional neural network unit 134, a circulatory neural network unit 136, and a lesion section detection unit 138, It may include a display unit 140, a storage unit 124. The communication unit 122 may wirelessly receive an image of the human body taken through the capsule endoscope 110 and transmit it to the processor 130.

According to one embodiment, the capsule-type endoscope 110 is provided with an image sensor (or camera) for photographing the front of the endoscope proceeding direction, or an image sensor (or camera) for photographing the side of the endoscope proceeding direction It can, but is not limited to this. For example, when the capsule-type endoscope 110 is configured to photograph a side surface in the traveling direction, the image sensor may photograph light reflected from the inside of the human body by being irradiated through a light source arranged toward the outside of the side surface of the endoscope 110. have. In this case, the light reflected from the inside of the human body and incident on the side of the endoscope is reflected by a reflector installed on the front surface of the endoscope advancing direction, and may be imaged by an image sensor arranged toward the front end of the endoscope advancing direction. At this time, the image sensor photographs a substantially circular (or annular) body image. The photographed body image may be transmitted to the processor 130 through the communication unit 122.

The processor 130 may include a pre-processing unit 132, a convolutional neural network unit 134, a circulating neural network unit 136, a lesion section detection unit 138, and a display unit 140. The pre-processing unit 132 may convert a circular body image photographed by the capsule endoscope 110 into a band-shaped or rectangular body image. Additionally or alternatively, the pre-processing unit 132 performs image correction to emphasize color and/or shape characteristics capable of expressing lesion characteristics of the body image well, or preprocessing to extract 3D shape information from the body image. Can proceed.

In the process of taking and collecting an internal image through the capsule endoscope 110, since thermal energy is generated in an electronic circuit including an image sensor of the capsule endoscope, heat noise is generated in the image captured by the capsule endoscope, thereby deteriorating the image quality of the image. Can be. The image may be corrected through the pre-processing unit 132 to compensate for deterioration of the image quality of the body image. In addition, the image in the body, since the image sensor is arranged at a position facing the reflector arranged on the front side of the traveling direction of the capsule endoscope 110, the light reflected inside the human body is reflected back from the reflector to the image sensor When photographed by, the central portion of the photographed image may include a shape of an image sensor reflected by the central portion of the reflector or the reflector. In this case, as illustrated in FIG. 3, the center of the body image may be displayed in a dark or empty form. Therefore, since the content of the central portion of the body image does not reflect the inside of the human body, it is necessary to remove it from the analysis object. To this end, the process of converting the portion of the annular image from the circular body image to the flat image except the center portion may be performed through the pre-processing unit 132.

The convolutional neural network unit 134 may detect a lesion by analyzing an internal image converted into a flat image and/or corrected by the preprocessing unit 132 using the convolutional neural network. The convolutional neural network is a type of deep neural network that is frequently used to analyze image patterns, and can learn a specific pattern while maintaining spatial information of the image. The convolutional neural network unit 134 may configure a neural network that repeats a convolution (composite) and subsampling process to find characteristics of an image in the body. In the convolution process of the convolutional neural network unit 134, a filter is used to process the body image, and the filter is generally defined as a square matrix. The convolutional neural network unit 134 may traverse the body image at a specified interval using a filter and calculate convolution (composite product) for each section. Through this calculation process, a feature map may be generated for each section of the body image, and a feature map of each section may be added to generate a final feature map. After generating the final feature map, a subsampling process may be performed to reduce the size by selecting a maximum value of a corresponding area of the final feature map. The convolutional neural network unit 134 may repeat the convolution process and the subsampling process for the body image, and output a one-dimensional matrix including only features of the body image. The output one-dimensional matrix is input to the input layer of the fully connected neural network, and the fully connected neural network may output an output value representing characteristics of the body image corresponding to the input value.

In the circulating neural network unit 136, a lesion section may be detected by analyzing a time-series pattern change of characteristic output values of body images output from the convolutional neural network unit 134. Cyclic neural networks are a type of deep neural network used to analyze time-series pattern changes in ordered input information such as sentences and voices. Cyclic neural networks learn data that changes with the passage of time, such as time series data, and can be composed of neural networks that interconnect the past time (t-1) and the next time (t+1) of the reference time (t). have. That is, the structure of the circulatory neural network is composed of an input layer, an output layer, and a hidden layer. The hidden layer may receive the output of the hidden layer at the previous point in time (t-1) and be updated to generate a value transmitted to the output layer. The cyclic neural network unit 136 inputs characteristic output values of temporally changing body images output from the convolutional neural network unit 134 to the input layer, and receives the output of the hidden layer at the previous point in time t-1 and is updated. By deriving the output value using the hidden layer, it is possible to derive a time-series pattern change over time of the body image.

The lesion section detection unit 138 compares a value representing a time series pattern change over time of the body image derived from the circulatory neural network unit 136 with a threshold value, and determines a time series pattern change section that is greater than or equal to the threshold value You can judge by section. That is, the lesion section detecting unit 138 compares the time-series pattern change value according to the time of the body image with a preset threshold that is a criterion for determining the lesion, and a section having a value smaller than the preset threshold is normal. The section representing the state and having a value greater than a predetermined threshold value may be determined to be a section having a high probability of having a lesion site.

The display 140 may output an image of the body determined by the lesion section detection unit 138 to correspond to the lesion section on the display so that the diagnosis and analysis can be performed by the diagnosis unit.

The storage unit 124 may store internal images received through the communication unit 122. In addition, the storage unit 124 is associated with the pre-processing unit 132, the convolutional neural network unit 134, the circulatory neural network unit 136, the lesion section detection unit 138, and the display unit 140 of the processor 130. Data can be saved.

According to the configuration of the lesion detection device 120 described above, the processor 130 receives the body image collected from the capsule endoscope 110 through the communication unit 122, and receives the circular image received through the communication unit 122 The image may be converted into a flat image through an image correction and conversion process through the pre-processing unit 132. When the pre-processing unit 132 performs correction and plane imaging on the body image, the corrected and transformed plane image is input to the convolutional neural network 134. The convolutional neural network 134 repeats the synthesis product and the subsampling process for the planar image, and finally calculates the characteristics of the body image through analysis by the fully connected neural network. Also, the circulatory neural network 136 may receive and analyze feature values of an ordered body image as input, and output a time-series pattern change over time of the body image. The time-series pattern change value according to the time of the output body image is compared with the threshold value in the lesion section detection unit 138, and a section having a time-series pattern change value greater than the threshold value is identified as a lesion section, and the display unit ( 140).

2 is a structural diagram of an capsule endoscope according to an embodiment of the present disclosure. As shown, the capsule endoscope 110 may include a light source 212, an image sensor 214, a reflection device 216, and a data transmission unit 218. The capsule-type endoscope 110 that has entered the body 230 may collect various body image data including images of lesion tissue or normal tissue.

The light source 212 may irradiate light toward a side surface in the traveling direction of the capsule endoscope 110 and reflect the irradiated light on the inner surface of the body 230 facing the side surface of the endoscope 110. . The light reflected from the inner surface of the house 230 may be reflected back through the reflector 216 and incident on the image sensor 214. In this case, the reflection device 216 may be made of a material having properties that partially and/or partially reflect light. The image sensor 214 may photograph the light reflected by the reflector 216 and convert it into an image, and transmit it to a recording device or a lesion sensing device outside the human body through the data transmission unit 218.

In one embodiment, the body image captured by the image sensor 214 may include an image of the tissue of the lesion 220. The light irradiated from the light source 212 of the capsule endoscope 110 and reflected from the body 230 may be reflected once again by the reflector 216 and incident on the image sensor 214 in the capsule endoscope 110. have. At this time, since the light incident on the image sensor 214 is light reflected from the internal surface of the body 360 degrees omnidirectional surrounding the side surface of the endoscope 110, it can be expressed as a circular internal image.

3 is an image of the body collected through the capsule endoscope according to an embodiment of the present disclosure. When viewed from the reflector 216 of the capsule endoscope 110, the image sensor 214 is located in the center of the inside of the endoscope facing the reflector 216, so the light reflected from the body reflects the reflector 216 In the process of being reflected again, the shape of the image sensor 214 may also be reflected. Accordingly, the body images 310, 312, 314, and 316 photographed by the image sensor 214 may have an empty shape in the center.

In one embodiment, as the capsule-type endoscope 110 moves the body 230 and acquires an image around it, the endoscope 110 may acquire a plurality of body images having a pattern that changes over time. . For example, if the capsule endoscope 110 acquires an internal body image 310 in which part of the lesion 320 starts to be visible at time t, as time passes, the lesions 320 are each at t+1 hour ) Is a more visible body image 312, an image of the body 320 where the lesion 320 is completely visible at t+2 hours, 314 an image of a body where part of the lesion 320 disappears at t+3 hours Can be obtained.

4 is an image of a result of executing a pre-processing and feature extraction process on an internal body image collected through a capsule endoscope according to an embodiment of the present disclosure. The body images 310, 312, 314, and 316 obtained in FIG. 3 are empty or dark circular images in the center. Therefore, for the convenience of diagnosing lesions, a correction process for removing a circle appearing in the center of the body image and converting the circle image into a plane image may be performed through the pre-processing unit 132.

According to an embodiment, for correcting an internal body image, an internal body radiated image photographed while the endoscope light source is emitted and an internal matte image photographed while the light source is matt. The pre-processing unit 132 receives the body light-emitting image and the body matte image through the communication unit 122 and subtracts the pixel value of the body light-emitting image from the pixel value of the body light-emitting image, thereby reducing the central dark portion of the body light-emitting image. Correction to delete can be performed. In addition, the pre-processing unit 132 of the inner body of the circular body image (310, 312, 314, 316) received through the capsule endoscope, the inner circle surrounding the dark central portion, and the radius of the outer circle of the inner circle By performing a pre-processing operation for converting the value minus the radius to the x and y coordinate axes, plane images 410, 412, 414, and 416 can be obtained.

That is, the body image 310 when the time obtained in FIG. 3 is t, the body image 312 when the time is t+1, the body image 314 when the time is t+2, and the time t The body image 316 at +3 is through the pre-processing unit 132, respectively, the body image 410 at time t, the body image 412 at time t+1, and time t The body image 414 at +2 and the body image 416 at time t+3 may be converted. In the body images 410, 412, 414, and 416, the lesion 420, as time elapses, begins to appear partially at the bottom of the image, shows a full shape, and then partially disappears from the top of the image. see.

5 is a structural diagram showing the configuration of a convolutional neural network 500 according to an embodiment of the present disclosure. The convolutional neural network 500 may be used in the convolutional neural network unit 134 to perform pattern analysis to detect lesions in each of the body images that are corrected and/or converted into a planar image by the preprocessing unit 132. The convolutional neural network 500 may execute a process of repeating convolution (composite) and subsampling to find the two-dimensional characteristics of the body image.

As shown, the convolutional neural network 500 traverses the body image at a specified interval using a filter defined as a square matrix for the body image 510, and performs convolution (composite) for each section. Can be calculated. A feature map may be generated for each section through convolution (composite product), and a final feature map 512 may be generated by summing feature maps of each section. After the final feature map 512 is generated, the size of the final feature map may be reduced through a process of subsampling 520 to reduce the size by selecting a maximum value of a corresponding area of the final feature map 512. The convolutional neural network 500 repeats the convolution (composite product) process and subsampling (520, 522) process to generate feature maps 512, 514, and 516 for the body image, and only the characteristics of the body image are repeated. You can print out a one-dimensional matrix. In addition, the fully connected neural network 518 receives a one-dimensional matrix representing the characteristics of the body image, and generates a final output value x representing the two-dimensional characteristics of the body image.

In one embodiment, the body image 410 when the time obtained in FIG. 4 is t, the body image 412 when the time is t+1, the body image 414 when the time is t+2, Each of the body images 416 when the time is t+3 may be input as an input value of the convolutional neural network 500. Each input body image 410, 412, 414, 416 repeats the convolution (composite) process and subsampling (520, 522) process to generate feature maps 512, 514, and 516, and finally Through the feature extraction process by the fully connected neural network 518, output values (x) at t, t+1, t+2, t+3 time including the two-dimensional features of the body image are generated. Can be.

6 is a structural diagram showing the configuration of a circulating neural network 600 according to an embodiment of the present disclosure. The circulatory neural network 600 is used in the circulating neural network unit 134 and analyzes time-series pattern changes of images in the body based on output values representing two-dimensional characteristics of the body image output from the convolutional neural network unit 134. By detecting the lesion section. The recurrent neural network 600 is an artificial neural network for learning and recognizing data that changes over time, such as time series data. The circulatory neural network 600 sends the result value calculated through the activation function of the concealment layer 620 based on the value x input to the input layer 610 to the output layer 630, while the concealment layer 620 itself Send the status value (hidden status value) back to itself and use it to calculate the next result value. As described above, in the circulatory neural network 600, the neurons of the hidden layer 620 at the reference point t are continuously recursively calculating values output from the same neuron at the previous point t-1 as their input. By executing, it is possible to express the change pattern over time of the input (x) as the output (y).

Also, the recurrent neural network 600 may share the same parameters or at least a portion of the parameters for each layer, unlike a general multi-layer neural network having different parameters for each layer. The recurrent neural network 600 may reduce learning time by significantly reducing the number of parameters to be learned by sharing at least some parameters for each layer. At this time, the circulating neural network 600 may express the result calculation differently as the parameters are changed.

)을 입력층(610)에 입력하고, 직전 시점(t-1)의 은닉 상태를 받아 갱신된 은닉층(610)의 출력 값(

)을 산출함으로써, 체내 이미지의 시간에 따른 시계열적 패턴 변화를 도출할 수 있다.In one embodiment, the circulatory neural network 600 includes values representing two-dimensional characteristics of the body image output from the convolutional neural network unit 134 (

) Is input to the input layer 610, and the output value of the updated hidden layer 610 is received by receiving the hidden state of the previous point in time t-1 (

By calculating ), it is possible to derive a time-series pattern change with time of the body image.

In another embodiment, the user may differently adjust the parameters of the color model of the cyclic neural network 600 to calculate different results. The user can calculate the result value differently using parameters such as Gray model, RGB model, HSV model, and YCbCr model for the body image collected through the capsule endoscope. The Gray model expresses an image with only brightness information without using color information, and can express pixel values of an image with a total of 256 levels of brightness values ranging from a black value of 0 to a white value of 255. The RGB model is the most basic color model, and a specific color can be composed of a combination of three components: red, green, and blue. The HSV model expresses color with three components: hue, saturation, and brightness. When the saturation value is 0, achromatic color and 255 can express the most vivid color. The YCbCr model can express an image using only pure color information without using brightness information. The user may calculate the body images of various shapes by adjusting the parameters of the color model of the circulatory neural network 600. For example, the user can express the body image collected through the capsule endoscope using the YCbCr model that uses only pure color information without using brightness information. Since the body image using the YCbCr model uses only pure color information without the brightness value, it is possible to effectively recognize the color of the same object regardless of the environment change in the body and collect a clearer and more accurate body image.

)를 임계값과 비교하여, 임계값을 초과하는 패턴 변화값(y_t)을 포함한 구간을 병변 가능성이 높은 병변 구간으로 감지할 수 있다. 즉, 병변구간 감지부(138)는, 시간 t 일 때의 체내 이미지의 시간에 따른 시계열적 패턴 변화(y_t)를 기 설정된 임계값과 비교하여, 기 설정 된 임계값보다 작으면 정상 구간, 기 설정 된 임계값보다 크면 병변 구간이라고 판단할 수 있다. 7 is a graph showing a lesion section when a result value of a circulating neural network according to an embodiment of the present disclosure exceeds a threshold value. The lesion section detection unit 138 changes the time series pattern with time of the body image derived from the circulatory neural network unit 136 (

) Is compared with a threshold value, and a section including a pattern change value y _t exceeding the threshold value may be detected as a lesion section having a high probability of lesion. That is, the lesion section detecting unit 138 compares the time-series pattern change (y _t ) with time of the body image at the time _t with a preset threshold, and if it is smaller than the preset threshold, the normal section, If it is larger than the preset threshold, it can be determined that it is a lesion section.

)의 범위를

라 가정하고, 임계값을 0.5로 가정하였을 때, 기 설정된 임계값인 0.5를 넘어가는 시계열적 패턴 변화(

)는 모두 병변이 포함된 병변 부위라고 판단할 수 있고, 0.5를 넘어가지 않는 시계열적 패턴 변화(

)는 모두 정상 부위라고 판단할 수 있다.In one embodiment, the time-series pattern change value with time of the body image derived from the circulatory neural network unit 136 (

) Range

D, and when the threshold is assumed to be 0.5, the time-series pattern changes beyond the preset threshold of 0.5 (

) Can be judged to be all lesions containing lesions, and time-series pattern changes not exceeding 0.5 (

) Can be judged to be all normal parts.

8 is a flowchart illustrating a method of detecting a lesion in an capsule type endoscope image using an artificial neural network according to an embodiment of the present disclosure. The method 800 for detecting a lesion in an encapsulated endoscope image using an artificial neural network may be started as a step S810 of capturing a human body organ and obtaining a circular body image. In the capsule endoscope 110, light irradiated through the light source 212 may be reflected inside the human body and incident on the image sensor 214. The reflected light incident on the image sensor 214 may be photographed as a circular body image and transmitted to the recording device or lesion detection device 120 outside the human body through the data transmission unit 218. At this time, when viewed from the reflection device of the capsule endoscope 110, since the image sensor is located in the center of the endoscope 110, the shape of the image sensor is also reflected in the process of reflecting light reflected from the inside of the human body back into the reflection device. Because it is reflected, the body image can be photographed in a substantially circular or annular shape.

For a circular body image transmitted to a lesion detection device outside the human body, pre-processing for correction of the circular image and conversion to a flat image may be performed through the pre-processing unit 132 (S820). The body image obtained through the capsule endoscope 110 is a circular body image with a dark or empty center, a darkening process of removing the dark part in the center for extracting lesions, and converting the circular image into a flat image. Can go through. As a method of correcting the body image, a method of subtracting the pixel value of the body image photographed while the light source of the capsule endoscope 110 is illuminated and the pixel value of the body image photographed while the light source is matt may be used. have. In addition, the pre-processing operation of converting the values of the inner circle image circumference of the inner circle image and the radius of the outer circle minus the radius of the inner circle to the x and y coordinate axes, respectively, among the body images of the dark center circle received through the capsule endoscope. By performing, it is possible to obtain a flat image.

Thereafter, the characteristics of the body image obtained using the convolutional neural network may be extracted (S830 ). The convolutional neural network unit 134 uses the convolutional neural network 500 to perform pattern analysis to detect lesions in the body image that has been corrected and/or converted into a planar image by the pre-processing unit 132, and then performs the analysis of the body image. An output value representing a two-dimensional characteristic can be generated.

)을 입력층(610)에 입력하고, 직전 시점(t-1)의 은닉 상태를 받아 갱신된 은닉층(610)의 출력 값(

)을 산출함으로써, 체내 이미지의 시간에 따른 시계열적 패턴 변화를 도출할 수 있다.Sequentially or simultaneously, a time-series pattern feature may be extracted from two-dimensional feature values of the image extracted from the convolutional neural network using a cyclic neural network (S840). In one embodiment, the circulatory neural network 600 includes values representing two-dimensional characteristics of the body image output from the convolutional neural network unit 134 (

) Is input to the input layer 610, and the output value of the updated hidden layer 610 is received by receiving the hidden state of the previous point in time t-1 (

By calculating ), it is possible to derive a time-series pattern change with time of the body image.

)를 임계값과 비교하여, 임계값을 초과하는 패턴 변화값(y_t)을 포함한 구간을 병변 가능성이 높은 병변 구간으로 감지할 수 있다. 즉, 병변구간 감지부(138)는, 시간 t 일 때의 체내 이미지의 시간에 따른 시계열적 패턴 변화(y_t)를 기 설정된 임계값과 비교하여, 기 설정 된 임계값보다 작으면 정상 구간, 기 설정 된 임계값보다 크면 병변 구간이라고 판단할 수 있다. 또한, 병변 구간에 대응되는 체내 이미지를 디스플레이부(140)에 디스플레이 할 수 있다.After deriving the time-series pattern change over time of the body image through the cyclic neural network, the time-series pattern change value is compared with the threshold value (S850), and an image corresponding to the time-series pattern change value exceeding the threshold value is displayed. (S860) You can. In one embodiment, the lesion section detection unit 138 changes the time series pattern with time of the body image derived from the circulatory neural network unit 136 (

) Is compared with a threshold value, and a section including a pattern change value y _t exceeding the threshold value may be detected as a lesion section having a high probability of lesion. That is, the lesion section detecting unit 138 compares the time-series pattern change (y _t ) with time of the body image at the time _t with a preset threshold, and if it is smaller than the preset threshold, the normal section, If it is larger than the preset threshold, it can be determined that it is a lesion section. In addition, the body image corresponding to the lesion section may be displayed on the display unit 140.

)를 임계값과 비교하여 임계값 이상의 값을 병변 구간이라고 판단한 후, 병변 구간에 대응되는 체내 이미지들을 디스플레이 할 수 있다. 9 is an exemplary view showing an image including a detected lesion and a lesion section on a screen of a display unit according to an embodiment of the present disclosure. Time-series pattern change with time of the body images calculated by the lesion section detector 138 (

) Is compared with a threshold value, and a value above the threshold value is determined to be a lesion section, and then images of the body corresponding to the lesion section may be displayed.

As shown, in the display unit 140, a graph indicating the lesion section detected by the lesion section sensing unit 138 is displayed on the first display area 910. When the diagnostician designates the lesion section in the graph displayed on the first display area 910, the body images corresponding to the lesion section are displayed in the third display area 930 in chronological order. At this time, in the first display area 910, the diagnostician can further click on the body image that is considered to require detailed analysis and diagnosis, so that an enlarged image of the body image can be displayed on the second display area 920. have.

10 is an exemplary view showing a screen of a display unit in which a threshold is automatically set through disease selection according to an embodiment of the present disclosure. The first display area 1012 may display internal images corresponding to values above a preset threshold. When the diagnostician selects a corresponding disease through the second display region 1014, the diagnostic threshold may be indicated in the third display region 1016. At this time, when the diagnostician clicks the detailed description button (for example, "Learn more") related to the disease selected in the third display area 1016, the display unit displays only the first images in the body corresponding to values above the corresponding threshold value. It can be displayed in the region 1022. When the diagnostician selects one of the body images displayed on the first display area 1022, the second display area 1024 and the third display area 1026 respectively show a graph showing the lesion section and an enlarged image of the selected body image. Can be displayed.

According to an embodiment, the diagnostician may select a disease that is considered to correspond to the patient in the second display area 1014 where the name of the disease is indicated. For example, if the diagnostician selects colorectal cancer in the second display area 1014, a threshold value of “0.8” corresponding to colorectal cancer may be displayed in the third display area 1016. The threshold value displayed on the third display area 1016 may indicate a threshold value determined to be suitable for the corresponding disease through experiments, or a threshold value previously set by a diagnostic person may be displayed.

11 is an exemplary view illustrating a screen of a display unit that changes the number of images output through threshold adjustment according to an embodiment of the present disclosure. The first display area 1110 may display a graph indicating a lesion section exceeding a threshold value. At this time, the second display area 1120 may display a user interface through which the diagnostician can adjust the threshold. The diagnostician may set the final threshold value by adjusting the threshold value through the user interface of the second display area 1120. The third display area 1130 may display images of the body corresponding to a value equal to or greater than the threshold value in a time-lapse order according to the threshold set in the second display area 1120.

According to an embodiment, the diagnostician may adjust the threshold value through the user interface (that is, the threshold adjustment button) in the second display area 1120. At this time, when the diagnostic person increases the threshold value, the lesion section exceeding the corresponding threshold value decreases, and only the body image having a relatively high time-series pattern change value may be displayed on the third display 1160. The third display 1160 may increase the accuracy of the diagnosis of a lesion of a diagnosis by displaying an internal image having a high pattern change value, that is, an image having a high probability of lesion. However, if the diagnostician raises the threshold value more than necessary, there is a possibility that a lesion section requiring analysis may be missed, and accuracy of the lesion analysis may be lowered.

Although the present disclosure has been described in connection with some embodiments in this specification, it should be understood that various modifications and changes may be made without departing from the scope of the present disclosure, which can be understood by those skilled in the art to which the present invention pertains. something to do. In addition, such modifications and variations should be considered within the scope of the claims appended hereto.

110: capsule endoscope 120: lesion detection device
122: communication unit 124: storage unit
130: processor 132: pre-processing unit
134: convolutional neural network unit 136: circulatory neural network unit
138: lesion section detection unit 140: display unit
212: light source 214: image sensor
216: reflector 218: data transmission unit
220: lesion 230: in the body
310, 312, 314, 316: body image
320: lesion 410, 412, 414, 416: flat body image
420: lesion 500: convolutional neural network
510: body image 512, 514, 516: feature map
518: one-dimensional matrix 520, 522: sub-sampling
600: circulatory neural network
910, 1012, 1022, 1110, 1140: 1st display
920, 1014, 1024, 1120, 1150: Second display
930, 1016, 1026, 1130, 1160: 3rd display

Claims (12)

In the system for detecting a lesion in the capsule-type endoscope image using an artificial neural network,
Capsule-type endoscope for taking a plurality of images of human organs, and obtaining a circular image inside the human body;
A pre-processing unit that corrects a circular image photographed by the capsule endoscope and converts it into a flat image;
A convolutional neural network unit extracting features of each of the preprocessed images;
A circulating neural network unit extracting time-series pattern features of the plurality of extracted features;
A lesion section detecting unit comparing the value of the time-series pattern feature with a threshold value;
A display unit that displays an image corresponding to a value of a time-series pattern feature that exceeds the threshold value among the plurality of photographed images; And
And a storage unit for storing the plurality of images, the extracted plurality of features, and a time-series pattern extracted from the plurality of features.
According to claim 1,
The capsule endoscope,
A light source irradiating light toward a side surface of the capsule endoscope;
A reflector reflecting light incident from the side surface of the capsule endoscope;
An image sensor photographing light reflected by the reflector; And
And a data transmission unit transmitting a plurality of images generated by the image sensor.
According to claim 1,
The convolutional neural network unit,
A system for extracting features of each of the plurality of images using a filter of a convolutional neural network from the plurality of images of the human organ obtained through the capsule endoscope.
According to claim 1,
The circulatory neural network unit,
A system for extracting time-series pattern features that change over time of the plurality of images of the human organs using the cyclic neural network from the plurality of features extracted from the convolutional neural network.
According to claim 1,
The lesion section detecting unit compares the time-series pattern feature extracted from the circulating neural network unit with a threshold value,
The threshold value, the threshold value in the lesion site section where the lesion is found, has a value that is a certain size higher than the threshold in the normal site section.
According to claim 1,
The display unit,
A system for displaying a plurality of body images corresponding to a plurality of times, and displaying the selected body image in a separate display section when a user selects at least one of the plurality of body images.
The method of claim 6,
The display unit includes a user interface configured to select a disease corresponding to the displayed plurality of body images, and to provide a preset threshold suitable for the disease,
And the display unit is configured to adjust the number of images corresponding to values of time-series pattern features exceeding the threshold value among the displayed plurality of body images through adjustment of the threshold value.
In the method of detecting a lesion in the capsule endoscope image using an artificial neural network,
Taking a plurality of images of human organs, using a capsule endoscope, and obtaining a circular image inside the human body;
Correcting, by the pre-processing unit, the photographed circular image and converting it into a flat image;
Extracting features of each of the photographed plurality of images by a convolutional neural network unit;
Extracting time-series pattern features of the plurality of features extracted by the circulatory neural network;
Comparing the value of the time-series pattern feature with a threshold value by the lesion section detection unit;
Displaying, by the display unit, an image corresponding to a value of a time-series pattern feature exceeding the threshold value among the plurality of photographed images;
And storing, by the storage unit, the plurality of images, the extracted plurality of features, and the time-series pattern extracted from the plurality of features.

The method of claim 8,
Extracting a plurality of features by the convolutional neural network,
And extracting features of each of the plurality of images from the plurality of images of the human organ obtained through the capsule endoscope, using a filter of a convolutional neural network.
The method of claim 8,
The step of extracting the time-series pattern feature by the circulating neural network unit,
And extracting, from the features of the plurality of images extracted from the convolutional neural network unit, a time-series pattern feature that changes with the passage of time of the plurality of images of the human organ, using the circulating neural network, Way.
The method of claim 11,
The step of comparing the value of the time-series pattern feature with the threshold value by the lesion section detecting unit includes comparing the time-series pattern feature extracted from the cyclic neural network portion with the threshold value,
The step of displaying an image corresponding to the value of the time-series pattern feature exceeding the threshold value is configured to adjust the number of images corresponding to the value of the time-series pattern feature exceeding the threshold value by adjusting the threshold value. Way.
A non-transitory computer readable storage medium storing computer program code for causing a computing device to execute any one of the methods of claims 8-11.

Images