KR102344041B1 - Method and system for diagnosing lesion using deep learning - Google Patents

Abstract

Disclosed herein are systems and methods for diagnosing lesions. A specific implementation example of such a technique is to set the suspected lesion region of the still image obtained when a still image capture request is made during video recording using an endoscope, etc. as the region of interest, and then set the red (R) channel, green (G) Cumulative distribution function (CDF: cumulative distribution function) value, smoothing the histogram for the derived CDF value, converting it into image data with improved contrast, and then synthesizing the lesion by sequentially performing transfer learning and deep learning on the synthesized image data. By detecting the lesion, it is possible to objectively diagnose the lesion and improve the accuracy of the lesion.

Description

Lesion diagnosis system and method

The present invention relates to a system and method for diagnosing a lesion, and more particularly, to a technology capable of objectively diagnosing a lesion from an endoscopic image obtained after acquiring an endoscopic image, and improving the accuracy of the lesion. .

During video recording, such as an endoscope image, a user may acquire a still image capture. However, when capturing a still image using a camera in a situation with insufficient light, such as a dark room, or when a part of an organ such as the gastrointestinal tract moves when taking an endoscopic image, global or local blur occurs due to camera shake phenomenon may occur.

Although various de-blurring technologies have been developed to correct blurred images in general images, in medical images such as endoscopes, information that may cause errors in diagnosis may be added during image processing, so the existing de-blurring techniques Blurring technology cannot be applied to medical images.

In order to solve this problem, the conventional image processing device continuously stores still images in the image buffer during video recording, and when an image capture request comes in, it automatically represents one of all the still images stored before a predetermined time regardless of the user's intention. Select video.

Most of the recently developed conventional image processing apparatuses store still images in advance in a buffer for a predetermined time (Δt) as shown in FIG. 1, and when a still image capture request is received, the clearest image is automatically selected and stored. do.

However, when motion blur occurs in the camera, blur may also occur in still images stored before the capture request, and a meaningless image may be selected in spite of selecting the sharpest image among pre-stored images.

Accordingly, since the conventional image processing apparatus provides a captured image that is not directly selected by the user without reflecting the user's intention, there is a problem in that an image with blur can be provided as a captured image.

As a related prior art, US Patent Publication No. US 2014/0132746 A1 (Image Capture Stabilization), etc. may be referred to.

1. “Blur detection for digital images using wavelet transform”, Hanghang Tong, Mingjing Li, Hongjinang Zhang, Changshui Zhang, 2004 IEEE International Conference on Multimedia and Expo (ICME), June 2004. 2. “Convolutional Neural Networks for No-Reference Image Quality Assessment”, 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.

Therefore, the technical task to be achieved by the present invention is to set the lesion suspect region of the still image obtained when a still image capture request is made during video recording using an endoscope as the region of interest and then detect the lesion based on the set region of interest. An object of the present invention is to provide a lesion diagnosis system and method capable of diagnosing a lesion and improving the accuracy of the lesion.

The present invention enables detection and classification of lesions in real time by processing only still images of the suspected lesion region.

The object of the present invention is not limited to the object mentioned above, and other objects and advantages of the present invention not mentioned can be understood by the following description, and will be more clearly understood by the examples of the present invention. Further, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the appended claims.

According to these features, the lesion diagnosis system according to an embodiment includes:

an image acquisition unit for acquiring a still image of a region of interest for a suspected lesion region among videos acquired through an endoscope; After correcting the resolution of the still image to a predetermined resolution, a cumulative distribution function (CDF) value for each pixel is derived, and a histogram of the derived CDF value is smoothed to convert it into image data of a predetermined size. preprocessor; It may include a lesion detector that sequentially performs transfer learning and deep learning learning on the image data and detects a lesion based on a learning result.

Preferably, the still image may include a red channel, a green channel, and a blue channel image, and the preprocessor includes a cumulative distribution function for each pixel for each of the green channel and the blue channel. After deriving a cumulative distribution function (CDF) value and then smoothing the histogram for the derived CDF value to convert it into image data of a predetermined size, the image data for each channel is synthesized and delivered to the lesion detection unit. have.

The preprocessor derives a cumulative distribution function (CDF) value for each pixel for each of the green channel, the green channel, and the blue channel, and then smooths the histogram for the derived CDF value to convert it into image data of a predetermined size After synthesizing the image data for each channel, it may be provided to transmit the image data to the lesion detection unit.

Preferably, the lesion detection unit,

ImageNet learning module for performing transfer learning based on a learning model built based on an image network of image data of a predetermined size; and a deep learning learning module that receives the transfer learning control value of the ImageNet learning module as an initial value, performs deep learning learning, and detects and classifies lesions based on the learning result.

Preferably, the deep learning learning module acquires the position information and probability information of the rectangular coordinate system through the RPN (Region Proposal Network) technique for the features extracted through the ResNET structure consisting of a convolutional layer and a pool layer, and the detected position It may be provided to detect and classify a lesion through the obtained probability information.

Preferably, the lesion detection unit,

It may further include a lesion verification module that verifies the detection and classification results of lesions based on a comparison of the lesions detected based on the learning results of transfer learning of the ImageNet learning module and the lesions detected based on the learning results of the deep learning learning module. have.

A method for diagnosing a lesion according to an embodiment includes: an image acquisition step of acquiring a still image of a region of interest for a suspected lesion region among videos acquired through an endoscope; A cumulative distribution function (CDF) value for each pixel is derived for each of the green channel, green channel, and blue channel images among the red channel, green channel, and blue channel images of the acquired still image of the region of interest. , a pre-processing step of converting the derived CDF value into image data of a predetermined size by smoothing the histogram;

An ImageNet learning step of detecting a lesion by performing transfer learning based on a learning model built based on an image network of image data of a predetermined size; And by receiving the control value of transfer learning of the ImageNet learning module as an initial value, the image data is extracted through the ResnET structure consisting of a predetermined number of convolutional layers and one full layer. The position of the rectangular coordinate system through the RPN technique It may include a deep learning learning step of acquiring information and probability information and detecting a lesion through the probability information obtained at the detected position.

The lesion diagnosis method according to an embodiment includes a lesion verification step of verifying the detection and classification results of lesions based on the comparison of the lesions detected based on the learning results of transfer learning and the lesions detected based on the learning results of the deep learning learning module. may include more.

According to an embodiment, when a still image capture request is made while shooting a video using an endoscope, the suspected lesion region of the acquired still image is set as the region of interest, and then the red (R) channel, the green (G) channel of the region of interest, and a cumulative distribution function (CDF: cumulative distribution function) for each pixel for each of the green channel and blue channel images except for the red channel image among the blue (B) channel images, or for each of the green channel, green channel, and blue channel images function) value, smoothing the histogram for the derived CDF value, converting it into image data with improved contrast, and then synthesizing, and sequentially performing transfer learning and deep learning learning on the synthesized image data to detect lesions. Accordingly, it is possible to objectively diagnose the lesion and improve the accuracy of the lesion.

According to an embodiment, by processing only a still image for the suspected lesion region, it is possible to detect and classify the lesion in real time.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention to be described later, so the present invention is a matter described in such drawings should not be construed as being limited only to
1 is a conceptual diagram for explaining a conventional image processing process.
2 is a diagram illustrating a configuration of a lesion diagnosis system according to an exemplary embodiment.
3 is an exemplary view showing an original image according to an embodiment.
4 is an exemplary view showing an image of each channel according to an embodiment.
5 is an exemplary view showing a pre-processed image according to an embodiment.
6 is another exemplary view showing a pre-processed image according to an embodiment.
7 is a detailed configuration diagram of a lesion detection unit of a system according to an exemplary embodiment.
8 is an overall flowchart illustrating a lesion diagnosis process according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings.

FIG. 2 is a diagram showing the configuration of a lesion diagnosis system according to an embodiment, FIG. 3 is an exemplary view showing an original image of the image acquisition unit of FIG. 2, and FIG. 4 is an image of each channel of the preprocessor shown in FIG. FIG. 5 is an exemplary view showing the preprocessed image of the preprocessing unit shown in FIG. 2 , FIG. 6 is another exemplary view showing the image of the preprocessing unit shown in FIG. 2 , and FIG. 7 is shown in FIG. It is a detailed configuration diagram of the lesion detection unit.

2 to 7 , in the lesion diagnosis system according to an embodiment, among red (R) channel, green (G) channel, and blue (B) channel images of a still image of a region of interest obtained from a video of an endoscope Preprocessing is performed on each of the images of the green channel and blue channel except for the red channel without information on the lesion, and transfer learning and deep learning learning are sequentially performed on the preprocessed image data to detect and diagnose the lesion. In this case, the system may include an image acquisition unit 100 , a preprocessor 200 , and a lesion detection unit 300 .

As shown in FIG. 3 , the image acquisition unit 100 may acquire a still image of a region of interest suspected of a lesion from among videos acquired through an endoscope.

The preprocessor 200 may acquire image data of a predetermined size from the still image of the ROI from the image acquisition unit 100 . Here, the still image includes a red channel image, a green channel image, and a blue channel image, as shown in FIG. 4 , and in this case, referring to FIG. 4 , the red channel image is a lesion part, especially It can be seen that there is little information about gastric ulcer.

Accordingly, the preprocessor 200 may perform preprocessing on each of the remaining green channel and blue channel images except for the red channel image among the red channel, green channel, and blue channel images. For example, the preprocessor 200 derives a cumulative distribution function (CDF) value of each pixel for each of the green channel and blue channel images among the obtained still images, and then sets the derived CDF value to CLAHE (Contrast Limited). It is converted into image data of a predetermined size using the Adaptive Histogram Equalization) technique.

를 Percentile로 매핑 시키는 함수로서, 주어진 확률분포에서 확률변수가 특정 값보다 작거나 같은 확률을 나타낸다. 도출된 Percentile에 대한 누적분포함수(PDF: Percentile Distribution Function)을 적분하면 CDF 값이 도출될 수 있으며, 다음 식 1을 만족한다.The cumulative distribution function (CDF) is x as the value of the distribution,

It is a function that maps to percentage, indicating the probability that a random variable is less than or equal to a specific value in a given probability distribution. By integrating the percentile distribution function (PDF) with respect to the derived percentage, the CDF value can be derived, and the following equation (1) is satisfied.

[Equation 1]

X is an arbitrary variable having a real value, has (a, b], and satisfies a<b.

In addition, the CLAHE technique outputs image data of a predetermined size by smoothing the histogram of the CDF values for each of the green channel and blue channel images, and outputs images for each green channel and blue channel image. The data is synthesized and provided to the lesion detection unit 300 . That is, the CLAHE technique specifies the contrast amplification in the vicinity of a given pixel value by the slope of the conversion function of the cumulative distribution function (CDF), and the histogram value of the corresponding pixel value is proportional to the slope of the CDF. set In this case, the CLAHE technique sets the amplification limit by clamping the histogram to a predefined value before calculating the CDF. Although the process of performing preprocessing on each green channel and blue channel image of a still image using the CDF and CLAHE techniques is not specifically specified in this specification, it should be understood at the level of those skilled in the art.

Accordingly, referring to FIG. 5 , the image of the preprocessor 200 that preprocesses the green channel and blue channel images is uniform and has higher contrast compared to the images preprocessed for still images using the conventional CLAHE technique. , the quality of the still image is improved, and the failure of lesion detection due to a partially dark image can be prevented.

As another example, the preprocessor 200 derives a cumulative distribution function (CDF) value of each pixel for each of the green channel, green channel, and blue channel images among the obtained still images, and then sets the derived CDF value to CLAHE (Contrast Limited Adaptive Histogram Equalization) is used to convert image data of a predetermined size. Image data for each of the output green channel, green channel, and blue channel images are synthesized and provided to the lesion detector 300 .

Accordingly, referring to FIG. 6 , it can be seen that the image of the preprocessor 200 is uniform and the contrast is higher than that of the image of FIG. 5 .

On the other hand, the lesion detection unit 300 receives the image data of the preprocessor 200 and sequentially performs transfer learning and deep learning learning to detect and classify the lesion, and the lesion detection unit 300 is the ImageNet learning module 310 ), a deep learning learning module 320 , and a lesion verification module 330 .

The ImageNet learning module 310 performs transfer learning based on a learning model built based on the image network of image data of a predetermined size of the preprocessor 200, and at this time, the control value derived from the transfer learning of the ImageNet learning module 310 is transmitted to the deep learning learning module 320 as an initial value.

The deep learning learning module 320 that has received the control value of the ImageNet learning module 310 as an initial value performs deep learning learning on the image data of the preprocessor 200 using the control value as an initial value, and based on the learning result It can detect and classify lesions.

Here, the deep learning learning module 320 acquires and detects the location information and probability information of the rectangular coordinate system through the RPN (Region Proposal Network) technique for the features extracted through the ResNET structure consisting of a convolutional layer and a pool layer. It may be provided to detect and classify the lesion through the probability information obtained at the location.

As an example, the deep learning learning module 320 is a preprocessor ( 200), and with the extracted features, the position information and probability score of the rectangular coordinate system are obtained through the RPN technique. The convolutional layer is created using a deep learning method using a convolution value between a patch image and kernels.

Accordingly, the deep learning learning module 320 classifies lesions such as normal, gastric ulcer, and gastric cancer by learning based on a pre-established model for the probability score at the detected position. Through such deep learning learning, it is possible to classify and detect lesions such as normal, gastric ulcer and gastric cancer disclosed in Non-Patent Documents 1 and 2.

The learning result of the deep learning learning module 320 and the learning result of the ImageNet learning module 320 are transmitted to the lesion verification module 330 . The lesion verification module 330 is configured to compare the lesion detected based on the learning result of the transfer learning of the ImageNet learning module 310 with the lesion detected based on the learning result of the deep learning learning module 320. can be verified.

8 is a flowchart illustrating an operation for diagnosing a lesion according to an embodiment, and a method for diagnosing a lesion according to another embodiment will be described with reference to FIG. 8 .

According to an embodiment, in step S11 , a still image of the region of interest for the suspicious lesion region may be acquired from among the videos acquired through the endoscope using the image acquisition unit 100 of FIG. 2 .

According to an embodiment, in steps S12 to S14 , the preprocessor 200 separates the still image into a red channel image, a green channel image, and a blue channel image, and then the remaining green channel images and blue channel images except for the red channel. Alternatively, preprocessing is performed on each of the green channel, green channel, and blue channel images, and then preprocessed image data for each channel image is synthesized and transmitted to the lesion detector 300 .

Next, in steps S15 to S17, the lesion detector 300 performs transfer learning using the ImageNet technique on the preprocessed image data, and then performs learning using the deep learning technique as an initial value as a control value to classify the lesion and Detection is performed, and if the lesion of the transfer learning result matches the lesion of the deep learning learning result, the matched lesion is output.

When a still image capture request is made during video recording using an endoscope, the suspected lesion region of the obtained still image is set as the region of interest and then the lesion is detected based on the set region of interest, so that the lesion can be objectively diagnosed and accuracy can be improved.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. Included are magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

100: image acquisition unit
200: preprocessor
300: lesion detection unit

Claims (9)

an image acquisition unit that acquires a still image of a region of interest for a suspected lesion region among videos acquired through an endoscope;
After correcting the resolution of the still image to a predetermined resolution, a cumulative distribution function (CDF) value for each pixel is derived, and a histogram of the derived CDF value is smoothed to convert it into image data of a predetermined size. preprocessor;
and a lesion detection unit that sequentially performs transfer learning and deep learning on the image data and detects a lesion based on a learning result. The lesion diagnosis system according to claim 1, wherein the still image includes a red channel image, a green channel image, and a blue channel image. The method of claim 2, wherein the pre-processing unit
After deriving a cumulative distribution function (CDF) value for each pixel for each of the green channel and the blue channel, smoothing the histogram for the derived CDF value, and converting it into image data of a predetermined size, each A lesion diagnosis system, characterized in that provided to synthesize the image data for each channel and transmit it to the lesion detection unit. The method of claim 2, wherein the pre-processing unit
For each of the green channel, green channel, and blue channel, a cumulative distribution function (CDF) value for each pixel is derived, and the histogram is smoothed for the derived CDF value, which is converted into image data of a predetermined size. The lesion diagnosis system, characterized in that it is provided to synthesize the image data for each channel and transmit it to the lesion detection unit. According to claim 1, wherein the lesion detection unit,
ImageNet learning module for performing transfer learning based on a learning model built based on an image network of image data of a predetermined size; and
A lesion diagnosis system comprising a deep learning learning module that receives the transfer learning control value of the ImageNet learning module as an initial value, performs deep learning learning, and detects and classifies lesions based on the learning result.
The method of claim 5, wherein the deep learning learning module
The features extracted through the ResNET structure consisting of a convolutional layer and a pool layer are acquired through the RPN (Region Proposal Network) technique to obtain the position information and probability information of the rectangular coordinate system, and the lesion through the probability information obtained from the detected position. A lesion diagnosis system, characterized in that it is provided to detect and classify. According to claim 5, wherein the lesion detection unit,
Further comprising a lesion verification module that verifies the detection and classification results of lesions based on the comparison of the lesions detected based on the learning results of transfer learning of the ImageNet learning module and the lesions detected based on the learning results of the deep learning learning module Characterized by a lesion diagnosis system. In the lesion diagnosis method performed by the lesion diagnosis system of claim 1,
An image acquisition step of acquiring a still image of a region of interest for a suspected lesion region among videos acquired through an endoscope;
A cumulative distribution function (CDF) value for each pixel is derived for each of the green channel, green channel, and blue channel images among the red channel, green channel, and blue channel images of the acquired still image of the region of interest. , a pre-processing step of converting the derived CDF value into image data of a predetermined size by smoothing the histogram;
ImageNet learning step of detecting lesions by performing transfer learning based on a learning model built based on an image network of image data of a predetermined size; and
By receiving the control value of transfer learning of the ImageNet learning module as an initial value, the image data is extracted through the ResnET structure consisting of a predetermined number of convolutional layers and one full layer, and the position information of the rectangular coordinate system and the A lesion diagnosis method, comprising: a deep learning learning step of acquiring probability information and detecting a lesion through the probability information acquired at the detected position. The method of claim 8, wherein the lesion diagnosis method comprises:
Diagnosis of a lesion, characterized in that it further comprises a lesion verification step of verifying the detection and classification result of the lesion based on the comparison of the lesion detected based on the learning result of transfer learning and the lesion detected based on the learning result of the deep learning learning module Way.

Images