KR102393661B1 - Endoscope system for multi image, image providing method of the system, and a recording medium having computer readable program for executing the method - Google Patents

Abstract

Disclosed are a multi-image endoscopy system, an image providing method thereof, and a recording medium recording a computer readable program for executing the method. The multi-image endoscope system uses a plurality of images obtained using a plurality of light sources of different wavelength bands, and includes an image output unit, an image analysis unit, and a danger warning unit. The image output unit outputs an image corresponding to some preset wavelength bands among different wavelength bands, the image analysis unit analyzes a plurality of images to determine an abnormal part, and the risk warning unit determines a preset abnormal part as a result of image analysis Outputs a hazard warning signal to the user. According to such a configuration, the examiner pays proper attention to diagnose using a normally familiar image, but when an abnormal situation occurs in the image, it is possible to read the image more carefully. while being able to observe precisely.

Description

Multi-image endoscopy system, image providing method, and recording medium recording a computer readable program for executing the method METHOD}

The present invention relates to an endoscope system and an image providing method thereof, and more particularly, to an endoscope system capable of acquiring multiple images such as a white light image, a fluorescence image, and a spectroscopic image, and a method for providing the image.

In medical optical imaging devices such as endoscopes, imaging techniques for improving the contrast of lesions as well as techniques for improving image resolution are being developed.

Specifically, in addition to the traditional white light imaging technique showing the surface structure of a tissue, a spectral imaging technique showing the spectral characteristics of the tissue in contrast, fluorescence imaging showing the fluorescence characteristics of the tissue in contrast Multi-modality imaging technology such as technology is being introduced as an endoscopic imaging technology.

In particular, as a fluorescence imaging technique, techniques for imaging with image contrast, such as autofluorescence of human tissue or induced fluorescence caused by binding of a fluorescent molecule injected through a blood vessel to a tissue, are being developed.

As such, the multi-image endoscopy system that provides image information using light of different wavelengths, such as visible light, narrowband spectroscopy, and fluorescence, provides various levels of image information (anatomy, function, molecule) necessary for diagnosis in real time to medical staff. It can greatly improve the accuracy of medical diagnosis because it can provide

However, since many patients are examined in a short time at the actual examination site, it is practically impossible for the endoscopy examiner to focus on all of the images constituting the multiple images to grasp image information.

Accordingly, since the examiner does not find a lesion that can be sufficiently detected in the image, the accuracy and reliability of endoscopic diagnosis may be deteriorated.

US 20120316421 A1

The present invention has been devised to solve the above-mentioned problems of the prior art, and it is to provide a multi-image endoscopy system and a method for providing the image, which enable faster and more precise observation of a multi-endoscopic image including various diagnostic information. The purpose.

In order to achieve the above object, a multi-image endoscope system according to the present invention is a system using a plurality of images obtained using a plurality of light sources of different wavelength bands, and includes an image output unit, an image analysis unit, and a danger warning unit. .

The image output unit outputs an image corresponding to some preset wavelength bands among different wavelength bands, the image analysis unit analyzes a plurality of images to determine an abnormal part, and the risk warning unit determines a preset abnormal part as a result of image analysis Outputs a hazard warning signal to the user.

According to such a configuration, the examiner makes a diagnosis with appropriate attention using a normally familiar image, but when an abnormal situation occurs in the image, the image can be read more carefully. while being able to observe precisely.

In this case, the multi-image endoscopy system may further include an abnormal image output unit for outputting an image in which an abnormal region is displayed in a preset output format. According to such a configuration, it is possible to effectively transmit not only whether an abnormal situation occurs in the image, but also information on an abnormal part or abnormal state to the examinee.

In addition, the image analyzer divides a plurality of images into preset patches, respectively, and combines the feature values of the same patch region of different modality images (eg, white light image, spectroscopic image, fluorescence image) to determine whether the patch region is abnormal. can According to such a configuration, it is possible to automatically determine an abnormal region using feature values of the same regions in different modality images.

In this case, a weight related to importance may be given to a feature value of a patch region corresponding to a part of the image among the feature values, and in this case, the weight may be determined by machine learning using preset image data. According to such a configuration, it is possible to more efficiently adjust the weights of factors for abnormality determination using machine learning.

In addition, the image analyzer divides a plurality of images into preset patches, respectively, and uses a combination of input sequences of feature values of the same patch region of different modality images (eg, white light image, spectroscopic image, and fluorescence image) to determine the abnormality of the patch region. It can be determined whether or not there is, and in this case, the combination of the feature value input order can be determined by machine learning using preset image data. According to such a configuration, it is possible to use a resource such as a memory more efficiently by sequentially using images instead of simultaneously.

Also, the light of different wavelength bands may be images obtained by time division using a light source device and an image detection device synchronized with each other. According to this configuration, since the light source device and the image detection device are synchronized and used in time division, it is not necessary to have all image processing devices corresponding to different wavelength ranges in the endoscope system to obtain multiple images. It is possible to quickly provide multiple images while simplifying.

In addition, the invention implementing the system in the form of a method and a recording medium recording a computer readable program for executing the method are disclosed together.

According to the present invention, the endoscopic image examiner makes a diagnosis using a normally familiar image, but when an abnormal situation occurs in the image, the image can be read more carefully, so that the multiple endoscopic image containing various diagnostic information can be viewed more carefully. It allows for quick and precise observation.

In addition, it is possible to effectively deliver not only whether an abnormal situation occurs in the image, but also information on an abnormal part or abnormal state to the examinee.

Also, it is possible to automatically determine an abnormal region by using the feature values of the same regions in different modality images.

In addition, by using machine learning, it is possible to more efficiently adjust the weights of factors for abnormality determination.

In addition, by using images sequentially rather than simultaneously, resources such as memory can be used more efficiently.

In addition, since the light source device and the image detection device are synchronized and used in time division, it is not necessary to have all image processing devices corresponding to different wavelength regions in the endoscope system to obtain multiple images, thereby simplifying the structure of the optical system and providing multiple images. can be provided quickly.

1 is a schematic block diagram of a multi-image endoscopy system according to an embodiment of the present invention;
2 to 4 are photographs of endoscopic images obtained using white light, fluorescence, and narrowband spectroscopy, respectively.
5 is a diagram schematically illustrating an operation process of an image analysis unit.
6 is a diagram illustrating an example of a process in which an image analysis unit analyzes an image using different modality images at the same time;
7 is a diagram illustrating an example of a process in which an image analyzer sequentially uses different modality images to analyze an image;
8 to 10 are diagrams for explaining an example in which the danger warning unit outputs a danger warning signal through a display screen;
11 to 13 are views each showing examples of a form in which an abnormal image output unit outputs an abnormal region image;

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a schematic block diagram of a multi-image endoscopy system according to an embodiment of the present invention. In FIG. 1 , a multi-image endoscopy system 100 is a system using a plurality of images acquired using a plurality of light sources of different wavelength bands, and includes an image output unit 110 , an image analysis unit 120 , and a danger warning unit ( 130 ), and an abnormal image output unit 140 .

In FIG. 1 , the image output unit 110 is implemented as a graphic processing unit, the image analysis unit 120 is implemented as an abnormality image analysis unit, and the risk warning unit 130 is implemented as a risk indicator. Each of the components of the multi-image endoscopy system 100 in FIG. 1 may be implemented with only hardware, but it will be generally implemented with hardware and software operating on the hardware.

The image output unit 110 outputs images corresponding to some preset wavelength bands among different wavelength bands. In this case, the different wavelength bands may be implemented in any preset combination, and white light, spectroscopy of white light, and fluorescence will generally be used. In addition, the output wavelength band may also be selected from among various input wavelength bands, but in general, a white light wavelength band is preferable. For example, a real-time visible light image may be output to the display device.

2 to 4 are photographs of endoscopic images obtained using white light, fluorescence, and narrowband spectroscopy, respectively. As can be seen in FIGS. 2 to 4 , when light of different wavelength bands is used, images having different characteristics can be obtained, respectively.

In this case, the light of different wavelength bands may be images obtained by time division using a light source device and an image detection device synchronized with each other. According to this configuration, since the light source device and the image detection device are synchronized and used in time division, it is not necessary to have all image processing devices corresponding to different wavelength ranges in the endoscope system to obtain multiple images. It is possible to quickly provide multiple images while simplifying.

To this end, for example, a memory device for continuously acquiring and storing at least one type of image among a visible light image, a fluorescence image, and a spectroscopic image in a time division manner may be included, and is implemented as Trimodal image buffers in FIG. 1 .

The image analyzer 120 analyzes the plurality of acquired images to determine an abnormal region from the diagnostic image. More specifically, information related to a lesion or disease (screening, localization, segmentation, classification) is analyzed by using at least one of fluorescence and spectroscopic images acquired while displaying a visible light image.

5 is a diagram schematically illustrating an operation process of an image analysis unit. First, a process of detecting a lesion by acquiring the lesion image shown in FIGS. 2 to 4 in real time, localizing and segmenting the lesion, and then analyzing the lesion area is shown.

In this case, the image analyzer 120 may determine whether the patch region is abnormal by dividing the plurality of images into preset patches, respectively, and combining feature values of the same patch region of different modality images. According to such a configuration, it is possible to automatically determine an abnormal region using feature values of the same regions in different modality images.

6 is a diagram illustrating an example of a process in which the image analyzer analyzes an image using different modality images at the same time. In FIG. 6 , it can be seen a process of analyzing an image by simultaneously using images respectively acquired in different wavelength regions for the same region.

Here, the image analyzer 120 may apply a previously disclosed Convolutional Neural Network (CNN) deep learning technique to extract feature values of the same patch region of different modality images.

To this end, a weight may be assigned to a feature value of a patch region corresponding to some image among the feature values, and in this case, the weight may be determined by machine learning using preset image data. According to such a configuration, it is possible to more efficiently adjust the weights of factors for abnormality determination using machine learning. In FIG. 1 , a configuration required for the image analysis unit 120 to perform image analysis using machine learning is shown as a training database.

In addition, the image analysis unit 120 may divide a plurality of images into preset patches, respectively, and determine whether the patch region is abnormal by using a combination of input sequences of feature values of the same patch region of different modality images (eg, For example, using a Recurrent Neural Network (RNN)), in this case, the feature value input order combination may be determined by machine learning using preset image data.

According to such a configuration, it is possible to use a resource such as a memory more efficiently by sequentially using images instead of simultaneously. 7 is a diagram illustrating an example of a process in which an image analyzer sequentially uses different modality images to analyze an image.

The danger warning unit 130 outputs a danger warning signal to the user when a preset abnormal part is determined as a result of image analysis. In other words, it performs a function of automatically determining the risk of a disease or lesion based on the analyzed data and notifying the user immediately.

According to this configuration, the examiner pays proper attention to diagnose using the normally familiar image, but when an abnormal situation occurs in the image, it is possible to read the image more carefully, so that the multi-endoscopic image containing various diagnostic information can be performed more quickly. while being able to observe precisely.

In this case, the form of the warning signal may be implemented in various forms that the user can consider, and output a specific mark on the display screen checked by the examinee, output a specific warning sound, or give vibration to the endoscope device carried by the examinee can be exemplified.

8 to 10 are diagrams for explaining an example in which the danger warning unit outputs a danger warning signal through the display screen, when there is no indication, when only a preset danger warning signal is output, the danger warning signal and the lesion area Examples of output cases are shown respectively.

The abnormal image output unit 140 outputs an image in which the abnormal region is displayed in a preset output format. According to such a configuration, it is possible to effectively transmit not only whether an abnormal situation occurs in the image, but also information on an abnormal part or abnormal state to the examinee.

11 to 13 are diagrams each showing examples of a form in which the abnormal image output unit outputs abnormal region images. 11 shows an example of outputting an abnormal region together on an image output by the image output unit 110 on a single screen, FIG. 12 shows an example of outputting a picture-in-picture (PNP) format on a single screen, and FIG. An example of output on a separate screen in the form of dual display is shown.

The present invention also discloses a specific method for utilizing images of various modalities for one scene, such as a visible light-spectral-fluorescence image, using a multi-image endoscopy system, and a specific method for applying the same to an actual clinical workflow.

More specifically, based on the visible light-narrowband spectroscopic-fluorescence image data acquired in real time, 1) basic image data for screening is displayed for rapid diagnosis, and 2) the acquired multi-endoscopic image data is analyzed in real time to The presence or absence, location of the lesion, the shape of the lesion (contour), and the type of lesion are automatically determined/analyzed in real-time (abnormality image analysis unit), and 3) the optimal display format is automatically determined based on the real-time analysis data. Control to change (display mode control unit), 4) automatically determine the level of risk, severity level of a disease or lesion based on real-time analysis data, and notify the user immediately, so that medical staff can use the Disclosed is an image providing method of an intelligent multi-image endoscopy system including a function (risk indicator) for selectively inducing precise observation of spectroscopic and fluorescence images.

According to this configuration, in order to effectively observe the precise medical data of the visible light-narrowband spectroscopic-fluorescence image, the optimal display mode is provided, and at the same time, the risk of disease or lesion is automatically determined and immediately notified to the user. It induces precise observation of spectroscopic and fluorescence images selectively only in the case of a specific risk or higher, and through this, both accuracy and efficiency of diagnosis can be secured.

Although the present invention has been described with reference to some preferred embodiments, the scope of the present invention should not be limited thereto, but should also extend to modifications or improvements of the above embodiments supported by the claims.

100: multi-image endoscopy system
110: video output unit
120: image analysis unit
130: hazard warning unit
140: abnormal video output unit

Claims (17)

A multi-image endoscopy system using a plurality of images obtained using a white light source, a fluorescent light source, and a white light spectral light source, respectively, comprising:
an image output unit for outputting an image corresponding to the white light source;
an image analysis unit that analyzes the plurality of images to determine an abnormal region;
a danger warning unit for outputting a danger warning signal to the user when a preset abnormal part is determined as a result of the image analysis; and
and an abnormal image output unit for outputting an abnormal image in which the abnormal region is displayed in a preset output format,
The image analysis unit determines the abnormal region by using the images obtained by using the white light source, the fluorescent light source, and the spectral light source, respectively,
The abnormal image output unit determines an output form of the abnormal image from among a plurality of preset output forms corresponding to the abnormal image based on the analysis data of the image analysis unit,
The multi-image endoscopy system, characterized in that the images corresponding to the fluorescent light source and the spectral light source are images obtained by time division using a light source device and an image detection device synchronized with each other.
delete The method according to claim 1,
The image analysis unit divides the plurality of images into preset patches, extracts feature values for each patch by applying a Convolutional Neural Network (CNN) to the same patch regions of different modality images, and combines the extracted feature values to obtain the A multi-image endoscopy system, characterized in that it is determined whether there is an abnormality in the patch area.
4. The method according to claim 3,
The multi-image endoscopy system according to claim 1, wherein the image analyzer assigns a weight to a feature value of a patch region corresponding to a part of the image among the feature values.
5. The method according to claim 4,
The multi-image endoscopy system, characterized in that the weight is determined by machine learning using preset image data.
The method according to claim 1,
The image analyzer divides the plurality of images into preset patches, respectively, and determines whether the patch region is abnormal by using a combination of input sequences of feature values of the same patch region of different modality images. system.
7. The method of claim 6,
The multi-image endoscopy system, characterized in that the combination of the feature value input sequence is determined by machine learning using preset image data.
delete An image providing method of a multi-image endoscopy system using a plurality of images obtained using a white light source, a fluorescent light source, and a white light spectral light source, respectively, comprising:
an image output step of outputting an image corresponding to the white light source;
an image analysis step of analyzing the plurality of images to determine an abnormal region;
a hazard warning step of outputting a hazard warning signal to the user when a preset abnormality is determined as a result of the image analysis; and
and an abnormal image output step of outputting an abnormal image in which the abnormal region is displayed in a preset output format,
In the image analysis step, the abnormal region is determined by using the images obtained using the white light source, the fluorescent light source, and the spectral light source, respectively,
The abnormal image output step determines an output form of the abnormal image from among a plurality of preset output forms corresponding to the abnormal image based on the analysis data of the image analysis step,
The image providing method of the multi-image endoscopy system, characterized in that the images corresponding to the fluorescent light source and the spectral light source are images obtained by time division using a light source device and an image detection device synchronized with each other.
delete 10. The method of claim 9,
The image analysis step divides the plurality of images into preset patches, extracts feature values for each patch by applying a Convolutional Neural Network (CNN) to the same patch regions of different modality images, and combines the extracted feature values. An image providing method of a multi-image endoscopy system, characterized in that determining whether the patch region is abnormal.
12. The method of claim 11,
In the image analysis step, a weight is assigned to a feature value of a patch region corresponding to a part of the image among the feature values.
13. The method of claim 12,
The weight is determined by machine learning using preset image data.
10. The method of claim 9,
The image analysis step divides the plurality of images into preset patches, and determines whether the patch region is abnormal by using a combination of input sequences of feature values of the same patch region of different modality images. A method for providing an image of an endoscopic system.
15. The method of claim 14,
The image providing method of a multi-image endoscopy system, characterized in that the combination of the feature value input order is determined by machine learning using preset image data.
delete A recording medium in which a computer readable program for executing the method of any one of claims 9 and 11 to 15 is recorded.

Images