KR20210109847A - Method and apparatus for providing digestive endoscopic image information analysis service based on visual integrated learning and detection model - Google Patents

Abstract

An endoscopic image information analysis service providing apparatus in accordance with an embodiment of the present invention includes an apparatus for providing a digestive tract endoscopic image information analysis service based on a visual integrated learning and detection model which comprises: a learning data construction unit constructing a learning model corresponding to endoscopic image information based on prior learning data; an analysis processing unit analyzing real time image information received from an endoscope apparatus using one or more learning models generated in accordance with the learning data; and a reporting processing unit processing the analysis result of the analysis processing unit to provide reporting information corresponding to the real time image information to a user terminal or an administrator terminal, wherein the analysis processing unit performs a visual learning-based multi-label classification analysis and a segmentation decoder-based detection process for digestive tract endoscopic image data on the basis of a pre-constructed visual integrated learning model. The present invention can enhance the reliability and accuracy of endoscopy.

Description

TECHNICAL FIELD The present invention relates to a method and apparatus for providing digestive endoscopic image information analysis service based on visual integrated learning and detection model.

The present invention relates to a service providing method and an apparatus therefor. More specifically, the present invention relates to a method and apparatus for providing an endoscopic image information analysis service of the digestive tract based on a visual integrated learning and detection model.

A method of inserting an endoscope attached to a cable through a subject's mouth or anus is used to obtain information inside the human body, particularly medical information.

According to this endoscopic insertion test method, it is possible to directly control the endoscope through a cable made of a conducting wire or an optical fiber while viewing the endoscopic image, so it is easy to secure data inside the human body, but the movement during control depends on the skill level of the examiner. However, since subjective image interpretation is reflected, there is a problem in that the quality of the examination is different for each examiner.

For example, in the case of a colonoscopy, which is one of the digestive tract, a normal examination must be completed from the anus to the cecum. occurs In addition, polyps in the large intestine may be detected and treated immediately by an experienced examiner, but in other cases, they may be discovered only after post-mortem analysis of endoscopic images.

As such, there are a number of quality factors that depend on the subjective proficiency of the examiner in endoscopy, and this is a factor that lowers the reliability and accuracy of the endoscopy.

The present invention has been devised to solve the above problems, and provides a learning-based gastrointestinal endoscopy image information analysis service capable of real-time analysis to complement the subjective factors of the examiner, thereby improving the reliability and accuracy of the endoscopy The purpose of the present invention is to provide a service providing device and method that can support precision medicine.

An apparatus according to an embodiment of the present invention for solving the above problems, in the apparatus for providing an endoscopic image information analysis service, a learning data construction unit for constructing a learning model corresponding to the endoscopic image information based on prior learning data ; an analysis processing unit that analyzes real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and a reporting processing unit that processes the analysis result of the analysis processing unit and provides reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analysis processing unit is based on a pre-built visual integrated learning model, and a visual integrated learning and detection model-based endoscopic image information analysis service providing apparatus for performing visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing step is based on a pre-built visual integrated learning model , performing visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing step is based on a pre-built visual integrated learning model , A program for executing an endoscopic image information analysis service providing method comprising the step of performing visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data on a computer is recorded as a computer-readable recording medium can be implemented.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing step is based on a pre-built visual integrated learning model , A computer program recorded in a computer-readable recording medium for executing a method for providing an endoscopic image information analysis service on a computer, comprising the step of performing visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data can be implemented as

According to an embodiment of the present invention, it is possible to provide an endoscopic image information analysis service based on a learning algorithm capable of real-time analysis, so that it is possible not only to supplement the examiner's subjective factors, but also to evaluate the examiner's objective proficiency and generate additional learning data. By doing so, it is possible to provide a service providing apparatus and method capable of improving the reliability and accuracy of endoscopy and supporting precision medicine.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.
2 is a block diagram illustrating a service providing apparatus according to an embodiment of the present invention in more detail.
3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.
4 and 5 are diagrams for explaining a model building and analysis process based on a visual integrated learning and detection model having higher efficiency in endoscopic image analysis of the digestive tract among learning models according to an embodiment of the present invention.

The following is merely illustrative of the principles of the invention. Therefore, those skilled in the art will be able to devise various devices which, although not explicitly described or shown herein, embody the principles of the present invention and are included within the spirit and scope of the present invention. Moreover, it is to be understood that all conditional terms and examples listed herein are, in principle, expressly intended solely for the purpose of enabling the concept of the present invention to be understood, and not limited to the specifically enumerated embodiments and states as such. should be

Moreover, it is to be understood that all detailed description reciting the principles, aspects, and embodiments of the invention, as well as specific embodiments, are intended to cover structural and functional equivalents of such matters. It is also to be understood that such equivalents include not only currently known equivalents, but also equivalents developed in the future, i.e., all devices invented to perform the same function, regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing conceptual views of illustrative circuitry embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc. may be tangibly embodied on computer-readable media and be understood to represent various processes performed by a computer or processor, whether or not a computer or processor is explicitly shown. should be

In addition, clear use of terms presented as processor, control, or similar concepts should not be construed as exclusively referring to hardware having the ability to execute software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM (RAM) and non-volatile memory. Other common hardware may also be included.

The above-described objects, features and advantages will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in the description of the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.

First, as shown in FIG. 1 , the system according to an embodiment of the present invention includes a service providing apparatus 100 that provides an endoscopic image analysis service, and one or more user terminals that receive and output analysis information according to service use. 200 , an endoscope apparatus 400 for outputting endoscope image information, and a manager terminal 300 for managing service information.

First, each of the service providing apparatus 100 , the user terminal 200 , the manager terminal 300 , and the endoscope apparatus 400 may be configured as a service network formed by wire or wirelessly through a communication network.

To this end, the service providing apparatus 100 , the user terminal 200 , the manager terminal 300 , and the endoscope apparatus 400 may include one or more communication modules for accessing a communication network for mutual communication.

The communication network may be implemented as a wired network such as, for example, a local area network (LAN), a wide area network (WAN), or a value added network (VAN). In addition, the communication network is a mobile radio communication network, satellite communication network, Bluetooth (Bluetooth), Wibro (Wireless Broadband Internet), HSDPA (High Speed Downlink Packet Access), Wi-Fi (Wi-Fi), LTE (Long Term Evolution) ) can be implemented with any kind of wireless network, such as If necessary, the communication network may be a network in which wired and wireless are mixed.

In such a configuration, the service providing apparatus 100 may process data transmission and reception with the service registered user terminal 200, and according to the learning data set and built by the manager terminal 300, the endoscope apparatus ( 400) of real-time image information can be analyzed and processed.

In addition, the service providing apparatus 100 may transmit the reporting information generated according to the analysis process to the user terminal 200, and the user terminal 200 may check real-time image information of the endoscope apparatus 400 together with the reporting information. can

Here, the reporting information may be output through a display device and an audio output device, and the location, area, detection sound alert, screen blink alert, sickness information in the form of a report, lesion image information (continuous lesion image frame) etc.), found foreign material information, etc. may be exemplified.

According to this reporting information, the user can be provided with analysis information corresponding to the endoscopic image information in real time, the examination status can be objectively grasped, and the lesion that is difficult to visually recognize according to the skill level can be easily inspected. video analysis service can be provided.

In addition, the service providing apparatus 100 may not only output the reporting information, but also quantitatively calculate the examiner's examination quality according to the analysis result. The quantitatively calculated inspection quality information may be transmitted to the manager terminal 300 and used for service management.

And, the service providing apparatus 100 may extract additional learning data that can contribute to improving the accuracy of the learning model from the analyzed result data, and this learning data is reapplied to the pre-learning model of the pre-generated learning data building unit. This allows additional learning to improve accuracy and efficiency.

Such analysis result information, learning data, and reporting information may be stored in a database inside or outside the service providing apparatus 100 , and may be managed by the service providing apparatus 100 or the manager terminal 300 .

In particular, the service providing apparatus 100 may include a learning data construction unit for constructing learning data, an analysis processing unit for performing analysis processing, and a reporting processing unit for processing and providing analysis information to the user terminal 200 .

More specifically, the analysis processing unit of the service providing apparatus 100 enables an anonymization of the patient's personal information in the image signal input from the endoscope apparatus 400 and pre-processing, and for this purpose, the analysis area according to the apparatus corresponds to the endoscopic image information can be set individually.

Here, the anonymized preprocessed image may have a time difference between frames (time difference input like frame1, frame3, frame 5), and may be processed to be stored separately in a buffer composed of a circular queue. have.

The training data building unit can build a plurality of analysis models pre-generated by the pre-learning data in parallel, and when the anonymized pre-processed image data is input in the analysis processing unit, the AI processing result data of the corresponding learning model can be output. have.

For example, the learned first model may detect a lesion in preprocessed image data in real time and output result information.

Accordingly, the reporting processing unit may process the result information into a reporting screen interface and provide it to the user terminal 200 .

In addition, the analysis processing unit may determine a specific time section in which detailed lesion analysis is required among the endoscopic image data, and the images stored in the buffer corresponding to the determined time section may be input to the learned second model and used to predict the result. have.

In response to the prediction analysis result according to the application of the learning model of the images used for prediction, the reporting processing unit displays the images stored in the buffer together with the average, standard deviation, and maximum/minimum graphs in a tile form and displays the predicted values of each image. A screen interface displayed on the screen may be provided to the user terminal 200 .

2 is a block diagram illustrating a service providing apparatus 100 according to an embodiment of the present invention in more detail.

Referring to FIG. 2 , first, the training data construction unit may include a modeling processing unit and an advanced sample mining unit, and the analysis processing unit may include an analysis reporting engine and a learning model processing unit.

The modeling processing unit builds one or more learning models based on the pre-learning data.

To this end, the modeling processing unit may include a data anonymization processing unit, a data tagging processing unit, and a learning data modeling unit.

The data anonymization processing unit removes personal information of the pre-learning data, extracts only the necessary area training data according to the model, and delivers it to the data tagging processing unit.

The data tagging processing unit may perform tagging processing such as labeling corresponding to the training data. The tagging process may include a process in which tagging information is directly assigned by an administrator according to a learning algorithm, or a process in which tagging information is automatically assigned according to a predetermined classification criterion may be performed.

In addition, the learning data modeling unit models one or more learning models required for real-time endoscopic image analysis. Here, each of the learning models may be provided to the learning model processing unit of the analysis processing unit, and the analysis processing unit may perform learning-based analysis of real-time image information based on each learning model.

Here, a plurality of the learning models can be generated and processed in parallel with respect to one real-time endoscopic image, thereby enabling faster and faster image processing. The learning model for this may be, for example, a lesion detection model that directly detects a lesion, a type analysis model that classifies and analyzes the shape of a feature extracted from an image, and a quality control model that quantitatively calculates the quality of an examiner. have.

Accordingly, the analysis processing unit may perform image analysis and reporting based on a learning model through an analysis reporting engine that analyzes a real-time endoscopic image.

Here, the image processing module may input an image, store a buffer, provide it to the learning model processing unit, provide and manage analysis results, and the image correction module may perform partial correction such as distortion, enlargement, or reduction of the image. can

In addition, the analysis result may be stored in the analysis result storage module, and may be transmitted to the reporting processing unit. Here, the reporting processing unit may process the analysis result for the user terminal 200 or the manager terminal 300 , respectively, and transmit necessary reporting information to each user terminal 200 or the manager terminal 300 in real time.

In addition, the reporting information processed by the reporting processing unit may be configured in the form of historical data such as log information as well as real-time analysis and used for post-analysis.

Accordingly, the service providing apparatus 100 according to an embodiment of the present invention can not only enable real-time analysis of endoscopic images, but also build a database that enables post-learning data construction and inspection quality management. Possible domains can be expanded based on learning.

In particular, the service providing apparatus 100 according to an embodiment of the present invention may include an advanced sample mining unit, and the advanced sample mining unit may extract advanced samples that are difficult to analyze in an analysis process based on a learning model, and the The selected advanced learning data may be transferred to the modeling processing unit and processed to be used to improve the accuracy of the learning model.

To this end, the advanced sample mining unit may include a data anonymization processing unit that performs pre-processing of anonymization of new learning data, a new data analysis unit that obtains analysis information of new data, and an advanced data selection unit that selects advanced data.

3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.

Referring to FIG. 3 , first, the service providing apparatus 100 constructs learning data according to pre-processing and classification from pre-collected endoscopic image data.

And, the service providing apparatus 100 generates an endoscopic image analysis model based on pre-learning data.

Thereafter, the service providing apparatus 100 analyzes and corrects real-time image information received from the endoscope apparatus 400 based on the model, and generates reporting data.

Thereafter, the service providing apparatus 100 processes the generated reporting data and provides an analysis service of a real-time endoscopic image to the user terminal 200, where the reporting data may be stored and managed by matching with real-time image information.

On the other hand, the service providing apparatus 100 may select sample data for advancing image analysis modeling from the analysis data, and use this to construct new learning data. This process is performed by the operation of the advanced sample mining unit described above. can be processed.

4 and 5 are diagrams for explaining a model construction and analysis process based on a visual integrated learning and detection model having higher efficiency in endoscopic image analysis of the digestive tract among learning models according to an embodiment of the present invention.

Referring to Figure 4, the gastrointestinal endoscopy image analysis based on the visual integrated learning and detection model is an integrated predictive model analysis that can classify and segment images calculated through classification-based predictive model processing and visual-based image segmentation model processing. may include

Here, the classification model analysis is a primary classification model, and there is an insufficient aspect for medical images having similar lesions and other areas, and it is difficult to know where the lesions and other areas are classified.

In contrast, the visual-based image segmentation model can perform sophisticated segment-specific classification, but may act as noise in a medical environment.

Accordingly, the analysis processing unit according to an embodiment of the present invention can perform gastrointestinal endoscopic image analysis based on a visual integrated prediction model capable of image classification and segmentation in which each model can complement each other, and the learning data construction unit It is possible to build a learning model by collecting learning data for creating a visual integrated learning model for

Here, the integrated prediction model may be generated as a learning model based on an artificial intelligence algorithm, and by determining an appropriate classification and analysis model according to the lesion location based on the learning model, more accurate image analysis and lesion detection are possible.

More specifically, referring to FIG. 5 , FIG. 5 is a visual integrated learning model creation and detection process for detecting polyps of the colonoscopy in the digestive tract and performing quality assessment of the colonoscopy. As specifically illustrated, it may be used to construct a learning model of the learning data processing unit and generate detection reporting information by the analysis processing unit.

More specifically, referring to FIG. 5 , the visual integrated learning and detection model may include a multi-label classification model, a detector model, and a decoder, and may be driven by a learning data building unit and an analysis processing unit.

The learning data construction unit learns the global multi label classifier corresponding to the colonoscopy image data by the multi label classification model, and then the layer output information calculated according to the learning It can be used as characteristic information of a detector model for polyp detection.

Here, the multi-label classification information may be applied as global classification information, and a multi-label classification model for this may be pre-built. The multi-label classification model may be a model for classifying all classes included in the input image.

For example, all classifier classes used as targets in the digestive tract image may have various sizes and textures. Accordingly, applying the global prediction to the entire image may decrease accuracy.

Accordingly, the multi-label classification model according to an embodiment of the present invention can input the input image to the learning network according to various scales based on a Gaussian pyramid algorithm, and calculate the average probability information accordingly It can be output as label classification information.

In addition, the detector model may include a learning model for detecting a bounding box for the target region from the digestive tract image.

The detector model may use a convolutional layer obtained by convolution processing the layer output calculated from the multi-label classification model as a pretrained backbone network, and these layers are image features for detection It can be used as expression information.

And, the visual integrated learning model may include a segmentation decoder. The segmentation decoder may perform instance segmentation on a class (polyps, lesions, etc.) requiring particularly detailed analysis among image instances detected in the detector model.

Accordingly, the decoder can upscaling the feature information of the input instance (the convolutional output of the multi-label classification model used in the detector model) to enable more accurate detection, and this upscaling process reduces the deconvolution operation. made possible through

Accordingly, the visual integrated learning model can be used for lesion detection and quality control in the analysis processing unit, and the convolutional layer output of the multi-label classification model is used as the feature expression information input of the detector model and the decoder, and detection in the detector model Some class instances that have been processed may be segmented through a decoder.

The method according to the present invention described above may be produced as a program to be executed by a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed in a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (4)

In the endoscopic image information analysis service providing apparatus,
a learning data building unit for constructing a learning model corresponding to the endoscopic image information based on the pre-learning data;
an analysis processing unit that analyzes real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and
and a reporting processing unit that processes the analysis result of the analysis processing unit and provides reporting information corresponding to the real-time image information to a user terminal or an administrator terminal,
The analysis processing unit performs visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data based on a pre-built visual integrated learning and detection model.
A device for providing an endoscopic image information analysis service based on a visual integrated learning and detection model. According to claim 1,
The multi-label classification model is based on a Gaussian pyramid algorithm, inputs and processes the input gastrointestinal endoscopy image to a learning network according to a plurality of scales, calculates the corresponding average probability information, to output label classification information
Endoscopy image information analysis service providing device. According to claim 1,
In the segmentation decoder, the segmentation decoder performs instance segmentation on a specific class requiring detailed analysis among the image instances detected in the detector model on which the multi-label classification analysis has been performed and outputs it.
Endoscopy image information analysis service providing device. In the method of providing an endoscopic image information analysis service,
building a learning model corresponding to the endoscopic image information based on the pre-learning data;
analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and
processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal,
The analyzing includes performing visual learning-based multi-label classification analysis and segmentation decoder-based detection processing of gastrointestinal endoscopy image data, based on a pre-built visual integrated learning and detection model.
A method of providing an endoscopic image information analysis service based on a visual integrated learning and detection model.

Images