KR20210110440A - Method and apparatus for providing endoscopic image information analysis service for lesion analysis of digestive tract based on predictive model - Google Patents

Abstract

An endoscopic image information analysis service providing apparatus in accordance with an embodiment of the present invention comprises: a learning data construction unit constructing a learning model corresponding to endoscopic image information based on prior learning data; an analysis processing unit analyzing real time image information received from an endoscope apparatus using one or more learning models generated in accordance with the learning data; and a reporting processing unit processing the analysis result of the analysis processing unit to provide reporting information corresponding to the real time image information to a user terminal or an administrator terminal, wherein the analysis processing unit applies the real time image information to one or more estimation models of the learning models, and analyzes digestive tract lesion probability information in a specific frame section in accordance with an occurrence of an event, and the reporting processing unit provides the reporting information including the image information and the digestive tract lesion probability information of the specific frame section to the user terminal or the administrator terminal. The present invention can enhance reliability and accuracy of endoscopy.

Description

Method and apparatus for providing endoscopic image information analysis service for lesion analysis of digestive tract based on predictive model

The present invention relates to a service providing method and an apparatus therefor. More specifically, the present invention relates to a learning-based endoscopic image information analysis service providing method and apparatus for performing lesion analysis of the digestive tract based on a predictive model.

A method of inserting an endoscope attached to a cable through a subject's mouth or anus is used to obtain information inside the human body, particularly medical information.

According to this endoscopic insertion test method, it is possible to directly control the endoscope through a cable made of a conducting wire or an optical fiber while viewing the endoscopic image, so it is easy to secure data inside the human body, but the movement during control depends on the skill level of the examiner. However, since subjective image interpretation is reflected, there is a problem in that the quality of the examination is different for each examiner.

For example, in the case of a colonoscopy, which is one of the digestive tract, a normal examination must be completed from the anus to the cecum. occurs In addition, polyps in the large intestine may be detected and treated immediately by an experienced examiner, but in other cases, they may be discovered only after post-mortem analysis of endoscopic images.

As such, there are a number of quality factors that depend on the subjective proficiency of the examiner in endoscopy, and this is a factor that lowers the reliability and accuracy of the endoscopy.

The present invention has been devised to solve the above problems, and provides a learning-based gastrointestinal endoscopy image information analysis service capable of real-time analysis to complement the subjective factors of the examiner, thereby improving the reliability and accuracy of the endoscopy The purpose of the present invention is to provide a service providing device and method that can support precision medicine.

An apparatus according to an embodiment of the present invention for solving the above problems, in the apparatus for providing an endoscopic image information analysis service, a learning data construction unit for constructing a learning model corresponding to the endoscopic image information based on prior learning data ; an analysis processing unit that analyzes real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and a reporting processing unit that processes the analysis result of the analysis processing unit and provides reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analysis processing unit converts the real-time image information into one of the learning models Applied to the above prediction model, and analyzes gut lesion probability information in a specific frame section according to the occurrence of an event, the reporting processing unit provides reporting information including image information and the gut lesion probability information of the specific frame section to the user terminal Alternatively, it is provided as an administrator terminal.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing comprises: converting the real-time image information into one of the learning models Applying to the above prediction model, and analyzing the digestive tract lesion probability information in a specific frame section according to the occurrence of an event, wherein the providing includes image information of the specific frame section and the gut lesion probability information and providing the reporting information to the user terminal or the manager terminal.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing comprises: converting the real-time image information into one of the learning models Applying to the above prediction model, and analyzing the digestive tract lesion probability information in a specific frame section according to the occurrence of an event, wherein the providing includes image information of the specific frame section and the gut lesion probability information It may be implemented as a computer-readable recording medium in which a program for executing an endoscopic image information analysis service providing method including the step of providing reporting information to the user terminal or the manager terminal on a computer is recorded.

A method according to an embodiment of the present invention for solving the above-described problems, the method comprising: building a learning model corresponding to the endoscopic image information based on the prior learning data; analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal, wherein the analyzing comprises: converting the real-time image information into one of the learning models Applying to the above prediction model, and analyzing the digestive tract lesion probability information in a specific frame section according to the occurrence of an event, wherein the providing includes image information of the specific frame section and the gut lesion probability information It may be implemented as a computer program recorded in a computer-readable recording medium for executing a method of providing an endoscopic image information analysis service including the step of providing reporting information to the user terminal or the manager terminal in a computer.

According to an embodiment of the present invention, it is possible to provide an endoscopic image information analysis service based on a learning algorithm capable of real-time analysis, so that it is possible not only to supplement the examiner's subjective factors, but also to evaluate the examiner's objective proficiency and generate additional learning data. By doing so, it is possible to provide a service providing apparatus and method capable of improving the reliability and accuracy of endoscopy and supporting precision medicine.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.
2 is a block diagram illustrating a service providing apparatus according to an embodiment of the present invention in more detail.
3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.
4 to 5 are diagrams for explaining a lesion analysis process of the digestive tract based on the predictive model of the analysis processing unit.

The following is merely illustrative of the principles of the invention. Therefore, those skilled in the art will be able to devise various devices which, although not explicitly described or shown herein, embody the principles of the present invention and are included within the spirit and scope of the present invention. Moreover, it is to be understood that all conditional terms and examples listed herein are, in principle, expressly intended solely for the purpose of enabling the concept of the present invention to be understood, and not limited to the specifically enumerated embodiments and states as such. should be

Moreover, it is to be understood that all detailed description reciting the principles, aspects, and embodiments of the invention, as well as specific embodiments, are intended to cover structural and functional equivalents of such matters. It is also to be understood that such equivalents include not only currently known equivalents, but also equivalents developed in the future, i.e., all devices invented to perform the same function, regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing conceptual views of illustrative circuitry embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc. may be tangibly embodied on computer-readable media and be understood to represent various processes performed by a computer or processor, whether or not a computer or processor is explicitly shown. should be

In addition, clear use of terms presented as processor, control, or similar concepts should not be construed as exclusively referring to hardware having the ability to execute software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM (RAM) and non-volatile memory. Other common hardware may also be included.

The above-described objects, features and advantages will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in the description of the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.

First, as shown in FIG. 1 , the system according to an embodiment of the present invention includes a service providing apparatus 100 that provides an endoscopic image analysis service, and one or more user terminals that receive and output analysis information according to service use. 200 , an endoscope apparatus 400 for outputting endoscope image information, and a manager terminal 300 for managing service information.

First, each of the service providing apparatus 100 , the user terminal 200 , the manager terminal 300 , and the endoscope apparatus 400 may be configured as a service network formed by wire or wirelessly through a communication network.

To this end, the service providing apparatus 100 , the user terminal 200 , the manager terminal 300 , and the endoscope apparatus 400 may include one or more communication modules for accessing a communication network for mutual communication.

The communication network may be implemented as a wired network such as, for example, a local area network (LAN), a wide area network (WAN), or a value added network (VAN). In addition, the communication network is a mobile radio communication network, satellite communication network, Bluetooth (Bluetooth), Wibro (Wireless Broadband Internet), HSDPA (High Speed Downlink Packet Access), Wi-Fi (Wi-Fi), LTE (Long Term Evolution) ) can be implemented with any kind of wireless network, such as If necessary, the communication network may be a network in which wired and wireless are mixed.

In such a configuration, the service providing apparatus 100 may process data transmission and reception with the service registered user terminal 200, and according to the learning data set and built by the manager terminal 300, the endoscope apparatus ( 400) of real-time image information can be analyzed and processed.

In addition, the service providing apparatus 100 may transmit the reporting information generated according to the analysis process to the user terminal 200, and the user terminal 200 may check real-time image information of the endoscope apparatus 400 together with the reporting information. can

Here, the reporting information may be output through a display device and an audio output device, and the location, area, detection sound alert, screen blink alert, sickness information in the form of a report, lesion image information (continuous lesion image frame) etc.), found foreign material information, etc. may be exemplified.

According to this reporting information, the user can be provided with analysis information corresponding to the endoscopic image information in real time, the examination status can be objectively grasped, and the lesion that is difficult to visually recognize according to the skill level can be easily inspected. video analysis service can be provided.

In addition, the service providing apparatus 100 may not only output the reporting information, but also quantitatively calculate the examiner's examination quality according to the analysis result. The quantitatively calculated inspection quality information may be transmitted to the manager terminal 300 and used for service management.

And, the service providing apparatus 100 may extract additional learning data that can contribute to improving the accuracy of the learning model from the analyzed result data, and this learning data is reapplied to the pre-learning model of the pre-generated learning data building unit. This allows additional learning to improve accuracy and efficiency.

Such analysis result information, learning data, and reporting information may be stored in a database inside or outside the service providing apparatus 100 , and may be managed by the service providing apparatus 100 or the manager terminal 300 .

In particular, the service providing apparatus 100 may include a learning data construction unit for constructing learning data, an analysis processing unit for performing analysis processing, and a reporting processing unit for processing and providing analysis information to the user terminal 200 .

More specifically, the analysis processing unit of the service providing apparatus 100 enables an anonymization of the patient's personal information in the image signal input from the endoscope apparatus 400 and pre-processing, and for this purpose, the analysis area according to the apparatus corresponds to the endoscopic image information can be set individually.

Here, the anonymized preprocessed image may have a time difference between frames (time difference input like frame1, frame3, frame 5), and may be processed to be stored separately in a buffer composed of a circular queue. have.

The training data building unit can build a plurality of analysis models pre-generated by the pre-learning data in parallel, and when the anonymized pre-processed image data is input in the analysis processing unit, the AI processing result data of the corresponding learning model can be output. have.

For example, the learned first model may detect a lesion in preprocessed image data in real time and output result information.

Accordingly, the reporting processing unit may process the result information into a reporting screen interface and provide it to the user terminal 200 .

In addition, the analysis processing unit may determine a specific time section in which detailed lesion analysis is required among the endoscopic image data, and the images stored in the buffer corresponding to the determined time section may be input to the learned second model and used to predict the result. have.

In response to the prediction analysis result according to the application of the learning model of the images used for prediction, the reporting processing unit displays the images stored in the buffer together with the average, standard deviation, and maximum/minimum graphs in a tile form and displays the predicted values of each image. A screen interface displayed on the screen may be provided to the user terminal 200 .

2 is a block diagram illustrating a service providing apparatus 100 according to an embodiment of the present invention in more detail.

Referring to FIG. 2 , first, the training data construction unit may include a modeling processing unit and an advanced sample mining unit, and the analysis processing unit may include an analysis reporting engine and a learning model processing unit.

The modeling processing unit builds one or more learning models based on the pre-learning data.

To this end, the modeling processing unit may include a data anonymization processing unit, a data tagging processing unit, and a learning data modeling unit.

The data anonymization processing unit removes personal information of the pre-learning data, extracts only the necessary area training data according to the model, and delivers it to the data tagging processing unit.

The data tagging processing unit may perform tagging processing such as labeling corresponding to the training data. The tagging process may include a process in which tagging information is directly assigned by an administrator according to a learning algorithm, or a process in which tagging information is automatically assigned according to a predetermined classification criterion may be performed.

In addition, the learning data modeling unit models one or more learning models required for real-time endoscopic image analysis. Here, each of the learning models may be provided to the learning model processing unit of the analysis processing unit, and the analysis processing unit may perform learning-based analysis of real-time image information based on each learning model.

Here, a plurality of the learning models can be generated and processed in parallel with respect to one real-time endoscopic image, thereby enabling faster and faster image processing. The learning model for this may be, for example, a lesion detection model that directly detects a lesion, a type analysis model that classifies and analyzes the shape of a feature extracted from an image, and a quality control model that quantitatively calculates the quality of an examiner. have.

Accordingly, the analysis processing unit may perform image analysis and reporting based on a learning model through an analysis reporting engine that analyzes a real-time endoscopic image.

Here, the image processing module may input an image, store a buffer, provide it to the learning model processing unit, provide and manage analysis results, and the image correction module may perform partial correction such as distortion, enlargement, or reduction of the image. can

In addition, the analysis result may be stored in the analysis result storage module, and may be transmitted to the reporting processing unit. Here, the reporting processing unit may process the analysis result for the user terminal 200 or the manager terminal 300 , respectively, and transmit necessary reporting information to each user terminal 200 or the manager terminal 300 in real time.

In addition, the reporting information processed by the reporting processing unit may be configured in the form of historical data such as log information as well as real-time analysis and used for post-analysis.

Accordingly, the service providing apparatus 100 according to an embodiment of the present invention can not only enable real-time analysis of endoscopic images, but also build a database that enables post-learning data construction and inspection quality management. Possible domains can be expanded based on learning.

In particular, the service providing apparatus 100 according to an embodiment of the present invention may include an advanced sample mining unit, and the advanced sample mining unit may extract advanced samples that are difficult to analyze in an analysis process based on a learning model, and the The selected advanced learning data may be transferred to the modeling processing unit and processed to be used to improve the accuracy of the learning model.

To this end, the advanced sample mining unit may include a data anonymization processing unit that performs pre-processing of anonymization of new learning data, a new data analysis unit that obtains analysis information of new data, and an advanced data selection unit that selects advanced data.

3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.

Referring to FIG. 3 , first, the service providing apparatus 100 constructs learning data according to pre-processing and classification from pre-collected endoscopic image data.

And, the service providing apparatus 100 generates an endoscopic image analysis model based on pre-learning data.

Thereafter, the service providing apparatus 100 analyzes and corrects real-time image information received from the endoscope apparatus 400 based on the model, and generates reporting data.

Thereafter, the service providing apparatus 100 processes the generated reporting data and provides an analysis service of a real-time endoscopic image to the user terminal 200, where the reporting data may be stored and managed by matching with real-time image information.

On the other hand, the service providing apparatus 100 may select sample data for advancing image analysis modeling from the analysis data, and use this to construct new learning data. This process is performed by the operation of the advanced sample mining unit described above. can be processed.

4 to 5 are diagrams for explaining a lesion analysis process of the digestive tract based on the predictive model of the analysis processing unit.

4 to 5, the analysis processing unit according to an embodiment of the present invention, based on one or more learning models provided by the learning data construction unit, to detect a suspected lesion (tumor) in the gastrointestinal endoscopy and the detection information may be transmitted to the reporting processing unit.

When checking the results by analyzing the images in units of frames in the existing real-time endoscopy of the digestive tract, the results may change frame by frame without outputting the same prediction results in a specific section. As such, it may be difficult to diagnose specific lesions.

Accordingly, the analysis processing unit according to an embodiment of the present invention may analyze the classification result in a specific section, process the result through the reporting processing unit, and display it on the screen to assist the diagnosis. Also, the reporting processing unit may segment the lesion area and output a detailed shape through the display.

To this end, the analysis processing unit may store the latest image among the gastrointestinal endoscopy images in the buffer, obtain the prediction result by applying the image of the buffer to the prediction model at a specific moment, and calculate the result value according to the prediction result to the reporting processing unit can transmit The reporting processing unit may process prediction reporting information according to numerical calculations, such as average and standard deviation, from the result value and output it together with the image through the screen interface of the user terminal 200 .

Specifically, the analysis processing unit can designate the size of the buffer to store the image to be used for analysis, and allows the image to be stored in the buffer in the form of a circular queue.

In addition, the buffer-stored image may be input to the prediction model at the point in time when analysis is required to obtain a prediction result, and may be transmitted to the reporting processing unit. In addition, the reporting processing unit processes the average, standard deviation, and images stored in the buffer in the form of tiles, obtained from the prediction results, and generates reporting information to be displayed together on the prediction result screen of each image, and the generated reporting Information may be output to the user terminal 200 .

In contrast, as shown in FIG. 4 , an image input in units of frames in each moving image may be input to a prediction model that is generated and provided in advance by the training data construction unit. The analysis processing unit may determine occurrence of an event corresponding to a partial frame requiring detailed analysis among the prediction results, and separately store the partial frame image extracted from the buffer. Here, the prediction model may be composed of a plurality of parallel models according to various prediction objects, and a final result probability may be calculated according to an average value of results extracted from each model.

Accordingly, images and probability information stored in the buffer may appear on the screen of the user terminal 200 , and the predicted value may be used as auxiliary information for precise diagnosis.

5 illustrates a parallel model-based learning process according to an embodiment of the present invention, it is possible to minimize the probability noise that may occur when inferring a lesion of a frame of moving image data using a model learned from a still image.

As shown in FIG. 5, a time series fusion module constructed by merging the late fusion and early fusion methods is constructed, and the learning model of the still image is convoluted. By applying frames with high reliability among the model and past N frame results and using them for lesion detection, it is possible to minimize probability noise and improve detection precision.

The method according to the present invention described above may be produced as a program to be executed by a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed in a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (4)

In the endoscopic image information analysis service providing apparatus,
a learning data building unit for constructing a learning model corresponding to the endoscopic image information based on the pre-learning data;
an analysis processing unit that analyzes real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and
and a reporting processing unit that processes the analysis result of the analysis processing unit and provides reporting information corresponding to the real-time image information to a user terminal or an administrator terminal,
The analysis processing unit applies the real-time image information to one or more predictive models among the learning models, and analyzes digestive tract lesion probability information in a specific frame section according to the occurrence of an event,
The reporting processing unit provides reporting information including the image information of the specific frame section and the digestive tract lesion probability information to the user terminal or the manager terminal
Endoscopy image information analysis service providing device. According to claim 1,
The reporting processing unit,
The lesion probability information and the image information of the specific frame section are processed in a tile form to generate reporting information to be displayed together on the prediction result screen of each image.
Endoscopy image information analysis service providing device. According to claim 1,
The analysis processing unit,
Configure a time series fusion module composed by merging the late fusion and early fusion methods,
lesion detection using the time series fusion module, a model obtained by convolutional processing of a learning model of a still image, and frames with high reliability among the past N frame results.
Endoscopy image information analysis service providing device. In the method of providing an endoscopic image information analysis service,
building a learning model corresponding to the endoscopic image information based on the pre-learning data;
analyzing real-time image information received from the endoscope apparatus using one or more learning models generated according to the learning data; and
processing the analysis result of the analysis processing unit, and providing reporting information corresponding to the real-time image information to a user terminal or an administrator terminal,
The analyzing step is
applying the real-time image information to one or more predictive models among the learning models, and analyzing the digestive tract lesion probability information in a specific frame section according to the occurrence of an event,
The providing includes providing reporting information including image information of the specific frame section and the digestive tract lesion probability information to the user terminal or the manager terminal
A method of providing an endoscopic image information analysis service.

Images