KR102672710B1 - Apparatus and method for synthesizing endoscopic image - Google Patents

Abstract

The present invention relates to an endoscopic image synthesis device and method using artificial intelligence. The present disclosure can provide a device for synthesizing virtual endoscopic images. The virtual endoscopic image synthesis device includes an input unit configured to receive at least one of text containing attribute values related to the endoscopic image, lesion location information, mask information, and an actually captured endoscopic image, the input text, and the location of the lesion. A processor configured to synthesize a virtual endoscopic image based on at least one of information, mask information, and an actually captured endoscopic image, wherein the processor combines an actually captured endoscopic image and at least one attribute related to the captured endoscopic image. Using at least one of the value, lesion location information, and mask information as learning data, at least one of text containing attribute values related to the endoscopic image, lesion location information, mask information, and actually captured endoscopic image is input, and a virtual A virtual endoscopic image synthesis device may be provided, characterized in that it includes an image synthesis module including an artificial intelligence model learned to synthesize endoscopic images.

Description

Endoscopic image synthesis device and synthesis method {APPARATUS AND METHOD FOR SYNTHESIZING ENDOSCOPIC IMAGE}

The present invention relates to an endoscopic image synthesis device and method using artificial intelligence.

Artificial intelligence image synthesis technology is a technology that adds or modifies digital elements to actual images through a combination of computer vision and artificial intelligence technology. This technology is used in various fields such as virtual reality (VR), augmented reality (AR), film production, advertising, and game development.

Recently, medical image synthesis technology using artificial intelligence has been used in the medical field for various purposes such as diagnosis, treatment, and research. For example, medical image synthesis technology can be used to improve the resolution of medical images or remove noise. Through this, medical experts can perform more accurate and detailed analysis and more accurately determine the patient's condition. As another example, medical image synthesis can be used to visualize diseases such as lesions or tumors. By combining medical image data and artificial intelligence algorithms, the location, size, and shape of the tumor can be visually expressed more clearly.

However, a large number of image data are required to learn various medical artificial intelligence models such as lesion detection, segmentation, and classification, but it is difficult to secure a large number of images due to the nature of the medical field. Accordingly, there is a need for technology to synthesize images that can be used to train artificial intelligence models.

Republic of Korea Patent Registration No. 10-2513394

The purpose of the present disclosure is to provide an apparatus and method for synthesizing large amounts of endoscopic images used to learn a stomach, small intestine, or large intestine lesion prediction model.

In addition, the present disclosure provides an image synthesis device and method that allows easily synthesizing a virtual endoscopic image using at least one of a plurality of attribute values, location information of the lesion to be created, mask information, and an actually captured endoscopic image. It is for that purpose.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The present disclosure can provide a device for synthesizing virtual endoscopic images. The virtual endoscopic image synthesis device includes an input unit configured to receive at least one of text containing attribute values related to the endoscopic image, lesion location information, mask information, and an actually captured endoscopic image, the input text, and the location of the lesion. A processor configured to synthesize a virtual endoscopic image based on at least one of information, mask information, and an actually captured endoscopic image, wherein the processor combines an actually captured endoscopic image and at least one attribute related to the captured endoscopic image. Using at least one of the value, lesion location information, and mask information as learning data, at least one of text containing attribute values related to the endoscopic image, lesion location information, mask information, and actually captured endoscopic image is input, and a virtual A virtual endoscopic image synthesis device may be provided, characterized in that it includes an image synthesis module including an artificial intelligence model learned to synthesize endoscopic images.

In one embodiment, the display unit is configured to display an input window that receives at least one of the actually captured endoscopic image, attribute values related to the captured endoscopic image, location information of the lesion, and the mask information, The learning data may be generated by matching at least one attribute value input through the input window to the captured endoscopic image.

In one embodiment, the processor further includes a sentence generation module that generates text containing the attribute values based on attribute values related to the endoscope image, and the learning data includes at least one input through the input window. It can be generated by matching the text generated based on the attribute value of and at least one attribute value input through the input window to the captured endoscopic image.

In one embodiment, the sentence generation module generates a plurality of texts based on attribute values related to the captured endoscopic image, and the learning data may be generated by matching the plurality of texts to the captured endoscopic image. there is.

In one embodiment, attribute values related to the captured endoscopic image may include attribute values related to global stand SES-CD score evaluation items.

In one embodiment, the attribute values related to the captured endoscopic image include the type of organ imaged, the age of the imaged subject, the name of the diagnosis, the field of view of the endoscopic imaging device, the field of view of the lesion, the location of the imaged lesion, the shape of the imaged lesion, It may include attribute values related to at least one of the size of the imaged lesion, the type of endoscopic imaging device, the type of endoscopic procedure tool, and the endoscopic imaging situation.

In one embodiment, the artificial intelligence model is configured to synthesize a lesion image with an actually captured endoscopic image based on at least one of text containing attribute values related to the endoscopic image, location information of the lesion, and the mask information. It can be learned.

In addition, the present disclosure uses an actually captured endoscopic image and at least one attribute value related to the captured endoscopic image as learning data, text containing attribute values related to the endoscopic image, location information of the lesion, mask information, and actually captured endoscopic image. A step of learning an artificial intelligence model to synthesize a virtual endoscopic image by receiving at least one of the endoscopic images, text containing attribute values related to the endoscopic image, lesion location information, mask information, and at least one of the actually captured endoscopic images. A virtual endoscopic image comprising receiving one input and the artificial intelligence model synthesizing a virtual endoscopic image based on at least one of the input text, lesion location information, mask information, and actually captured endoscopic image. A synthesis method may be provided.

In addition to this, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be further provided.

In addition to this, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

The image synthesis device and method according to the present disclosure enable the synthesis of a large number of images of various lesions even without actual endoscopic data. A large amount of image data generated in this way can be used as training data for an artificial intelligence model to predict lesions through endoscopic images. That is, according to the present disclosure, a large amount of learning data (virtual endoscope images) for learning an artificial intelligence model can be generated using only a small amount of actually captured endoscope images.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is an overall system diagram of the present disclosure.
Figure 2 is a block diagram of a server included in the image synthesis device of the present disclosure.
Figure 3 is a block diagram of a terminal included in the image synthesis device of the present disclosure.
Figure 3 is a block diagram of a processor included in the image synthesis device of the present disclosure.
Figure 5 is a flowchart of an image synthesis method according to the present disclosure.
Figure 6 is a flowchart showing a method of learning an artificial intelligence model included in the image synthesis device according to the present disclosure.
Figures 7 and 8 are user interface screens for generating learning data for an artificial intelligence model.
9 and 10 are conceptual diagrams showing an example of an artificial intelligence model included in the image synthesis device according to the present disclosure.

Like reference numerals refer to like elements throughout this disclosure. This disclosure does not describe all elements of the embodiments, and general content or overlapping content between embodiments in the technical field to which this disclosure pertains is omitted. The term 'part, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'part, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

In this specification, 'device according to the present disclosure' includes all various devices that can perform computational processing and provide results to the user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server device is a server that processes information by communicating with external devices and may include an application server, computing server, database server, file server, game server, mail server, proxy server, and web server.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ), all types of handheld wireless communication devices, and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD). may include.

The image synthesis device according to the present disclosure may be implemented by at least one of a server and a terminal. Specifically, the artificial intelligence model learning device according to the present disclosure may be implemented by either a server or a terminal, or may be implemented as a system through data transmission and reception between the server and the terminal.

Hereinafter, an image synthesis device according to the present disclosure will be described.

Referring to FIG. 1, the image synthesis device according to the present disclosure may include at least one of a server and a terminal.

The server 10 is connected to the terminal 20 through a network, and can receive data necessary for image synthesis from the terminal 20 and then transmit the image synthesis result to the terminal 20.

Meanwhile, it is obvious to those skilled in the art that the terminal 20 is not limited to the above-described portable terminal, and may include a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a processor. do.

As described above, the image synthesis device according to the present disclosure can be implemented through data transmission and reception between the server 10 and the terminal 20.

Hereinafter, each of the server 10 and the terminal 20 for implementing the image synthesis device according to the present disclosure will be described.

Figure 2 is a block diagram of a server included in the image synthesis device of the present disclosure.

The server 100 according to the present disclosure may include at least one of a communication unit 110, a storage unit 120, and a processor 130.

The communication unit 110 may communicate with at least one of a terminal, an external storage (eg, a database 140), an external server, and a cloud server.

Meanwhile, in an external server or cloud server, it may be configured to perform at least a portion of the role of the processor 130. In other words, data processing or data computation can be performed on an external server or cloud server, and the present invention does not impose any special restrictions on this method.

Meanwhile, the communication unit 110 may support various communication methods depending on the communication standard of the communicating object (eg, electronic device, external server, device, etc.).

For example, the communication unit 110 supports wireless LAN (WLAN), wireless-fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, digital living network alliance (DLNA), wireless broadband (WiBro), and WiMAX ( World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication) , Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal) Serial Bus) technology may be used to communicate with a communication target.

Next, the storage unit 120 can be configured to store various information related to the present invention. In the present invention, the storage unit 120 may be provided in the device itself according to the present invention. Alternatively, at least a portion of the storage unit 120 may refer to at least one of a database (DB) 140 and cloud storage (or cloud server). In other words, it can be understood that the storage unit 120 is sufficient as a space to store information necessary for the device and method according to the present invention, and there are no restrictions on physical space. Accordingly, hereinafter, the storage unit 120, the database 140, external storage, and cloud storage (or cloud server) will not be separately distinguished, and all will be referred to as the storage unit 120.

Next, the processor 130 may be configured to control the overall operation of the device related to the present invention. The processor 130 may process signals, data, information, etc. input or output through the components discussed above, or provide or process appropriate information or functions to the user.

The processor 130 may include at least one CPU (Central Processing Unit) and perform the function according to the present invention.

At least one component may be added or deleted in response to the performance of the components shown in FIG. 2. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the device.

Hereinafter, the terminal included in the image synthesis device of the present disclosure will be described in detail.

Figure 3 is a block diagram of a terminal included in the image synthesis device of the present disclosure.

Referring to FIG. 3, the terminal 200 according to the present disclosure may include a communication unit 210, an input unit 220, a display unit 230, and a processor 240. The components shown in FIG. 3 are not essential for implementing the image synthesis device according to the present disclosure, so the terminal described in this specification may have more or fewer components than the components listed above.

Among the components, the communication unit 210 may include one or more components that enable communication with an external device, for example, a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and location information. It may contain at least one of the modules.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-1302 (recommended standard 1302), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The input unit 220 is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and includes at least one of at least one camera, at least one microphone, and a user input unit. can do. Voice data or image data collected from the input unit can be analyzed and processed as a user's control command.

The display unit 230 is intended to generate output related to vision, hearing, or tactile sensation, and may include at least one of a display unit, an audio output unit, a haptip module, and an optical output unit. A touch screen can be implemented by forming a layered structure with the touch sensor or being integrated with the display unit. This touch screen functions as a user input unit that provides an input interface between the device and the user, and can simultaneously provide an output interface between the device and the user.

The display unit displays (outputs) information processed in the device. For example, the display unit may display execution screen information of an application program (eg, an application) running on the device, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

In addition to the above-described components, the above-described terminal may further include an interface unit and memory.

The interface unit serves as a passageway for various types of external devices connected to this device. These interface units include a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and a port for connecting a device equipped with an identification module (SIM) ( port), an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In this device, appropriate control related to external devices connected to the interface unit can be performed.

The memory can store data that supports various functions of this device and programs for processor operation, and can store input/output data (e.g., music files, still images, videos, etc.), and can store data that supports various functions of this device. A number of application programs (application programs or applications) running in the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

These memories include flash memory type, hard disk type, SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), and multimedia card micro type. , card-type memory (e.g., SD or It may include at least one type of storage medium among (only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory may be a database that is separate from the device, but is connected wired or wirelessly.

Meanwhile, the above-described terminal includes a processor 240. The processor includes a memory that stores data for an algorithm for controlling the operation of components within the device or a program that reproduces the algorithm, and at least one processor (not shown) that performs the above-described operations using the data stored in the memory. It can be implemented as: At this time, the memory and processor may each be implemented as separate chips. Alternatively, the memory and processor may be implemented as a single chip.

Meanwhile, as shown in FIG. 4, a processor included in at least one of the server and the terminal may include a plurality of modules for implementing an image synthesis device to be described later. Specifically, the processor 300 may include a sentence generation module 310 and an image synthesis module 320. The artificial intelligence model learning method described later is described as being implemented by the operations of the modules, but the performance of each step described later does not necessarily need to be performed by the modules.

In addition, the processor may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in the drawings below on the present device.

Meanwhile, at least one component may be added or deleted in accordance with the performance of the components shown in FIGS. 1 to 4. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the device.

Below, the artificial intelligence described in the present invention will be described in detail.

Functions related to artificial intelligence according to the present disclosure are operated through processors and memories mounted on the above-described servers and terminals. The processor may consist of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, a plurality of weights may be updated so that loss or cost values obtained from the artificial intelligence model are reduced or minimized during the learning process. Artificial neural networks may include deep neural networks (DNN), for example, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the examples described above.

According to an exemplary embodiment of the present disclosure, a processor may implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that allows machines to learn by imitating human biological neurons. Methodology of artificial intelligence includes supervised learning, in which the answer (output data) to the problem (input data) is determined by providing input data and output data together as training data according to the learning method, and only input data is provided without output data. In unsupervised learning, in which the solution (output data) to the problem (input data) is not determined, and a reward is given from the external environment whenever an action is taken in the current state, , It can be divided into reinforcement learning, which conducts learning in the direction of maximizing these rewards. In addition, artificial intelligence methodologies can be divided according to the architecture, which is the structure of the learning model. The architecture of widely used deep learning technology is convolutional neural network (CNN) and recurrent neural network (RNN). , Transformer, generative adversarial networks (GAN), etc.

The devices and systems may include artificial intelligence models. An artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. Artificial intelligence models may be composed of neural networks (or artificial neural networks) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities. Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. By way of example, a device may include an input layer, a hidden layer, and an output layer. The neural network that makes up the device can infer the result (output) to be predicted from arbitrary input (input) by changing the weight of neurons through learning.

The processor creates a neural network, trains or learns a neural network, performs calculations based on received input data, generates an information signal based on the results, or generates a neural network. The network can be retrained. Neural network models include CNN (Convolution Neural Network), R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), and RNN, such as GoogleNet, AlexNet, and VGG Network. (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network , LSTM (Long Short-Term Memory) Network, Classification Network, etc., but the processor is not limited thereto and may include one or more processors for performing operations according to neural network models. For example, a neural network may include a deep neural network.

Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), and LSTM. (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield) Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network) ), Liquid State Machine (LSM), Extreme Learning Machine (ELM), Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), It may include at least one of KN (Kohonen Network), AN (Attention Network), DDPM (Denoising diffusion probabilistic model), Latent Diffusion, and Stable Diffusion, but is not limited to this and may include any neural network. Technicians will understand.

According to an exemplary embodiment of the present disclosure, the processor may support a Convolution Neural Network (CNN), a Region with Convolution Neural Network (R-CNN), a Region Proposal Network (RPN), a Recurrent Neural Network (RNN), such as GoogleNet, AlexNet, VGG Network, etc. ), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC/QA, Text Analysis, Dialog System, GPT-3 , GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, and Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation for ResNet data intelligence. , but is not limited to this. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Below, an image synthesis method using the above-described components will be described in detail. The image synthesis method described later is implemented through data transmission and reception between the server and the terminal described above, and some steps of the method may be performed in either the server or the terminal. However, it is obvious to those skilled in the art that the method described below is not limited to this and can be performed independently in either a server or a terminal.

Figure 5 is a flowchart of an image synthesis method according to the present disclosure.

Referring to FIG. 5, a method of learning an artificial intelligence model is performed based on at least one of the captured endoscopic image, attribute values related to the captured endoscopic image, location information of the lesion, mask information, and lesion image (S110).

The image synthesis module 320 may include an artificial intelligence model that receives predetermined data and synthesizes a virtual endoscope image. The artificial intelligence model can be learned based on actually captured endoscope images.

In one embodiment, the captured endoscopic image may be any one of a gastric endoscopic image, a small intestine endoscopic image, and a colonoscopic image, but is not limited thereto.

The artificial intelligence model uses the actually captured endoscopic image, at least one attribute value related to the captured endoscopic image, lesion location information, mask information, and lesion image as learning data, and provides text containing attribute values related to the endoscopic image. , it can be learned to synthesize a virtual endoscopic image by receiving lesion location information and mask information. A detailed description of the learning data and learning method for learning an artificial intelligence model is described later.

Next, a step is performed in which at least one of text containing attribute values related to the endoscopic image, lesion location information, and mask information is input to the learned artificial intelligence model (S120).

The user can input attribute values related to the endoscope image or text containing the attribute values through the input unit included in the terminal 20.

In one embodiment, when attribute values related to an endoscopic image are input, a plurality of items and at least one attribute value corresponding to each of the plurality of items may be displayed on the display unit included in the terminal 20. The user can input attribute values related to the endoscopic image by selecting at least one of the displayed attribute values.

In this case, the sentence generation module 310 may generate text in the form of a sentence based on the input attribute value.

For example, when a plurality of attribute values 'wall view', 'TI location', 'flat', and 'polyp' are input by the user, the sentence generation module 310 generates 'make a wall view picture taken at the You can create the sentence 'colon TI location, and flat polyp at the center of the picture.'

Meanwhile, the terminal 20 can receive text input in the form of a completed sentence including attribute values related to the endoscope image from the user.

For example, the terminal 20 may receive text from the user in the form of the sentence 'make a wall view picture taken at the colon TI location, and flat polyp at the center of the picture.'

Meanwhile, attribute values related to an endoscopic image may include a plurality of attribute values corresponding to each of a plurality of items.

In one embodiment, the attribute value may include an attribute value related to a global stand SES-CD score evaluation item.

For example, the attribute values include the type of organ imaged, the age of the imaged subject, the name of the diagnosis, the field of view of the endoscopic imaging device, the field of view of the lesion, the location of the imaged lesion, the shape of the imaged lesion, the size of the imaged lesion, and the endoscopic image. It may include attribute values corresponding to each type of device, type of endoscopic procedure tool, and endoscopic imaging situation.

The 'type of organ imaged' item is the type of organ that is the subject of imaging of the endoscopic image. In one embodiment, the attribute value corresponding to the 'type of organ imaged' item may be any one of the stomach and large intestine. .

The 'Age of the subject of filming' is the age of the person who is the subject of the endoscopic image. In one embodiment, the attribute value corresponding to the 'age of the subject of photography' item may be any one of 'under 10 years old', '10 to 20 years old', '20 to 60 years old', and 'over 60 years old'. .

The ‘Diagnosis Name’ item refers to the diagnosis name corresponding to the lesion found in the endoscopic image. In one embodiment, the attribute value corresponding to the 'diagnosis name' item may be any one of ulcer, polyp, and cancer.

The 'field of view of the endoscope imaging device' item refers to the field of view viewed by the endoscope device. In one embodiment, the attribute value corresponding to the 'field of view of the endoscopic imaging device' item may be any one of lumen, wall, and J turn.

The 'lesion field of view' item refers to the imaging direction of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'lesion field of view' item may be any one of front, oblique, and lateral.

The 'Location of imaged lesion' item refers to the organ where the lesion included in the endoscopic image is located. In one embodiment, the attribute values corresponding to the 'location of imaged lesion' item are esophagus (E1/2), stomach (S1/2/3/4/5), duodenum (D1/2/3), Terminal ileum (TI), cecum (C), ascending colon (AC), hepatic flexure (HF), transverse colon (TC), splenic flexure (SF), descending colon (DC), sigmoid colon (SC), and rectum ( It may be any one of R).

The 'Shape of the imaged lesion' item refers to the shape of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'shape of photographed lesion' item is one of flat, sessile, pedunculated, oval, and irregular. It can be.

The 'size of imaged lesion' item refers to the size of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'size of imaged lesion' item may be any one of aphthous, large, and very large.

The 'Type of endoscopic imaging device, type of endoscopic procedure tool' item refers to the type of equipment used for endoscopic imaging. In one embodiment, the attribute value corresponding to the item 'Type of endoscopic imaging device, type of endoscopic surgical tool' may be one of forceps and snare.

The 'endoscopic imaging situation' item refers to under what circumstances the endoscopic imaging was performed. In one embodiment, the attribute value corresponding to the 'endoscopic imaging situation' item may be any one of biopsy, dye injection, and narrow-band imaging.

Meanwhile, without being limited to the above-described embodiment, the attribute value matching the captured endoscopic image is at least one of the presence or absence of circular plastic (part of the endoscope device) on the outer edge of the image and the presence of boxes and letters in the endoscopic image. It may contain attribute values corresponding to .

Additionally, the user may input at least one of an actually captured endoscopic image, lesion location information, and mask information for endoscopic image synthesis.

When an actually captured endoscopic image is input, an endoscopic image synthesis may be performed by applying the user-entered attribute values, lesion location information, and mask information to the input endoscopic image.

Meanwhile, the location information of the lesion defines the location on the endoscope image where the user wants to create the lesion. In one embodiment, the location information of the lesion may include the lesion's center coordinates, X-direction thickness, and y-direction thickness values.

Meanwhile, mask information defines areas that appear dark and areas that appear bright depending on the amount of light and the direction of light irradiation during endoscopic imaging, or areas that appear clearly and areas that appear unclear due to the presence or absence of floating objects during endoscopic imaging. . In one embodiment, the mask information may be an image of the same size as the endoscopic image to be synthesized, and the pixel value for each area constituting the image may be defined as one of a plurality of preset pixel values.

For example, the pixel value constituting the mask information may be any one of three types of pixel values, each of which includes a pixel value defining a dark area, a pixel value defining a bright area, and the presence of floaters. It may be a pixel value that defines an area.

Furthermore, the area where the floating matter exists can be defined with more detailed pixel values. For example, the area where the floating matter exists may be defined by pixel values segmented according to the transparency of the water, the density of the floating matter present in the water, etc.

In another embodiment, mask information may define background characteristics of an endoscopic image to be generated. For example, the mask information includes the overall degree of inflammation inside the organ that is the target of endoscopic imaging, the state of the organ according to age, the degree of tortuosity of the organ, the state of pressure on the lesion or around the lesion to image the lesion at a specific angle, etc. You can define a value that defines .

Finally, a step of synthesizing a virtual endoscopic image based on the input text, at least one of the actually captured endoscopic image, lesion location information, and mask information is performed (S130).

The artificial intelligence model can synthesize a virtual endoscope image that reflects the attribute values included in the text input from the user described above. The artificial intelligence model can synthesize virtual endoscopic images in which lesions of various sizes and shapes are created at various locations.

Additionally, the artificial intelligence model can synthesize an endoscopic image using at least one of the text input from the user, the actually captured endoscopic image input from the user, location information of the lesion, and mask information.

In one embodiment, the artificial intelligence model utilizes the actually captured endoscopic image input from the user when synthesizing the endoscopic image, and includes attribute values input from the user in the actually captured endoscopic image (or attribute values extracted from text), Endoscopic images can be synthesized by applying lesion location information and mask information.

In another embodiment, the artificial intelligence model may determine the location of the lesion included in the endoscopic image to be synthesized by using the location information of the lesion input from the user when synthesizing the endoscopic image.

In another embodiment, the artificial intelligence model may use mask information input from the user during endoscopic mirror image synthesis to determine clearly visible and not clearly visible areas in the endoscopic image to be synthesized.

Meanwhile, the artificial intelligence model can synthesize an endoscopic image by applying the information input from the user at once for synthesizing the endoscopic image, or it can synthesize the image by applying different types of information input from the user in stages.

For example, when a user inputs an actually captured endoscopic image, text containing at least one attribute value, lesion location information, and mask information to synthesize an endoscopic image, the artificial intelligence model uses the actually captured endoscopic image as input. An image to which attribute values are applied is first synthesized, an image to which the location information of the lesion is applied is secondarily synthesized to the primarily synthesized image, and the mask information is applied to the secondarily synthesized image to produce a final image. can be synthesized. At this time, each step-by-step synthesized image may be displayed on the terminal 20.

Hereinafter, a method for learning an artificial intelligence model included in the image synthesis device according to the present disclosure will be described in detail.

FIG. 6 is a flowchart showing a method of learning an artificial intelligence model included in the image synthesis device according to the present disclosure, FIGS. 7 and 8 are user interface screens for generating learning data of the artificial intelligence model, and FIGS. 9 and 10 are a flowchart of the present disclosure. This is a conceptual diagram showing an example of an artificial intelligence model included in an image synthesis device according to .

Referring to FIG. 6, when the terminal 20 generates learning data for learning an artificial intelligence model, a step of displaying the captured endoscopic image and a plurality of attribute values is performed (S210).

To this end, the display unit included in the terminal 20 may display an input window for inputting the captured endoscopic image and attribute values related to the captured endoscopic image.

In one embodiment, the attribute value may include an attribute value related to a global stand SES-CD score evaluation item.

For example, the attribute values may be displayed on the terminal 20 for each item, and the displayed items include the type of organ imaged, age of the imaged subject, name of diagnosis, field of view of the endoscopic imaging device, field of view of the lesion, and type of lesion imaged. The item may be at least one of the following: location, shape of the imaged lesion, size of the imaged lesion, type of endoscopic imaging device, type of endoscopic procedure tool, and endoscopic imaging situation.

The 'type of organ imaged' item is the type of organ that is the subject of imaging of the endoscopic image. In one embodiment, the attribute value corresponding to the 'type of organ imaged' item may be any one of the stomach and large intestine. .

The 'Age of the subject of filming' is the age of the person who is the subject of the endoscopic image. In one embodiment, the attribute value corresponding to the 'age of the subject of photography' item may be any one of 'under 10 years old', '10 to 20 years old', '20 to 60 years old', and 'over 60 years old'. .

The ‘Diagnosis Name’ item refers to the diagnosis name corresponding to the lesion found in the endoscopic image. In one embodiment, the attribute value corresponding to the 'diagnosis name' item may be any one of ulcer, polyp, and cancer.

The 'field of view of the endoscope imaging device' item refers to the field of view viewed by the endoscope device. In one embodiment, the attribute value corresponding to the 'field of view of the endoscopic imaging device' item may be any one of lumen, wall, and J turn.

The 'lesion field of view' item refers to the imaging direction of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'lesion field of view' item may be any one of front, oblique, and lateral.

The 'Location of imaged lesion' item refers to the organ where the lesion included in the endoscopic image is located. In one embodiment, the attribute values corresponding to the 'location of imaged lesion' item are esophagus (E1/2), stomach (S1/2/3/4/5), duodenum (D1/2/3), Terminal ileum (TI), cecum (C), ascending colon (AC), hepatic flexure (HF), transverse colon (TC), splenic flexure (SF), descending colon (DC), sigmoid colon (SC), and rectum ( It may be any one of R).

The 'Shape of the imaged lesion' item refers to the shape of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'shape of photographed lesion' item is one of flat, sessile, pedunculated, oval, and irregular. It can be.

The 'size of imaged lesion' item refers to the size of the lesion included in the endoscopic image. In one embodiment, the attribute value corresponding to the 'size of imaged lesion' item may be any one of aphthous, large, and very large.

The 'Type of endoscopic imaging device, type of endoscopic procedure tool' item refers to the type of equipment used for endoscopic imaging. In one embodiment, the attribute value corresponding to the item 'Type of endoscopic imaging device, type of endoscopic surgical tool' may be one of forceps and snare.

The 'endoscopic imaging situation' item refers to under what circumstances the endoscopic imaging was performed. In one embodiment, the attribute value corresponding to the 'endoscopic imaging situation' item may be any one of biopsy, dye injection, and narrow-band imaging.

Meanwhile, without being limited to the above-described embodiment, the attribute value displayed on the terminal 20 includes at least one of the presence or absence of circular plastic (part of the endoscope device) on the outer edge of the image and the presence of boxes and letters in the endoscope image. It may contain attribute values corresponding to .

Next, a step is performed in which the terminal 20 receives at least one attribute value related to the captured endoscope image among the plurality of attribute values displayed (S210).

The user refers to the captured endoscopic image and directly enters the attribute value corresponding to the captured endoscopic image, or selects at least one of the plurality of attribute values displayed on the terminal 20 to determine the attribute corresponding to the captured endoscopic image. You can enter a value.

For example, referring to FIG. 7, the terminal 20 displays an actually captured endoscope image 310 and displays an input window in which a plurality of items and attribute values corresponding to each of the plurality of items are displayed in the form of a check box. It can be. Specifically, the terminal 20 displays the 'age of the subject of filming' item (age, 320), and the attribute values corresponding to age are 'less than 10 years old' (321), '10 to 20 years old' (322), ' '20 to 60 years old' (323) and 'over 60 years old' (324) may each be displayed in the form of a check box.

Furthermore, referring to FIG. 7 , the terminal 20 may display an input window 330 where lesion location information and mask information can be input.

For example, the user selects the location of the lesion in the input window where the location information and mask information of the lesion displayed on the terminal 20 can be input, or selects an area that is not easily visible due to dark, an area that is clearly visible, or an area that is not easily visible due to floaters. You can specify areas that are not allowed.

Next, the step of generating text based on the attribute values input by the user proceeds (S320).

When the user inputs at least one attribute value to be applied to endoscopic image synthesis, the sentence generation module 310 may generate text in the form of a completed sentence including the selected attribute value.

For example, referring to FIG. 8, a user may select 'lumen', 'C', 'flat', 'sessile', 'forcep', When 'NBI' is selected, the sentence generation module 310 can generate the completed sentence 'a lumen view picture taken at the colon C location' (341).

Meanwhile, the sentence generation module 310 may generate a plurality of different types of sentences based on attribute values input by the user. For example, the sentence generation module 310 may generate 'Pedunculated polyp on the lumen view using NBI with a cap from cancer patient aged 10 to 20.' in response to the same attribute value. and ‘Pedunculated polyp on the lumen view.’ and 'Large polyp using NBI with a cap from cancer patient' can be created.

In addition, referring to FIG. 8, the terminal 20 can generate lesion location information 342 and mask information 343 by utilizing information input through the lesion location information and mask information input window 330. there is.

Finally, a step of learning an artificial intelligence model is performed using at least one of the captured endoscopic image, attribute values input by the user, text generated based on the attribute values, lesion location information, and mask information. (S240).

Here, learning data for learning the artificial intelligence model may be generated by matching at least one of at least one attribute value, lesion location information, and mask information input through the user to the captured endoscopic image. That is, the learning data can be generated by matching attribute values selected by the user, the location of the lesion the user wants to create, and information defining clearly visible and not clearly visible areas when capturing an endoscopic image with the imaged endoscope. .

Furthermore, the learning data may be generated by matching text generated based on at least one attribute value input through the user and at least one attribute value input through the input window to the captured endoscopic image. . That is, the learning data can be generated by matching not only the user-selected attribute value to the captured endoscope image, but also text generated based on the user-selected attribute value.

Here, there may be a plurality of texts matching the captured endoscope image. As described above, the sentence generation module 310 can generate different sentences from the same attribute value. Each of the different sentences may be matched to the captured endoscopic image.

Meanwhile, location information of lesions included in the captured endoscopic image may be matched to the learning data. In one embodiment, the lesion location information may include the lesion's center coordinates, X-direction thickness, and y-direction thickness values.

For example, referring to FIG. 9, training data includes a captured endoscopic image (Image), text including at least one attribute value, lesion location information (bounding box), and mask information ( may include at least one of a segmentation mask). Using the above learning data, the artificial intelligence model is trained to synthesize a virtual endoscope image by receiving text containing attribute values related to the endoscope image (Text2Image), or is trained to synthesize a virtual endoscope image by receiving an endoscope image. (Image2Image), or it can be learned (Multi-modal2Image) to synthesize a lesion image with an actually captured endoscopic image based on text containing attribute values related to the endoscopic image.

Meanwhile, mask information may be matched to the learning data. For example, the pixel value constituting the mask information may be any one of three types of pixel values, each of which includes a pixel value defining a dark area, a pixel value defining a bright area, and the presence of floaters. It may be a pixel value that defines an area.

Meanwhile, a lesion image may be matched to the learning data. The lesion image is an image obtained by cropping the image corresponding to the location information of the lesion designated by the user among the images input from the user. The above-mentioned cut video can be used as learning data for learning artificial intelligence models.

On the other hand, when the artificial intelligence model receives an actually captured endoscope image, it can be used for learning as is or a model that gradually adds or removes noise (diffusion model) is applied to generate a learning image with noise applied or removed, and then artificially It can be used for learning intelligence models.

Figure 10 is a conceptual diagram showing a latent space included in an artificial intelligence model, but is not limited thereto.

As described above, the image synthesis device and method according to the present disclosure enable the synthesis of a large number of images of various lesions even without actual endoscopic data. A large amount of image data generated in this way can be used as training data for an artificial intelligence model to predict lesions through endoscopic images. That is, according to the present disclosure, a large amount of learning data (virtual endoscope images) for learning an artificial intelligence model can be generated using only a small amount of actually captured endoscope images.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code and, when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be Read Only Memory (ROM), Random Access Memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which this disclosure pertains will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims (10)

In a virtual endoscopic image synthesis device,
an input unit configured to receive at least one of text including attribute values related to the endoscopic image, lesion location information, mask information, and an actually captured endoscopic image;
A processor configured to synthesize a virtual endoscopic image based on at least one of the input text, lesion location information, mask information, and an actually captured endoscopic image,
The processor,
Using at least one of the actually captured endoscopic image, at least one attribute value related to the captured endoscopic image, lesion location information, and mask information as learning data, text containing attribute values related to the endoscopic image, lesion location information , an image synthesis module including an artificial intelligence model learned to synthesize a virtual endoscopic image by receiving at least one of mask information and an actually captured endoscopic image,
The artificial intelligence model is,
Learned to synthesize a lesion image with an actually captured endoscopic image based on at least one of text containing attribute values related to the endoscopic image, location information of the lesion, and the mask information,
The mask information defines the brightness and darkness of the area according to the amount of light and the direction of light irradiation during endoscopic imaging, or defines the sharpness of the area according to the presence or absence of floating objects during endoscopic imaging,
A virtual endoscope image synthesis device, wherein the mask information has the same size as the synthesized virtual endoscope image. According to paragraph 1,
It further includes a display unit configured to display an input window that receives at least one of the actually captured endoscopic image, attribute values related to the captured endoscopic image, location information of the lesion, and the mask information,
The learning data is,
A virtual endoscopic image synthesis device, characterized in that it is generated by matching at least one attribute value input through the input window to the captured endoscopic image. According to paragraph 2,
The processor,
It further includes a sentence generation module that generates text containing the attribute values based on attribute values related to the endoscope image,
The learning data is,
A virtual endoscope image, characterized in that it is generated by matching at least one attribute value input through the input window and text generated based on at least one attribute value input through the input window to the captured endoscopic image. Synthesis device. According to paragraph 3,
The sentence generation module generates a plurality of texts based on attribute values related to the captured endoscopic image,
The learning data is,
A virtual endoscopic image synthesis device, characterized in that it is generated by matching the plurality of texts to the captured endoscopic image. According to paragraph 3,
Attribute values related to the captured endoscopic image are,
A virtual endoscopic image synthesis device comprising attribute values related to global stand SES-CD score evaluation items. According to clause 5,
Attribute values related to the captured endoscopic image are,
Type of organ imaged, age of image subject, name of diagnosis, field of view of endoscopic imaging device, field of view of lesion, location of imaged lesion, shape of imaged lesion, size of imaged lesion, type of endoscopic imaging device, endoscopic procedure tools A virtual endoscopic image synthesis device comprising attribute values related to at least one of the type and endoscopic imaging situation. delete In a method of synthesizing a virtual endoscope image performed by a processor,
Using the actually captured endoscopic image and at least one attribute value related to the captured endoscopic image as learning data, at least one of text containing attribute values related to the endoscopic image, lesion location information, mask information, and the actually captured endoscopic image A step of receiving one input and learning an artificial intelligence model to synthesize a virtual endoscope image;
Receiving at least one of text including attribute values related to the endoscopic image, lesion location information, mask information, and an actually captured endoscopic image; and
A step of the artificial intelligence model synthesizing a virtual endoscopic image based on at least one of the input text, lesion location information, mask information, and actually captured endoscopic image,
The artificial intelligence model is,
Learned to synthesize a lesion image with an actually captured endoscopic image based on at least one of text containing attribute values related to the endoscopic image, location information of the lesion, and the mask information,
The mask information defines the brightness and darkness of the area according to the amount of light and the direction of light irradiation during endoscopic imaging, or defines the sharpness of the area according to the presence or absence of floating objects during endoscopic imaging,
A virtual endoscope image synthesis method, wherein the mask information has the same size as the synthesized virtual endoscope image. According to clause 8,
The step of learning the artificial intelligence model is,
Further comprising the step of displaying an input window for inputting the actually captured endoscopic image and attribute values related to the captured endoscopic image,
The learning data is,
A virtual endoscopic image synthesis method, characterized in that it is generated by matching at least one of at least one attribute value, lesion location information, and mask information input through the input window to the captured endoscopic image. A program coupled to a computer and stored in a computer-readable recording medium to execute the image synthesis method of any one of claims 8 and 9.

Images