KR20240025489A - Method and appratus for determining vascular structure using artificial intelligence - Google Patents

Abstract

The present invention provides a vascular structure determination method performed by a processor of a vascular structure determination device, comprising the steps of acquiring a medical image of a blood vessel of an object, detecting a loss area of the blood vessel in the medical image, and inputting the loss image. Generating a restored image for a blood vessel by inputting the detected loss area into a blood vessel prediction model learned to output a restored image, and determining the shape of the blood vessel for the loss area based on the generated restored image. It is composed.

Description

Method and device for determining blood vessel structure using artificial intelligence {METHOD AND APPRATUS FOR DETERMINING VASCULAR STRUCTURE USING ARTIFICIAL INTELLIGENCE}

The present invention relates to a method and device for determining blood vessel structure using artificial intelligence.

Recently, as Coronary Artery Diseases (CAD), such as angina pectoris and myocardial infarction, have rapidly increased, Percutaneous Coronary Intervention (PCI), which has a lower risk than other surgeries and procedures and can be treated quickly, is being performed. .

Percutaneous coronary intervention (PCI) requires administering a contrast agent to the patient's blood vessels and then taking an X-ray of the treatment area to identify different vascular structures for each patient. However, side effects from the contrast agent may occur depending on the patient, and long-term exposure to radiation is harmful to both the patient and the procedure, so it is important to quickly and accurately recognize blood vessels in contrast images for coronary intervention.

However, in actual X-ray cardiovascular medical images, there is a lot of noise due to background clutter such as organs, bones, and interventional devices, and it is difficult to quickly identify the vascular structure because it continues to move according to the heartbeat. In addition, if part of the coronary artery is completely blocked or chronically occluded (chronic total occlusion, CTO), it is impossible to confirm the vascular structure in medical images.

Accordingly, the reality is that delicate procedures for blood vessels of 1.5 mm to 2 mm or less depend on several X-ray images and the experience of the surgeon.

The technology behind the invention has been written to facilitate easier understanding of the invention. It should not be understood as an admission that matters described in the technology underlying the invention exist as prior art.

Meanwhile, deep learning technology is being applied in various fields related to medical imaging. For example, a method of extracting the central blood vessel of a coronary artery from a two-dimensional medical image using a CNN model has been previously disclosed.

However, the conventional method not only requires 3D medical images for real-time registration, but even if only 2D medical images are used, it is limited in accurately and quickly detecting or predicting blood vessels with complex and thin structures other than the central blood vessel with only one model. There is.

Accordingly, a method for clearly determining the structure of the missing area of blood vessels in medical images is required.

As a result, the inventors of the present invention developed a method and server for restoring lost areas of blood vessels using a vascular prediction model based on a generative adversarial network (GAN) that can respond to various variables present in medical images. It was composed.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the problems described above, a method for determining blood vessel structure according to an embodiment of the present invention is provided. The method includes obtaining a medical image of a blood vessel of an object, detecting a loss area of the blood vessel in the medical image, and detecting a loss area in a blood vessel prediction model learned to input the loss image and output a restored image. It is configured to include generating a restored image for a blood vessel by inputting , and determining a blood vessel shape for the loss area based on the generated restored image.

According to a feature of the present invention, the blood vessel prediction model is learned based on a GAN (generative adversarial networks) structure, and may correspond to a generator configured to generate a restored image by inputting a loss image of a blood vessel. .

According to another feature of the present invention, the blood vessel prediction model may be configured so that the generator and a discriminator that verifies the restored image generated by the generator compete with each other to update weights to improve the accuracy of the restored image. You can.

According to another feature of the present invention, the generator or the verifier of the blood vessel prediction model may be configured to learn using a loss function based on Hessian matrix characteristics. Specifically, to obtain reconstructed medical images considering the characteristics of thin and long blood vessels, Hessian-based Vesselness-loss can be used in addition to Hinge loss, contextual loss, SSIM loss, and Perceptual loss.

According to another feature of the present invention, the generator is configured to generate first feature maps corresponding to local context data by performing a convolution operation on each of the loss area and at least two different-sized kernels. Second division layers configured to extract second feature maps corresponding to far context data by performing a convolution operation on each of the division layers and the loss region and at least two different-sized kernels, wherein the first division layer includes: and second split layers may be arranged in parallel.

According to another feature of the present invention, the generator further includes concatenate layers configured to extract a near feature map and a far feature map by connecting each of the first feature maps and the second feature maps into one. can do.

According to another feature of the present invention, the generator further includes concatenate layers configured to extract a near feature map and a far feature map by connecting each of the first feature maps and the second feature maps into one. can do.

According to another feature of the present invention, the generator may be configured to generate the restored image for the loss area by applying a residual value to the near feature map and the far feature map.

According to another feature of the present invention, the step of detecting the loss area includes dividing the blood vessels in the medical image by inputting the medical image into a blood vessel segmentation model that has been learned to predict blood vessels in the target area using the medical image as input. The method may further include detecting a lost region in which at least a portion of the region is lost in the divided blood vessel.

In order to solve the problems described above, a device for determining blood vessel structure according to another embodiment of the present invention is provided. The device includes a communication interface, a memory, and a processor operably connected to the communication interface and the memory, wherein the processor acquires a medical image of a blood vessel of an object, and identifies a loss area of the blood vessel in the medical image. Generate a restored image for a blood vessel by inputting the detected loss area into a blood vessel prediction model that is trained to detect and output a restored image by inputting the loss image, and determine the shape of the blood vessel for the loss area based on the generated restored image. It is structured to make decisions.

Specific details of other embodiments are included in the detailed description and drawings.

The present invention can accurately restore and provide lost areas of blood vessels in medical images. In particular, by using a GAN-based blood vessel prediction model, the present invention can generate restored images corresponding to various blood vessel structures without relying on learning data.

The present invention uses a GAN-based generation model to restore coronary artery areas that are cut off due to lesions or have reduced image quality due to noise, etc., thereby restoring the blood vessel center line to a level that can be used as a reference during the procedure.

By quickly generating and providing restored images, the present invention can efficiently perform various procedures or surgeries related to blood vessels using the restored images in an actual clinical environment or at a real-life surgical procedure site.

The effects according to the present invention are not limited to the details exemplified above, and further various effects are included within the present invention.

Figure 1 is a block diagram showing the configuration of a blood vessel structure determination system according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention.
Figure 3 is a block diagram showing the configuration of a blood vessel structure determination device according to an embodiment of the present invention.
Figure 4 is a schematic flowchart of a method for determining blood vessel structure according to an embodiment of the present invention.
5A and 5B are schematic diagrams for explaining a method for learning a blood vessel prediction model according to an embodiment of the present invention.
Figure 6 is a schematic diagram illustrating the structure of a generator included in a blood vessel prediction model according to an embodiment of the present invention.
7A and 7B are schematic diagrams for explaining a prediction method of a blood vessel prediction model according to an embodiment of the present invention.
Figure 8a is a schematic diagram for explaining a method for determining blood vessel structure according to an embodiment of the present invention based on medical images.
Figure 8b is a schematic diagram illustrating performance improvement by applying a loss function in the method for determining blood vessel structure according to an embodiment of the present invention.
9A and 9B are schematic diagrams for explaining evaluation results of a method for determining blood vessel structure according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, but the present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the existence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, Or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, the first component may be renamed as the second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed as the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

As used in this document, the expression “configured to” depends on the situation, for example, “suitable for,” “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or fully combined or combined with each other, and as can be fully understood by those skilled in the art, various technical interconnections and operations are possible, and each embodiment may be implemented independently of each other. It may be possible to conduct them together due to a related relationship.

For clarity of interpretation of this specification, terms used in this specification will be defined below.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings.

As used herein, the term “subject” may refer to any subject who wishes to receive information about the vascular structure through X-ray imaging. For example, the subject may be a patient suffering from a disease related to the coronary artery, and the diagnosis site or target site may be the heart. Meanwhile, the entities disclosed herein may include all mammals, including humans.

The term “medical image” used in this specification may be a medical image composed of a plurality of cuts (frames). Specifically, the medical image may be a cross-sectional medical image obtained by irradiating X-rays toward an object. However, in addition to this, the medical image may be a 3D volume image, a single still image, a video composed of multiple cuts, or multiple images with various cross-sectional views.

As used herein, the term “loss area” may be an area in a medical image where some or all of the blood vessels are damaged or missing. In medical images, loss areas may exist due to various factors such as image quality, image shooting environment, and whether the subject is diseased.

The term “blood vessel prediction model” used in this specification may be a model learned to output a restored image using a loss image as input. Specifically, the blood vessel prediction model is learned based on the GAN (generative adversarial networks) structure, and may correspond to a generator configured to generate a restored image by inputting a loss image of a blood vessel. Additionally, the blood vessel prediction model may be configured so that a generator and a discriminator that verifies the restored image generated by the generator compete with each other to update weights to improve the accuracy of the restored image.

The term “blood vessel segmentation model” used in this specification may be a model learned to predict blood vessels that are target areas by inputting medical images. Specifically, the “blood vessel segmentation model” may be a model learned to segment and display an area where blood vessels are predicted to exist in a medical image.

In various embodiments, the blood vessel segmentation model is a VGG net based on a CNN (Convolutional Neural Network), R, DenseNet, and an FCN (Fully Convolutional Network) with an encoder-decoder structure, SegNet, DeconvNet, DeepLAB V3+, U-net, etc. It may consist of at least one model or two or more ensemble models among DNN (deep neural network), SqueezeNet, Alexnet, ResNet18, MobileNet-v2, GoogLeNet, Resnet50, Resnet101, and Inception-v3.

Figure 1 is a block diagram showing the configuration of a blood vessel structure determination system according to an embodiment of the present invention.

Referring to FIG. 1, the vascular structure determination system 1000 may be a system configured to predict vascular structure using an artificial neural network model. For this purpose, the vascular structure determination system 1000 includes an imaging device 100 for imaging blood vessels of an object, a medical staff device 200 for checking medical images of blood vessels and performing a diagnosis of the object, and an area lost from a blood vessel. It may include a blood vessel structure determination device 300 that restores .

The imaging device 100 may be a device that can acquire medical images of blood vessels of an object, such as a person or animal. For example, the imaging device 100 may acquire a medical image of an object by irradiating an X-ray capable of projecting the object toward the object.

In various embodiments, the imaging device 100 may acquire gray scale or RGB two-dimensional images, a single still image, a video composed of multiple cuts, etc. as medical images for determining the structure of a lost blood vessel. there is. For example, the imaging device 100 uses various types of imaging devices such as X-ray, ultrasound, computerized tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and single photon emission computed tomography (SPET). Medical images can be obtained.

If there is a lost part in the vascular image 11 captured by the imaging device 100, the medical staff device 200 requests the vascular structure determination device 300 to restore the image and provides the restored image 12. It may be a device that can display. For example, the medical staff device 200 may include a smartphone, tablet PC (personal computer), laptop, and PC.

In various embodiments, the medical staff device 200 is configured to install or execute a web or mobile application or program provided by the vascular structure determination device 300, and the vascular structure to be described later. A series of blood vessel structure determination methods performed by the determination device 300 may be performed.

The blood vessel structure determination device 300 may be a device that can acquire a medical image of a blood vessel of an object and determine a restored image to be inserted into the loss area. For example, the vascular structure determination device 300 may include a general-purpose computer, laptop, and data server.

Medical images captured by the imaging device 100 may include lost parts due to various causes such as cardiac motion artifact, image noise, and coronary artery disease. Because individual blood vessels have different structures even if they are of the same species, there are limitations in predicting the structure of blood vessels based on previously stored medical images of blood vessels.

Accordingly, the vascular structure determination device 300 of the present invention uses a vascular prediction model of a GAN (generative adversarial networks) structure that automatically learns the characteristics of input data and generates new data to predict blood vessels with different structures for each patient. The shape can be predicted and determined. Here, the blood vessel prediction model may correspond to a generator configured to generate a restored image using a loss image of a blood vessel as an input. The restored image generated by the generator can be verified through the discriminator included in the blood vessel prediction model. Specifically, the blood vessel prediction model may be configured so that the generator and verifier compete with each other to update weights to improve the accuracy of the reconstructed image.

In addition, the blood vessel structure determination device 300 of the present invention can newly configure the structure of the model so that the generator of the blood vessel prediction model can accurately separate the blood vessel and the background and accurately determine the fine structure of the blood vessel. Specifically, the blood vessel structure determination device 300 may be configured so that the generator of the blood vessel prediction model includes a plurality of convolution operation layers having kernels of different sizes, and a more detailed description will be provided later.

In this way, a restored image can be generated through the blood vessel prediction model of the GAN structure, and in the present invention, by additionally applying a plurality of loss functions, the accuracy of the blood vessel prediction model can be improved so that it can predict even the fine structure of blood vessels. there is.

In various embodiments, the vascular structure determination device 300 corrects the weights of each generator and verifier of the vascular prediction model using any one of the loss functions of Hinge Loss, contextual loss, SSIM loss, Perceptual loss, and Hessian-based loss. can do.

In various embodiments, the blood vessel structure determination device 300 may segment only the area where blood vessels exist before using the blood vessel prediction model to increase the prediction efficiency of the blood vessel prediction model. Specifically, the vascular structure determination device 300 inputs the medical image captured by the imaging device 100 into a blood vessel segmentation model learned to predict the target blood vessel by using the medical image as input to segment the blood vessels in the medical image. can do. Through this, the blood vessel structure determination device 300 can primarily distinguish the blood vessel from the background in which various noises exist. The blood vessel structure determination device 300 may detect a loss area in which at least a portion of the area is lost in the divided blood vessel. For example, the blood vessel structure determination device 300 may detect an area where some of the blood vessels are connected or an area where all of the blood vessels are separated.

As described above, the vascular structure determination device 300 can restore images of blood vessels through a blood vessel prediction model and a blood vessel segmentation model, thereby preventing atherosclerotic disease, coronary artery disease, cerebrovascular disease, Artery-related diseases such as renal vascular disease or peripheral vascular disease (PVD), giant cell arteritis, Takayasu's arteritis, Kawasaki disease, polyangiitis nodosa, vasculitis such as aneurysm or arterial dissection, varicose veins, thrombophlebitis or veins It can be used for detection, examination, and procedures of vein-related diseases such as thrombosis. For another example, the vascular structure determination device 300 may be used to treat angioplasty, aneurysm, angina (stiffness of the chest), aortic stenosis, aortitis, arrhythmia, arteriosclerosis, arteritis, atherosclerosis, carotid artery disease, aorta Coarctation, coronary artery disease, Eisenmenger's Syndrome, embolism, hypertension, hypotension, idiopathic infantile arterial calcification, Kawasaki disease (mucocutaneous lymph node syndrome, mucocutaneous lymph node disease, infantile polyarteritis), metabolic syndrome, microscopic Vascular angina, periarteritis nodosa (polyarteritis, polyarteritis nodosa), peripheral vascular disease, critical limb ischemia, diabetic angiopathy, phlebitis, pulmonary valve stenosis (pulmonary stenosis), renal artery stenosis, renovascular hypertension, asymptomatic ischemia, Takayasu Takayasu's Arteritis, Tetralogy of Fallot, transposition of the great vessels, arterial stem, varicose ulcer, varicose veins, vasculitis, fetal left ventricular aneurysm, neonatal cerebral hemorrhage, twin-to-twin transfusion syndrome (Twin. Twin transfusion syndrome), placental blood flow disorder, subarachnoid hemorrhage, intracerebral hemorrhage, other non-traumatic intracranial hemorrhage, cerebral infarction, occlusion and stenosis of the precranial artery, or occlusion and stenosis of the cerebral artery. . As another example, the vascular structure determination device 300 can be used to detect, test, or perform procedures on angiogenesis, a physiological process involving the proliferation of new blood vessels.

So far, the blood vessel structure determination system 1000 according to an embodiment of the present invention has been described. According to the present invention, the vascular structure determination system 1000 allows the vascular structure determination device 300 to immediately provide a restored image of the deformed or lost part of the blood vessel to the medical staff device 200, thereby enabling It can help to efficiently perform various procedures or surgeries related to blood vessels using restored images.

Hereinafter, with reference to FIG. 2, the medical staff device 200 that utilizes the result of determining the vascular structure will be described.

Figure 2 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention.

Referring to FIG. 2 , the medical staff device 200 may include a memory interface 210, one or more processors 220, and a peripheral interface 230. The various components within the medical staff device 200 may be connected by one or more communication buses or signal lines.

The memory interface 210 is connected to the memory 250 and can transmit various data to the processor 220. Here, the memory 250 is a flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, and cloud. , It may include at least one type of storage medium among the blockchain database.

In various embodiments, the memory 250 may store a web/app application or program for displaying a restored image for a lost area in a medical image for a blood vessel. Additionally, the memory 250 may store identification information of the entity (e.g., age, gender, disease, presence or absence of a disease), a medical image of the entity, a damaged area within the medical image, etc.

In various embodiments, memory 250 includes operating system 251, communications module 252, graphical user interface module (GUI) 253, sensor processing module 254, telephony module 255, and application module 256. ) can be stored. Specifically, the operating system 251 may include instructions for processing basic system services and instructions for performing hardware tasks. The communication module 252 may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) 253 can process a graphical user interface. Sensor processing module 254 may process sensor-related functions (e.g., processing voice input received through one or more microphones 292). The phone module 255 can process phone-related functions. Application module 256 may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions. In addition, the identifier device 100 may store one or more software applications 256-1 and 256-2 (eg, blood vessel structure determination application) associated with one type of service in the memory 250.

In various embodiments, memory 250 may store a digital assistant client module 257 (hereinafter referred to as the DA client module), thereby storing various user data 258 and instructions for performing client-side functions of the digital assistant. (e.g. user-customized vocabulary data, preference data, other data such as the user's electronic address book, etc.) may be stored.

Meanwhile, the DA client module 257 uses the user's voice input, text input, touch input, and/or gesture input through various user interfaces (e.g., I/O subsystem 240) provided in the medical staff device 200. It can be obtained.

Additionally, the DA client module 257 can output data in audiovisual and tactile forms. For example, the DA client module 257 may output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphics, video, animation, and vibration. In addition, the DA client module 257 can communicate with a digital assistant server (not shown) using the communication subsystem 280.

In various embodiments, the DA client module 257 may collect additional information about the surrounding environment of the medical staff device 200 from various sensors, subsystems, and peripheral devices to construct a context associated with user input. . For example, the DA client module 257 may infer the user's intention by providing context information along with user input to the digital assistant server. Here, context information that may accompany the user input may include sensor information, for example, lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the context information may include the physical state of the medical staff device 200 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As another example, context information may include information related to the software status of the medical staff device 200 (e.g., processes running on the medical staff device 200, installed programs, past and current network activity, background services, error logs, resource usage). etc.) may be included.

In various embodiments, memory 250 may include added or deleted instructions. Furthermore, the medical staff device 200 may also include additional components other than those shown in FIG. 2 or may exclude some components.

The processor 220 can control the overall operation of the medical staff device 200 and runs an application or program stored in the memory 250 to create a user interface that can display a restored image for a damaged or distorted area in a medical image. You can execute various commands to implement.

The processor 220 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). In addition, the processor 220 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) that integrates various computing devices that perform machine learning, such as a neural processing unit (NPU). .

In various embodiments, the processor 220 acquires a medical image of a blood vessel of an object, detects a loss area of the blood vessel in the medical image, inputs the loss image, and detects it in a blood vessel prediction model learned to output a restored image. By inputting the lost area, a restored image of the blood vessel can be generated, and the shape of the blood vessel for the lost area can be determined based on the generated restored image.

The peripheral interface 230 can be connected to various sensors, subsystems, and peripheral devices to provide data so that the medical staff device 200 can perform various functions. Here, any function performed by the medical staff device 200 may be understood as being performed by the processor 220.

The peripheral interface 230 may receive data from the motion sensor 260, the light sensor (light sensor) 261, and the proximity sensor 262, through which the medical staff device 200 can determine orientation, light, and proximity. It can perform detection functions, etc. For another example, the peripheral interface 230 may receive data from other sensors 263 (positioning system-GPS receiver, temperature sensor, biometric sensor), through which the medical staff device 200 may use other sensors. Functions related to (263) can be performed.

In various embodiments, the medical staff device 200 may include a camera subsystem 270 connected to the peripheral interface 230 and an optical sensor 271 connected thereto, through which the medical staff device 200 can take pictures and video. You can perform various shooting functions such as clip recording.

In various embodiments, medical staff device 200 may include a communication subsystem 280 coupled to a peripheral interface 230 . The communication subsystem 280 consists of one or more wired/wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

In various embodiments, medical staff device 200 includes an audio subsystem 290 coupled to a peripheral interface 230, which includes one or more speakers 291 and one or more microphones 292. By including this, medical staff device 200 can perform voice-activated functions, such as voice recognition, voice duplication, digital recording, and telephone functions.

In various embodiments, medical staff device 200 may include an I/O subsystem 240 coupled with a peripheral interface 230 . For example, the I/O subsystem 240 may control the touch screen 243 included in the medical staff device 200 through the touch screen controller 241.

For example, the touch screen controller 241 uses any one of a plurality of touch sensing technologies such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. to detect the user's touch and movement or touch. and cessation of movement can be detected. As another example, the I/O subsystem 240 may control other input/control devices 244 included in the medical staff device 200 through other input controller(s) 242. As an example, other input controller(s) 242 may control one or more buttons, rocker switches, thumb-wheels, infrared ports, USB ports, and pointer devices such as a stylus.

So far, the medical staff device 200 according to an embodiment of the present invention has been described. According to the present invention, the medical staff can use the medical staff device 200 to check the restored image of the loss area in the medical image captured by the imaging device 100, which is necessary for diagnosis, treatment, or surgery of the subject. Information can be obtained.

Hereinafter, the blood vessel structure determination device 300, which determines the structure of a blood vessel lost in a medical image, will be described with reference to FIG. 3 .

Figure 3 is a block diagram showing the configuration of a blood vessel structure determination device according to an embodiment of the present invention.

Referring to FIG. 3, the vascular structure determination device 300 may include a communication interface 310, a memory 320, an I/O interface 330, and a processor 340, and each component may include one or more communication buses. Alternatively, they can communicate with each other through signal lines.

The communication interface 310 can be connected to the imaging device 100 and the medical staff device 200 through a wired/wireless communication network to exchange data. For example, the communication interface 310 may receive a medical image of a subject from the imaging device 100 or the medical staff device 200. For another example, the communication interface 310 may transmit a restored image of a blood vessel based on a loss area in a medical image to the medical staff device 200 and may provide a user interface screen for visually displaying the image. there is.

Meanwhile, the communication interface 310 that enables transmission and reception of such data includes a wired communication port 311 and a wireless circuit 312, where the wired communication port 311 is one or more wired interfaces, for example, Ethernet. , Universal Serial Bus (USB), Firewire, etc. Additionally, the wireless circuit 312 can transmit and receive data with an external device through RF signals or optical signals. Additionally, wireless communications may use at least one of a plurality of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

The memory 320 may store various data used in the vascular structure determination device 300. For example, the memory 320 inputs the imaging device 100, identification information of the medical staff device 200, a blood vessel prediction model learned to output a restored image by inputting a real image, and a medical image as input to determine the target area. The configuration and learning data of a blood vessel segmentation model learned to predict blood vessels can be saved.

In various embodiments, memory 320 may include volatile or non-volatile recording media capable of storing various data, instructions, and information. For example, the memory 320 may be a flash memory type, hard disk type, multimedia card micro type, card type memory (e.g. SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage storage. , cloud, or blockchain database may include at least one type of storage medium.

In various embodiments, memory 320 may store configuration of at least one of operating system 321, communication module 322, user interface module 323, and one or more applications 324.

Operating system 321 (e.g., embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) is a variety of software to control and manage common system tasks (e.g., memory management, storage device control, power management, etc.) It may contain components and drivers and may support communication between various hardware, firmware, and software components.

The communication module 323 may support communication with other devices through the communication interface 310. The communication module 320 may include various software components for processing data received by the wired communication port 311 or the wireless circuit 312 of the communication interface 310.

The user interface module 323 may receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through the I/O interface 330 and provide a user interface on the display.

Applications 324 may include programs or modules configured to be executed by one or more processors 340 . Here, an application for determining blood vessel structure may be implemented on a server farm.

The I/O interface 330 may connect at least one of input/output devices (not shown) of the vascular structure determination device 300, such as a display, keyboard, touch screen, and microphone, to the user interface module 323. The I/O interface 330 may receive user input (eg, voice input, keyboard input, touch input, etc.) together with the user interface module 323 and process commands according to the received input.

The processor 340 is connected to the communication interface 310, the memory 320, and the I/O interface 330 to control the overall operation of the vascular structure determination device 300, and the application or Through the program, you can learn a blood vessel segmentation model and a blood vessel prediction model, and when inputting a new medical image, you can execute various commands to segment the target area in the medical image.

The processor 340 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). Additionally, the processor 340 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) in which various computing devices are integrated. Alternatively, the processor 340 may include a module for calculating an artificial neural network model, such as a Neural Processing Unit (NPU).

Hereinafter, with reference to FIGS. 4 to 7B, a method by which the processor 340 of the vascular structure determination device 300 determines the vascular structure shown in a medical image will be described.

Figure 4 is a schematic flowchart of a method for determining blood vessel structure according to an embodiment of the present invention.

Referring to FIG. 4, the processor 340 may acquire a medical image of the blood vessels of an object (S110). Specifically, the medical image may be a cross-sectional medical image obtained by irradiating X-rays toward an object. For example, the medical image may be a cross-sectional medical image of the heart to detect coronary arteries. Even if a medical image captured through the imaging device 100 is captured by administering a contrast agent, blood vessels may not be clearly displayed due to various factors, or may be mixed with other equipment or objects other than blood vessels and may be viewed as a single object. Accordingly, the processor 340 can segment only the blood vessels in the medical image so that the structure of the blood vessel can be efficiently predicted.

After step S110, the processor 340 may detect a blood vessel loss area in the medical image (S120). Specifically, the processor 340 may segment blood vessels in the medical image by inputting the medical image into a blood vessel segmentation model that has been learned to predict blood vessels that are target areas using the medical image as input. Through this, the processor 340 can distinguish blood vessels from backgrounds containing various types of noise. The processor 340 may detect a loss area in which at least a portion of the area is lost in the divided blood vessel. For example, the processor 340 may detect an area where some of the blood vessels are connected or an area where all of the blood vessels are separated.

After step S120, the processor 340 may generate a restored image for a blood vessel by inputting the detected loss area into a blood vessel prediction model learned to output a restored image by inputting the lost image (S130), and the generated restored image. Based on this, the blood vessel shape for the loss area can be determined (S140). Specifically, the blood vessel prediction model may be a model based on the GAN (generative adversarial networks) structure that generates new data by learning the characteristics of the input data on its own, and is configured to generate a restored image by taking the loss image of the blood vessel as input. It may correspond to a generator.

In this specification, the blood vessel prediction model for generating a restored image can be learned as follows, and can have a new structure that can predict fine blood vessel structures through the blood vessel prediction model.

5A and 5B are schematic diagrams for explaining a method for learning a blood vessel prediction model according to an embodiment of the present invention.

Referring to FIG. 5A, the processor 340 selects only the image data (ground truth, GT) 502 for the coronary artery to be used for learning in the medical image 501 for the intact coronary artery (C ^vessel ) without any lost areas. It can be extracted. The processor 340 can randomly extract an area to be learned from image data (GT) for the coronary artery (Random Sampling) and generate image data 503 indicating the area. The processor 340 generates learning ^data 504 by enlarging only the area required for learning, and finally creates an X- _ray ^patch ( 510), that is, the area of interest can be determined. However, here, the processor 340 may determine the region of interest with a sufficiently large size so that both the local and global regions of the coronary artery can be learned at once.

Referring to FIG. 5B, the processor 340 applies a mask (M) 511 to the area to be learned in the X-Ray patch (X _i ^patch ) 510 to generate masked image data (X _i ^blocked ) ( 512) can be created. The processor 340 may input the masked image data 512 to a generator of a blood vessel prediction model to obtain image data (˜X _i ^blocked ) 520 in which the damaged area is restored. The processor 340 can verify the authenticity of the restored image data (˜X _i ^blocked ) 520 through a discriminator of the blood vessel prediction model. The processor 340 may generate a restored medical image 501' based on image data 520 verified through a verifier.

Meanwhile, in this process, in order to obtain more accurate restoration results even for thin blood vessels, the processor 340 creates a probability map 5101 corresponding to the X-Ray ^patch ₅₁₀ and the restored image data (~ The weights of the generator (G) and the verifier (D) can be adjusted using a loss function so that the difference between the probability maps 5201 corresponding to the X _i ^blocked ) 520 is minimized.

In various embodiments, the processor 340 may adjust the weights of the generator (G) and the verifier (D) using a plurality of loss functions. Specifically, the processor 340 may adjust the weights using at least one loss function among Hinge Loss, contextual loss, SSIM loss, Perceptual loss, and Hessian-based loss. For example, when using Hinge Loss, the weight can be adjusted through [Equation 1] and [Equation 2] below. By using Hinge Loss, the processor 340 can determine a weight that can clarify the boundary between blood vessels and the background.

As another example, when using contextual loss, the weights can be adjusted through [Equation 3] below. By using contextual loss, the processor 340 can determine a weight that prevents blurring and increases the clarity of the restored image.

As another example, when using SSIM loss, the weights can be adjusted through [Equation 4] below. The processor 340 can determine a weight that can increase the luminance and contrast of the restored image by using SSIM loss.

As another example, when using perceptual loss, the weights can be adjusted through [Equation 5] below. Here, ϕ _i ^vgg (·) represents the feature map for the ith layer of the relu5_1 layer of VGG19 in the VGGNet-based blood vessel prediction model, and C _i , H _i and W _i represent the channel, height and width, respectively. The processor 340 can determine a weight that can increase the similarity between the damaged area and the restored image by using perceptual loss.

In this way, the processor 340 can learn the generator and verifier of the blood vessel prediction model using a loss function. In particular, in the present invention, learning optimized for thin and long blood vessel characteristics using Vesselness-loss based on Hessian matrix characteristics. The process can be preceded. Specifically, when using Hessian-based loss, first obtain the value p = (x, y), which represents the probability that each pixel of the medical image corresponds to a blood vessel, using [Equation 6] below, and then use [Equation 7] Weights can be adjusted through Here, Vesselness probability (V(·)) represents the probability that a specific pixel corresponds to a vessel, and is the ratio between the sizes of the two eigenvectors (Eigenvevtor) and the eigenvalues (λ1) (λ2), which can be obtained through the Hessian matrix ( It can be obtained through R). Also, R _b ² = |λσ)| / |λσ)| and S = root(λ ₁ ² + λ ₂ ² ). The processor 340 can determine a weight for dividing the blood vessel area from the background area by using Hessian-based loss.

In various embodiments, the processor 340 may adjust the weights of the generator and verifier of the blood vessel prediction model by applying one or more of the above loss functions. For example, the processor 340 may apply a loss function using a weighting constant as shown in [Equation 8] and [Equation 9] below.

Figure 6 is a schematic diagram illustrating the structure of a generator included in a blood vessel prediction model according to an embodiment of the present invention.

Referring to FIG. 6, the generator of the blood vessel prediction model has a multi-scale aggregation block (MAB) structure and considers both the near and far areas around the loss area to accurately generate a restored image for the blood vessel loss area. It can have a two-pronged structure. In the two branches, the generator of the blood vessel prediction model may perform a convolution operation on the masked image data (X _i ^blocked ) 512 using kernels of different sizes arranged in parallel. Specifically, the generator of the blood vessel prediction model performs a convolution operation on each of the loss area (masked image data (X _i ^blocked ) 512) and at least two kernels of different sizes to generate near-field context data Perform a convolution operation on each of the first division layers 601 and 603 configured to generate the first feature map, the loss area (masked image data (X _i ^blocked )), and at least two kernels of different sizes. It may include second segmentation layers 605 and 607 configured to generate a second feature map corresponding to the remote context data. The first and second division layers are arranged in parallel, a first feature map for distinguishing blood vessels from backgrounds other than blood vessels through the first division layers, and a second feature map for distinguishing fine blood vessels through the second division layers. can be obtained. The generator may include concatenate layers 611 and 613 configured to extract a near feature map and a far feature map by connecting each of the first feature maps and the second feature maps into one, and each feature An additional connection layer 621 may be included to integrate the map. In addition, the generator performs padding (e.g. reflect padding) to maintain the size of the data resulting from integrating the near and far feature maps, and applies residual values to improve the precision of the restored image. It may further include two convolution operation layers 631 and 633. The processor 340 can generate a restored image for the loss area through this new MAB structure.

The processor 340 may determine a restored image for the damaged area using the blood vessel prediction model learned as described above.

7A and 7B are schematic diagrams for explaining a prediction method of a blood vessel prediction model according to an embodiment of the present invention.

Referring to FIG. 7A , the processor 340 may extract a damaged area from a medical image 701 of a coronary artery (C ^broken ∪C ^vessel ) including the damaged area. In other words, the processor 340 may generate image data 702 indicating the damaged area using the blood vessel segmentation model mentioned in step S120. For example, a blood vessel segmentation model can detect an area where a blood vessel is expected to exist in pixels of a medical image and determine the loss area by detecting the end of the blood vessel. ^The processor 340 generates input data 703 by enlarging only the area to be input to the blood vessel prediction model, _and finally determines an You can. Here, too, the processor 340 can determine the region of interest with a sufficiently large size so that both the local and global regions of the coronary artery can be used to determine the restored image.

Referring to FIG. 7B, the processor 340 applies a mask (M) 711 to the area to be restored in the X-Ray patch (X _i ^patch ) ₇₁₀ ^to generate masked image data ( 712) can be generated. The processor 340 may input the masked image data 512 to the generator (G) of the blood vessel prediction model to obtain image data (˜X _i ^blocked ) 720 in which the damaged area is restored. In this way, the probability map 7201 of the image data (~X _i ^blocked ) 720 obtained through ^the blood vessel prediction model and _the probability map 7101 corresponding to the Without this, they may be identical, and the processor 340 may ultimately generate a restored image 701' most suitable for the corresponding medical image 701.

So far, a medical image segmentation method performed by the processor 340 of the vascular structure determination device 300 according to an embodiment of the present invention has been described. According to the present invention, the vascular structure determination device 300 can generate reconstructed images corresponding to various vascular structures without relying on learning data by using a GAN-based blood vessel prediction model.

Below, with reference to FIGS. 8A and 8B, a series of medical images to which the method for determining blood vessel structure according to an embodiment of the present invention is applied will be described.

Figure 8a is a schematic diagram for explaining a method for determining blood vessel structure according to an embodiment of the present invention based on medical images.

Referring to FIG. 8A, ^the method for determining the vascular structure of the present invention is that in ⁽ a) an Based on this, the area of interest can be detected. In addition, (c) determine the Thus, the image of the restored blood vessel can be determined.

Figure 8b is a schematic diagram illustrating performance improvement by applying a loss function in the method for determining blood vessel structure according to an embodiment of the present invention.

Referring to Figure 8b, when there is a medical image (a) with a determined learning area and a medical image (b) with the corresponding area completely deleted, (c) is generated through a blood vessel prediction method without correcting the weights of the generator and verifier. Indicates the restored image. In addition, (d) shows the restored image generated through the blood vessel prediction method when the weights of the generator and verifier were corrected by applying SSIM loss, and (e) shows the restored image generated by applying Hessian-based loss to correct the weights of the generator and verifier. represents the restored image generated through the blood vessel prediction method, and (f) represents the restored image created through the blood vessel prediction method when the weights of the generator and verifier are corrected with both loss functions. Specifically, it can be seen that in (d), where SSIM loss is applied, the luminance is adjusted and noise is reduced, resulting in visually improved results compared to (c), where no weight correction exists, and where Hessian-based loss is applied ( In e), the sharpness has been improved, so you can see that the edges of the blood vessels are clear without breaking. Accordingly, it can be confirmed that (f), which simultaneously applies SSIM loss and Hessian-based loss, determines a clearer and more accurate vascular structure.

Hereinafter, the evaluation results of the vascular structure determination device 300 according to an embodiment of the present invention will be described with reference to FIGS. 9A and 9B.

9A and 9B are schematic diagrams for explaining evaluation results of a method for determining blood vessel structure according to an embodiment of the present invention.

Referring to FIG. 9A, for a medical image (a) of a blood vessel area in which no damage area exists and a medical image (b) in which the degree of damage is adjusted step by step, (c) to (f) are images using conventional methods, respectively. shows a restored image, and (g) shows a restored image determined using the vascular structure determination method of the present invention. Specifically, it was confirmed that some conventional methods appear to correctly determine the structure of blood vessels for medical images with large blood vessels or structures in the damaged area that are not complex, but do not produce the same restored image as the original medical image for small blood vessels. You can. In addition, it can be confirmed that, unlike the present invention, conventional methods do not produce the correct restored image when an image of the damaged area is not given.

Referring to FIG. 9B, for a medical image (a) of a blood vessel area in which no damaged area exists and a medical image (b) in which a part of the blood vessel area is completely deleted, (c) to (f) show conventional methods, respectively. Shows the restored image used, and (g) shows the restored image determined using the vascular structure determination method of the present invention. Specifically, it can be seen that, unlike the present invention, which produces a restored image identical to the original medical image, conventional methods do not produce a correct restored image even for thick blood vessels.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be modified and implemented in various ways without departing from the technical spirit of the present invention. there is. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

1000: Vascular structure determination system
100: video recording device
200: Medical staff device
210: memory interface 220: processor
230: peripheral interface 240: I/O subsystem
241: Touch screen controller 242: Other input controller
243: touch screen
244: Other input control devices
250: Memory 251: Operating System
252: Communication module 253: GUI module
254: sensor processing module 255: phone module
256: Applications
256-1, 256-2: Application
257: Digital assistant client module
258: User data
260: motion sensor 261: light sensor
262: Proximity sensor 263: Other sensors
270: Camera subsystem 271: Optical sensor
280: Communication subsystem
290: Audio subsystem
291: Speaker 292: Microphone
300: Vascular structure determination device
310: communication interface
311: wired communication port 312: wireless circuit
320: memory
321: operating system 322: communication module
323: user interface module 324: application
330: I/O interface 340: Processor

Claims (16)

1. A method of determining blood vessel structure performed by a processor, comprising:
Obtaining a medical image of a blood vessel of an object;
detecting a loss area of the blood vessel in the medical image;
Generating a restored image for a blood vessel by inputting the detected loss area into a blood vessel prediction model learned to output a restored image by inputting the lost image;
determining a blood vessel shape for the loss area based on the generated restored image; A method for determining vascular structure comprising a. According to paragraph 1,
The blood vessel prediction model is,
It is learned based on the GAN (generative adversarial networks) structure,
A method for determining blood vessel structure, corresponding to a generator configured to generate a restored image by taking a loss image of the blood vessel as an input. According to paragraph 2,
The blood vessel prediction model is,
A method for determining blood vessel structure, wherein the generator and a discriminator that verifies the reconstructed image generated by the generator compete with each other to update weights to improve the accuracy of the reconstructed image. According to paragraph 3,
Before acquiring the medical image,
Method for determining blood vessel structure, further comprising: learning the generator or the verifier of the blood vessel prediction model using a loss function based on Hessian matrix characteristics. According to paragraph 2,
The generator is,
first division layers configured to generate first feature maps corresponding to short-range context data by performing a convolution operation on the loss region and each of at least two different-sized kernels; and
second segmentation layers configured to extract second feature maps corresponding to distant context data by performing a convolution operation on the loss region and each of at least two different-sized kernels; Includes,
The first and second split layers are arranged in parallel. According to clause 5,
The generator is,
Concatenate layers configured to extract a near feature map and a far feature map by connecting each of the first feature maps and the second feature maps into one; Including, a method for determining blood vessel structure. According to clause 6,
The generator is,
Configured to generate the restored image for the loss area by applying a residual value to the near feature map and the far feature map. According to paragraph 1,
The step of detecting the loss area is,
Segmenting blood vessels in the medical image by inputting the medical image into a blood vessel segmentation model learned to predict blood vessels of a target region using the medical image as input; and
Detecting a loss area in which at least some area is lost in the divided blood vessel; A method for determining blood vessel structure, further comprising: communication interface;
Memory; and
a processor operably connected to the communication interface and the memory; Including,
The processor,
Obtain a medical image of a blood vessel of an object, detect a loss area of the blood vessel in the medical image, input the loss image, and input the detected loss area into a trained blood vessel prediction model to output a restored image. An apparatus for determining blood vessel structure, configured to generate a restored image and determine a blood vessel shape for the loss area based on the generated restored image. According to clause 9,
The blood vessel prediction model is,
It is learned based on the GAN (generative adversarial networks) structure,
A blood vessel structure determination device corresponding to a generator configured to generate a restored image by taking a loss image of the blood vessel as an input. According to clause 10,
The blood vessel prediction model is,
An apparatus for determining blood vessel structure, wherein the generator and a discriminator that verifies the reconstructed image generated by the generator compete with each other to update weights to improve the accuracy of the reconstructed image. According to clause 11,
The processor,
A blood vessel structure determination device configured to learn the generator or the verifier of the blood vessel prediction model using a loss function based on Hessian matrix characteristics. According to clause 10,
The generator is,
first division layers configured to generate first feature maps corresponding to short-range context data by performing a convolution operation on the loss region and each of at least two different-sized kernels; and
second segmentation layers configured to extract second feature maps corresponding to distant context data by performing a convolution operation on the loss region and each of at least two different-sized kernels; Includes,
The first and second split layers are arranged in parallel. According to clause 13,
The generator is,
Concatenate layers configured to extract a near feature map and a far feature map by connecting each of the first feature maps and the second feature maps into one; Including, a vascular structure determination device. According to clause 14,
The generator is,
Configured to generate the restored image for the loss area by applying a residual value to the near feature map and the far feature map. According to clause 9,
The processor,
The medical image is input to a blood vessel segmentation model learned to predict the target blood vessel using the medical image as input, the blood vessels in the medical image are segmented, and a loss area in which at least some area is lost in the segmented blood vessel is further detected. Consisting of a vascular structure determination device.