WO2021193018A1

WO2021193018A1 - Program, information processing method, information processing device, and model generation method

Info

Publication number: WO2021193018A1
Application number: PCT/JP2021/009296
Authority: WO
Inventors: 悠介関; 雄紀坂口; 陽井口
Original assignee: テルモ株式会社
Priority date: 2020-03-27
Filing date: 2021-03-09
Publication date: 2021-09-30
Also published as: JPWO2021193018A1

Abstract

This program causes a computer to execute a process of acquiring patient information about a patient who is to undergo treatment and a medical image obtained by imaging a hollow organ of the patient, and inputting the acquired patient information and the acquired medical image to a model learned so as to output treatment information for providing assistance in the treatment to be performed on the patient when the patient information and the medical image are inputted thereto, to output the treatment information.

Description

Program, information processing method, information processing device and model generation method

The present invention relates to a program, an information processing method, an information processing device, and a model generation method.

Various methods have been proposed to support treatment based on medical images that visualize the inside of the human body, such as ultrasonic images, optical coherence tomography (OCT) images, and X-ray images. For example, in Patent Document 1, a designation of a range of a diseased part in an intravascular image is received from an operator, a shape of a therapeutic device suitable for the designated range of the diseased part is set, and a therapeutic device having the set shape is provided. A therapeutic device selection support system that searches and displays information from a database is disclosed.

JP-A-2007-75141

However, it is not easy to read information such as the diseased part from medical images and reflect it in treatment, and abundant knowledge and clinical experience are required. The invention according to Patent Document 1 only searches a database for a therapeutic device suitable for a disease site in a range specified by an operator, and does not suitably support treatment based on a medical image.

One aspect is to provide a program or the like that can suitably support treatment based on medical images.

The program according to one aspect is performed on the patient when the patient information about the patient to be treated and the medical image of the patient's luminal organ are acquired and the patient information and the medical image are input. A computer is made to execute a process of inputting the acquired patient information and a medical image and outputting the treatment information to a model trained to output the treatment information to support the treatment.

On one side, medical image-based treatment can be suitably supported.

It is explanatory drawing which shows the configuration example of the treatment support system. It is a block diagram which shows the configuration example of a server. It is explanatory drawing which shows an example of the record layout of the medical care DB. It is explanatory drawing which shows the outline of the learning model. It is explanatory drawing which shows the detail of a learning model. It is explanatory drawing which shows the display screen example of the display device. It is a flowchart which shows the procedure of the generation process of a learning model. It is a flowchart which shows the procedure of the output processing of the treatment information. It is a flowchart which shows the procedure of output processing of the treatment information which concerns on modification 1. It is explanatory drawing about the learning model which concerns on modification 2. It is explanatory drawing of the learning model which concerns on Embodiment 2. It is explanatory drawing which shows the display example of the 2nd medical image which concerns on Embodiment 2. It is explanatory drawing which shows the display example of the 2nd medical image which concerns on Embodiment 2. It is explanatory drawing which shows the display example of the 2nd medical image which concerns on Embodiment 2. It is explanatory drawing which shows the display example of the 2nd medical image which concerns on Embodiment 2. It is a flowchart which shows the procedure of the generation process of the learning model which concerns on Embodiment 2. It is a flowchart which shows the procedure of the output processing of the treatment information which concerns on Embodiment 2. It is a flowchart which shows the procedure of output processing of the treatment information which concerns on modification 3.

Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments thereof.
(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of a treatment support system. In the present embodiment, treatment information for supporting the treatment of the patient is presented to the user (medical worker) based on the patient information about the patient performing the intravascular treatment and the medical image of the patient's blood vessel. The treatment support system to be used will be described. The treatment support system includes an information processing device 1 and a diagnostic imaging system 2. The information processing device 1 and the diagnostic imaging system 2 are communication-connected to a network N such as a LAN (Local Area Network) or the Internet.

In the present embodiment, endovascular treatment will be described as an example, but the target luminal organ is not limited to blood vessels, and may be other luminal organs such as bile duct, pancreatic duct, bronchus, and intestine. ..

The diagnostic imaging system 2 includes an intravascular diagnostic imaging device 21, a fluoroscopic imaging device 22, and a display device 23. The intravascular image diagnostic device 21 is a device for imaging an intravascular tomographic image of a patient, for example, an IVUS (Intravascular Ultrasound) device that performs an ultrasonic examination using a catheter 211. The catheter 211 is a medical device inserted into a blood vessel of a patient, and includes a piezoelectric element that transmits ultrasonic waves and receives reflected waves from the blood vessels. The intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image based on the signal of the reflected wave received by the catheter 211 and displays it on the display device 23.

In the present embodiment, the intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image, but for example, an optical coherence tomographic image (OCT image) may be generated.

The fluoroscopic image capturing device 22 is a device unit for capturing a fluoroscopic image that sees through the inside of a patient, and is, for example, an angiography device that performs angiography examination. The fluoroscopic image capturing apparatus 22 includes an X-ray source 221 and an X-ray sensor 222, and the X-ray sensor 222 receives the X-rays emitted from the X-ray source 221 to image the X-ray fluoroscopic image of the patient. For example, an X-ray opaque marker is attached to the tip of the catheter 211, and the position of the catheter 211 is visualized in a fluoroscopic image. The perspective imaging device 22 displays a perspective image that visualizes the position of the catheter 211 on the display device 23 and presents it together with an intravascular tomographic image.

In the above, ultrasonic tomography, optical interference tomography, and angiography are given as examples of medical images, but the medical images are computed tomography (CT) images and magnetic resonance imaging (MRI). It may be an image or the like.

The information processing device 1 is an information processing device capable of transmitting and receiving various types of information processing and information, such as a server computer and a personal computer. In the present embodiment, it is assumed that the information processing device 1 is a server computer, and in the following, it will be read as server 1 for the sake of brevity. The server 1 may be a local server installed in the same facility (hospital or the like) as the diagnostic imaging system 2, or may be a cloud server communicated and connected via the Internet or the like. The server 1 is a treatment support device that outputs treatment information for supporting endovascular treatment of the patient based on the patient information about the patient to be treated and the medical image (tomographic image and fluoroscopic image) of the patient. Function. Specifically, as will be described later, the server 1 prepares in advance a learning model 50 (see FIG. 4) that performs machine learning to learn predetermined training data, inputs patient information and medical images, and outputs treatment information. It is done. The server 1 inputs the patient information and the medical image of the patient to be treated into the learning model 50, and acquires the treatment information from the learning model 50. The server 1 outputs the treatment information to the diagnostic imaging system 2 and displays it on the display device 23.

The treatment information is information for supporting the treatment of the patient, and is information for assisting the user (medical worker) who treats the patient. In the present embodiment, endovascular treatment is taken as an example, and information for supporting PCI (Percutaneous Coronary Intervention) using the catheter 211 is output as treatment information. For example, treatment information includes the position and properties of lesions such as plaques, therapeutic devices such as stents and balloons to be used, the dose and duration of contrast media recommended for fluoroscopic imaging, and post-treatment progress information (eg). Risk of complications) etc. The server 1 estimates these information using the learning model 50 and displays it on the display device 23. Further, the server 1 detects an object such as a lesion or a therapeutic device from the medical image, generates a second medical image processed so that the detected object can be identified, and displays the detected object on the display device 23.

In the present embodiment, the server 1 performs the treatment information generation process using the learning model 50. However, the learning model 50 constructed by the server 1 is installed in the diagnostic imaging system 2 and the treatment information is locally generated. The generation process may be executed.

FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.
The control unit 11 has one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), AI chips (AI semiconductors) and other arithmetic processing devices, and stores the program P stored in the auxiliary storage unit 14. Various information processing is performed by reading and executing. The main storage unit 12 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing related to communication, and transmits / receives information to / from the outside.

The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores a program P and other data necessary for the control unit 11 to execute processing. In addition, the auxiliary storage unit 14 stores the learning model 50 and the medical care DB 141. The learning model 50 is a machine learning model in which training data has been trained as described above, and is a model that outputs treatment information for supporting treatment of a patient by inputting patient information and a medical image. The learning model 50 is expected to be used as a program module constituting artificial intelligence software. The medical care DB 141 is a database that stores medical care data of patients.

The auxiliary storage unit 14 may be an external storage device connected to the server 1. Further, the server 1 may be a multi-computer composed of a plurality of computers, or may be a virtual machine virtually constructed by software.

Further, in the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays an image, and the like. Further, the server 1 includes a reading unit for reading a portable storage medium 1a such as a CD (CompactDisk), a DVD (DigitalVersatileDisc), a USB (UniversalSerialBus) memory, and an external hard disk, and is provided from the portable storage medium 1a. The program P may be read and executed. Alternatively, the server 1 may read the program P from the semiconductor memory 1b.

FIG. 3 is an explanatory diagram showing an example of the record layout of the medical care DB 141. The medical treatment DB 141 includes a medical treatment ID column, a patient information column, a treatment information column, and an image information string. The medical treatment ID column stores a medical treatment ID for identifying each medical treatment data. The patient information string, the treatment information column, and the image information string are associated with the medical treatment ID, respectively, and the patient information about the patient to be treated, the treatment information about the treatment performed for the patient, and the medical use obtained by examining the patient. I remember the image.

Patient information is information about the patient who has undergone treatment and is a medical record of the patient. Treatment information includes, for example, patient age, gender, diagnosis name, risk factors (presence or absence of lifestyle disease, etc.), history, treatment history, concomitant medications, blood test results, number of affected branches of blood vessels, left chamber, etc. Includes ejection rate, history of sudden changes related to cardiovascular disease (myocardial infarction, etc.).

The treatment information is information on endovascular treatment performed on the patient, for example, a record of PCI. Treatment information includes, for example, the date of PCI, the location of the treated lesion (hereinafter referred to as "lesion site"), the properties of the lesion, the puncture site of the catheter 211, the total amount of contrast agent used, and a fluoroscopic image. Imaging time, presence / absence and content of additional treatment before stent placement, presence / absence and content of additional treatment after stent placement, type of stent placed (for example, product name), diameter, length, total number, type of balloon, length , Maximum dilation diameter, maximum dilation pressure, dilation time, decay time, treatment of bifurcation lesions, post-treatment progress information (eg, presence or absence of complications). In the present embodiment, the diameters of the stent and the balloon are expressed by the diameter, but it is needless to say that the diameter may be the radius.

The medical image is an image of the patient's blood vessel, and as described above, a tomographic image inside the blood vessel (ultrasonic tomographic image, optical interference tomographic image, etc.) and a fluoroscopic image (angiography image) in which the inside of the patient is visualized by X-ray. , Computed tomography images, etc.), magnetic resonance imaging, etc. The image information also includes physiological functional evaluation results such as FFR (Fractional Flow Reserve). In the present embodiment, as an example, an ultrasonic tomographic image and an angiography image are used as medical images for inputting the learning model 50.

FIG. 4 is an explanatory diagram showing an outline of the learning model 50. The learning model 50 is a machine learning model that inputs patient information and medical images and outputs treatment information for supporting treatment of the patient. The server 1 performs machine learning to learn predetermined training data and generates a learning model 50 in advance. Then, the server 1 inputs the medical image of the patient acquired from the diagnostic imaging system 2 and the patient information about the patient into the learning model 50, and generates the treatment information.

Specifically, as shown in FIG. 4, the learning model 50 includes patient information stored in the medical care DB 141, a tomographic image imaged by the intravascular image diagnostic device 21, and a fluoroscopic image imaged by the fluoroscopic image capturing device 22. Is input. Then, the learning model 50 outputs the estimation results estimated for each item shown in the upper right of FIG. 4 as treatment information.

For example, treatment information includes information on lesions in blood vessels. Specifically, the information about the lesion includes the lesion site (for example, the type of blood vessel in which the lesion exists) and the properties of the lesion. The information about the lesion may include not only the position and properties but also the length of the lesion in the longitudinal direction of the blood vessel, the size (area) of the lesion in the cross section of the blood vessel, and the like.

The treatment information also includes information about the treatment device to be inserted into the blood vessel. Therapeutic devices are, for example, stents, balloons, etc. for dilating blood vessels. The therapeutic device is not limited to a stent or the like, and may include, for example, a rotablator for excision of a lesion. The learning model 50 includes the type of stent to be used (for example, product name), shape (diameter and length), total number of stents, presence / absence and content of additional procedures before stent placement, and presence / absence and contents of additional procedures after stent placement. Outputs stent information related to the stent, such as. In addition, the learning model 50 includes the type of balloon to be used (for example, product name), shape (length), expansion conditions (maximum expansion pressure, maximum expansion diameter, expansion time, damping time required for balloon contraction after expansion), and the like. , Outputs balloon information related to the balloon.

Note that the order of use of each therapeutic device may be included as information regarding the therapeutic device. Thereby, when the user selects each therapeutic device and performs the treatment, the selection of the therapeutic device can be suitably assisted.

In addition, the treatment information includes information regarding the imaging conditions of the fluoroscopic image. The information regarding the imaging conditions includes, for example, the dose of the contrast medium, the imaging time of the fluoroscopic image (“transparent time” in FIG. 4), and the like. The server 1 causes the learning model 50 to learn the dose of the contrast medium, the imaging time, etc. when the patient is treated, and can output the recommended dose of the contrast medium, the imaging time, and the like.

In addition, the treatment information includes progress information that estimates the progress of the patient after treatment. The progress information is, for example, an estimation result of estimating the risk of complications. The server 1 makes the learning model 50 learn the complications that have developed after the treatment for the patient who has been treated, and can output the complications that have a high probability of developing after the treatment as progress information. As will be described later, it is preferable to output a probability value for evaluating the degree of occurrence of each complication for each of a plurality of complications that may occur after treatment (see FIG. 6).

In FIG. 4, as an example, a tomographic image when a plaque is formed in the blood vessel of the patient and a fluoroscopic image showing the position of the catheter 211 at the time of generating the tomographic image are input to the learning model 50 together with the patient information. The case is illustrated. In this case, the learning model 50 outputs the position and properties of the lesion corresponding to the plaque, the stent to be used and its expansion diameter, the expansion pressure, the progress information after treatment, and the like.

The learning model 50 outputs the above-mentioned various information as treatment information, and also generates a second medical image processed so that a predetermined object in the image can be identified from the input medical image, and is one of the treatment information. Output as. The second medical image is an image obtained by processing an image area corresponding to an object such as a lesion as described above, and is an image displaying the image area of the object in a display mode different from that of other image areas. For example, the learning model 50 generates a second medical image in which the image region corresponding to the object is represented by a color other than black and white in the tomographic image represented by black and white. The server 1 outputs the second medical image together with the information such as the lesion site to the display device 23 and displays it.

The target object in the learning model 50 is not limited to the lesion, and may be, for example, a therapeutic device inserted into the blood vessel of the patient (for example, a stent already placed in the blood vessel of the patient). The learning model 50 may be capable of generating a second medical image that makes it possible to identify a specific object existing in the blood vessel.

For the sake of brevity in the following explanation, the image area corresponding to the object in the medical image is referred to as the "object area".

FIG. 5 is an explanatory diagram showing the details of the learning model 50. In FIG. 5, as the learning model 50, a model composed of the first model 51 and the second model 52 is illustrated as an example. The details of the learning model 50 will be described with reference to FIG.

The first model 51 is a machine learning model that inputs a medical image (tomographic image and a perspective image) and outputs a detection result of detecting an object in the image. For example, the first model 51 is a neural network model generated by deep learning, and is a CNN (Convolutional Neural Network) that extracts features of an input image from a large number of convolutional layers. The first model 51 includes an intermediate layer (hidden layer) in which a convolution layer for convolving the pixel information of the input image and a pooling layer for mapping the convoluted pixel information are alternately connected, and a feature amount (feature) of the input image is provided. Quantity map) is extracted.

In this embodiment, the first model 51 is described as being a CNN, but for example, GAN (Generative Adversarial Network), RNN (Recurrent Neural Network), SVM (Support Vector Machine), decision tree, and other learning. It may be a model based on an algorithm.

In the present embodiment, the server 1 generates a first model 51 that identifies on a pixel-by-pixel basis whether or not each pixel in the input medical image is a pixel corresponding to an object area. For example, the server 1 generates a semantic segmentation model (U-net or the like) or MASK R-CNN (Region CNN) as the first model 51.

The semantic segmentation model is a type of CNN, and is a type of EncoderDecoder model that generates output data from input data. The semantic segmentation model includes a deconvolution layer that maps (enlarges) the features obtained by compression to the original image size, in addition to the convolution layer that compresses the data of the input image. In the deconvolution layer, a label image that identifies which object exists in which position in the image on a pixel-by-pixel basis based on the feature amount extracted by the convolution layer, and binarizes which object each pixel corresponds to. To generate.

MASK R-CNN is a modification of Faster R-CNN mainly used for object detection, and is a model in which a deconvolutional layer is connected to Faster R-CNN. In MASK R-CNN, the feature amount of the image extracted by CNN and the coordinate range of the target object extracted by RPN (Region Proposal Network) are input to the reverse convolution layer, and finally the image area of the object in the input image. Generate a mask image to mask.

The server 1 generates these models as the first model 51 and uses them for object detection. It should be noted that all of the above models are examples, and the first model 51 may be sufficient as long as it can identify the position and shape of the object in the medical image. In the present embodiment, as an example, the first model 51 will be described as being a semantic segmentation model.

On the right side of FIG. 4, the object area in the intravascular tomographic image and the fluoroscopic image is conceptually shown by hatching. The object detected by the first model 51 is, for example, a lesion such as a plaque, a therapeutic device such as an indwelling stent, or the like. The server 1 inputs a medical image into the first model 51 and detects various objects.

Note that plaque is an example of a lesion, and for example, a narrowed part of a blood vessel, a calcified tissue, a dissection of a blood vessel wall (flap), a neointima, and other parts may be detected. In addition, the stent is an example of a therapeutic device, and other devices may be detected. Further, the lesion site and the therapeutic device are examples of objects, and other objects may be detected.

Return to Fig. 5 and continue the explanation. The server 1 learns the medical image for training by using the training data in which the data indicating the object area illustrated in FIG. 4 is labeled. Specifically, in the training data, labels (metadata) indicating the coordinate range corresponding to the object area and the type of the object are given to the medical image for training.

The server 1 inputs a medical image for training into the first model 51, and acquires the detection result of detecting the object as an output. Specifically, for each pixel in the object area, a label image in which a value indicating the type of the object is labeled is acquired as an output.

The server 1 compares the detection result output from the first model 51 with the coordinate range of the correct object area and the type of the object indicated by the training data, and parameters such as weights between neurons so that both can be approximated. Optimize. As a result, the server 1 generates the first model 51.

In the present embodiment, two types of images, a tomographic image and a fluoroscopic image, are processed. For example, the server 1 may prepare two first models 51 corresponding to each image, or the first model 51. May be configured as a multimodal model having two networks for processing tomographic and fluoroscopic images, respectively. This makes it possible to detect an object from each image.

In the present embodiment, the first model 51 accepts a plurality of frames of medical images (moving images) that are continuous in time series as input, and detects an object from the medical images of each frame. Specifically, the first model 51 receives a plurality of frames of medical images (for example, a tomographic image) that are continuous along the longitudinal direction of a blood vessel as input according to the scanning of the catheter 211. The first model 51 detects an object from a medical image of each frame continuous along the time axis.

In the following description, for convenience, the medical images of each continuous frame are simply referred to as "frame images".

The server 1 may input a plurality of frame images into the first model 51 one by one and detect an object individually from each frame image, but may input a plurality of consecutive frame images at the same time and input a plurality of frames at the same time. It is preferable to be able to detect the object area from the image at the same time. For example, the server 1 uses the first model 51 as a 3D-CNN (for example, 3D U-net) that handles three-dimensional input data. Then, the server 1 handles the two-dimensional frame image as three-dimensional data having the coordinates as two axes and the time when each frame image is acquired (the time when the frame image is generated) as one axis. The server 1 inputs a plurality of frame images (for example, 16 frames) for a predetermined unit time into the first model 51 as a set, and simultaneously outputs a label image in which an object area is labeled for each of the plurality of frame images. As a result, the object can be detected in consideration of the frame images before and after the continuous time series, and the detection accuracy can be improved.

In the above, it is possible to process a time-series frame image by using three-dimensional data including the time axis, but the present embodiment is not limited to this. For example, the server 1 may be able to detect an object from a plurality of consecutive frame images by using the first model 51 as a model in which CNN and RNN are combined. In this case, for example, an RSTM (Long-Short Term Memory) layer is inserted after the intermediate layer related to CNN, and the object is detected by referring to the feature amounts extracted from the frame images before and after. Even in this case, similarly to the above, the processing can be performed in consideration of the frame images before and after, and the detection accuracy can be improved.

Next, the second model 52 will be described. The second model 52 is a machine learning model that inputs patient information and medical images and outputs treatment information for supporting endovascular treatment of the patient. For example, the second model 52 is a CNN, includes an intermediate layer in which convolution layers and pooling layers are alternately connected, and extracts features of input data.

Although the second model 52 is described as being a CNN in the present embodiment, it may be a model based on other learning algorithms such as GAN, RNN, SVM, and decision tree.

The server 1 learns the patient information for training and the medical image by using the training data to which the correct treatment information is added. The patient information and medical image for training are patient information and medical image of a patient who has undergone treatment, and are medical records of the patient and medical images such as tomographic images and fluoroscopic images obtained at the time of treatment. The correct treatment information is a record of the treatment (PCI) given to the patient. The server 1 gives the patient information and the medical image stored in the medical treatment DB 141 and the treatment information as training data to the second model 52, and performs learning.

Specifically, the server 1 inputs data such as the patient's age, gender, and diagnosis name included in the patient information into the second model 52 as a categorical variable indicating the attributes of the medical image. The server 1 inputs the patient information as a categorical variable to the second model 52 together with the tomographic image and the fluoroscopic image, and acquires the treatment information of each item illustrated in the upper right of FIG. 4 as an output.

As shown in the upper right of FIG. 4, the output items of the second model 52 include items such as "lesion site" and "lesion property", which are indicated by classification results, and "total contrast time" and "perspective time". There are also items that are expressed as continuous values, such as ". In this way, the second model 52 handles the so-called classification problem and the regression problem at the same time. For example, all items are regarded as regression problems and continuous values are output, and the items to be shown as the classification result are changed from continuous values to binary values. You just have to convert. Alternatively, an output layer corresponding to each item may be provided separately, and the feature amount extracted in the intermediate layer may be input to each output layer to estimate each item separately. Alternatively, the second model 52 itself may be prepared for each item and estimated separately.

The server 1 may use the medical image of the blood vessel of the patient as it is for the input of the second model 52, but in the present embodiment, the object region detected by the first model 51 is processed so as to be identifiable. 2 Medical images are generated and input to the second model 52. The second medical image is an image in which the object area is displayed in different display modes depending on the type of the object. For example, the label image output from the first model 51 is superimposed on the original medical image.

For example, the server 1 processes the label image output from the first model 51 into a translucent mask and superimposes it on the original medical image. In this case, the server 1 changes the display mode of each object area according to the type of the object, such as changing the display color of the mask according to the type of the object. As a result, the server 1 generates a medical image that displays the object area in a display mode different from that of the other areas, and inputs it to the second model 52.

The server 1 inputs the patient information for training and the medical image obtained by processing the object area into the second model 52, and acquires the treatment information as an output. The server 1 compares the output treatment information with the correct treatment information, and optimizes parameters such as weights between neurons so that the two can be approximated. As a result, the server 1 generates the second model 52.

In the present embodiment, two types of models, the first model 51 for object detection and the second model 52 for generating treatment information, are combined, but the object can be detected on a rule basis. good. That is, the configuration in which the learning model 50 includes the first model 51 is not essential, and an object may be detected on a rule basis as a preprocessing for input to the learning model 50, and the detection result may be input to the learning model 50. ..

When generating treatment information for a patient to be newly treated, the server 1 acquires a medical image of the patient from the diagnostic imaging system 2 and inputs it to the learning model 50 to generate treatment information including the second medical image. do. The processing may be performed in real time at the time of image acquisition, or the recorded medical images (moving images) may be collectively acquired and processed after the fact.

The server 1 outputs the generated treatment information to the display device 23 and displays it. In the present embodiment, the treatment information is output to the diagnostic imaging system 2, but the treatment information may be output to a device other than the diagnostic imaging system 2 (for example, a personal computer) to display the treatment information. Of course.

FIG. 6 is an explanatory diagram showing an example of a display screen of the display device 23. The display device 23 displays each treatment information output from the server 1 and presents it to the user. For example, the display device 23 displays a list 71, a progress information column 72, a tomographic image 73, and a perspective image 74.

List 71 is a table showing the estimation results of each item output from the second model 52 in a list. List 71 includes, for example, information about lesions, information about therapeutic devices, information about conditions for taking fluoroscopic images, and other information. The display device 23 presents these treatment information to the user in the list 71 to support endovascular treatment.

The progress information column 72 is a display column showing the estimation result of estimating the progress of the user after the treatment, and is the estimation result regarding the onset of complications as described above. The server 1 uses the learning model 50 to calculate the probability value of onset for each of a plurality of complications, and displays the probability value in a list in the progress information column 72 in association with the complication name.

Further, the display device 23 displays the tomographic image 73 generated by the intravascular image diagnostic device 21 and the fluoroscopic image 74 taken by the fluoroscopic image capturing device 22. In this case, the display device 23 displays the second medical image obtained by processing the object area of the original medical image as the tomographic image 73 and the fluoroscopic image 74.

FIG. 7 is a flowchart showing the procedure of the generation process of the learning model 50. Based on FIG. 7, the processing content when learning the training data and generating the learning model 50 will be described.
The control unit 11 of the server 1 acquires training data to which correct treatment information is added to the patient information for training and the medical image (step S11). The patient information and medical image for training are the patient information and medical image of the patient who has been treated. The treatment information is information on the treatment performed on the patient, and includes the treatment record stored in the medical treatment DB 141 and the label data indicating the object area.

Based on the training data, the control unit 11 generates a first model 51 that detects an object when a medical image is input (step S12). For example, the control unit 11 generates the CNN related to the semantic segmentation as the first model 51 as described above. The control unit 11 inputs a medical image for training to the first model, and acquires the detection result of detecting the object area as an output. The control unit 11 compares the detected object region with the label data of the correct answer, optimizes parameters such as weights between neurons so that the two approximate each other, and generates the first model 51.

Further, the control unit 11 generates a second model 52 that outputs treatment information for supporting the treatment of the patient when the patient information and the medical image are input based on the training data (step S13). Specifically, the control unit 11 generates a second model 52 that outputs treatment information by inputting patient information and a second medical image obtained by processing an object area based on the detection result of the first model 51. The control unit 11 inputs the patient information for training and the second medical image obtained by processing the object area based on the label data of the correct answer into the second model, and acquires the treatment information as an output. The control unit 11 compares the output treatment information with the correct treatment information, optimizes parameters such as weights between neurons so that the two are close to each other, and generates the second model 52. The control unit 11 ends a series of processes.

FIG. 8 is a flowchart showing a procedure for outputting treatment information. Based on FIG. 8, the processing content when outputting the treatment information of the patient to be treated using the learning model 50 will be described.
The control unit 11 acquires patient information about the patient to be treated and a medical image of the blood vessel of the patient (step S31). The control unit 11 inputs the acquired medical image into the first model 51 and detects an object (step S32). Specifically, the control unit 11 inputs the tomographic image and the fluoroscopic image imaged by the intravascular image diagnostic apparatus 21 and the fluoroscopic image capturing apparatus 22 into the first model 51, and detects the tomographic image and the object in the fluoroscopic image. do.

The control unit 11 inputs the second medical image obtained by processing the object area based on the detection result in step S32 and the patient information into the second model 52 to generate treatment information (step S33). The control unit 11 outputs the generated treatment information to the display device 23 and displays it (step S34). Specifically, the control unit 11 outputs information such as a lesion in a blood vessel, a therapeutic device to be used, and imaging conditions of a fluoroscopic image, as well as a second medical image obtained by processing an object area as treatment information. It is displayed on the display device 23. The control unit 11 ends a series of processes.

From the above, according to the first embodiment, it is possible to suitably support the treatment based on the medical image by using the learning model 50 in which the training data has been learned.

Further, according to the first embodiment, the position, properties, etc. of the lesion in the blood vessel (luminal organ) can be estimated and presented to the user.

Further, according to the first embodiment, it is possible to estimate the type, shape, number of uses, order of use, etc. of the therapeutic device to be used and present it to the user.

Further, according to the first embodiment, it is possible to estimate and present suitable imaging conditions for the fluoroscopic image.

Further, according to the first embodiment, it is possible to estimate the progress information after the treatment such as the onset of complications and present it to the user.

Further, according to the first embodiment, the first model 51 that detects an object from a medical image and the second model 52 that generates treatment information based on the detection result by the first model 51 are combined. The correct position, shape, etc. of the object can be given to the second model 52, and the estimation accuracy of the treatment information can be improved.

Further, according to the first embodiment, it is possible to suitably assist the confirmation of the lesion portion or the like by generating the second medical image obtained by processing the object region.

Further, according to the first embodiment, the estimation accuracy of the treatment information can be improved by inputting a plurality of frame images generated in time series into the learning model 50.

Further, according to the first embodiment, by using not only the tomographic image in the blood vessel (luminal organ) but also the perspective image as the medical image, not only the local image but also the image of the entire blood vessel can be used as the learning model 50. It can be given and the estimation accuracy can be improved.

(Modification example 1)
In the first embodiment, a mode in which the treatment information is output using the learning model 50 has been described. In this modification, a mode in which a correction input for correcting the output treatment information is received from the user and re-learning is performed based on the corrected treatment information will be described.

FIG. 9 is a flowchart showing a procedure for outputting treatment information according to the first modification. After outputting the treatment information (step S34), the server 1 executes the following processing.
The control unit 11 of the server 1 receives the correction input of the treatment information output to the display device 23 from the user (step S35). For example, the control unit 11 receives a correction input for correcting the information of each item displayed in the list 71 on the display screen illustrated in FIG. Further, the server 1 accepts input of the correct coordinate range, type, etc. of the second medical image displayed as the tomographic image 73 when the coordinate range of the object area, the type of the object, etc. are different from the actual ones.

When the correction input of the treatment information is received, the control unit 11 performs re-learning using the patient information and the medical image input to the learning model 50 and the corrected treatment information as training data, and updates the learning model 50. (Step S36). That is, the control unit 11 optimizes parameters such as weights between neurons so that the treatment information output from the learning model 50 approximates the modified treatment information, and regenerates the learning model 50. The control unit 11 ends a series of processes.

From the above, according to the present modification 1, the learning model 50 can be optimized through the operation of this system.

(Modification 2)
In the first embodiment, an embodiment in which an object is detected by the first model 51 and a second medical image obtained by processing an object area is input to the second model 52 has been described. In this modification, a mode in which the type, dimensions, etc. of the object are specified from the detection result of the object, and the object information indicating the specified type, dimensions, etc. of the object is used as the input of the second model 52 will be described.

FIG. 10 is an explanatory diagram of the learning model 50 according to the second modification. The learning model 50 according to the present modification also includes the first model 51 and the second model 52 as in the first embodiment. In this modification, the server 1 performs image analysis of the object area detected by the first model 51 to specify the type, dimensions, and the like of the object.

For example, the server 1 specifies the type, size, etc. of the stent already placed in the patient's blood vessel. In this modification, the object to be specified is a stent, but it may be another object such as a lesion. The server 1 performs image analysis of the object region detected as the stent, identifies the name of the indwelling stent, and specifies the diameter, length, and the like of the stent. The server 1 inputs data such as the specified stent type and dimensions into the second model 52 as object information, and generates treatment information.

In this case, the server 1 may input not only the object information (text data) but also the original medical image input to the first model 51 into the second model 52. Alternatively, the server 1 may input the second medical image obtained by processing the object area into the second model 52 as in the first embodiment.

As described above, in this modification, image analysis is performed as preprocessing of the second model 52, object information is specified, and the object information is input to the second model 52. As a result, data such as the type and dimensions of the object can be given to the second model 52, and the estimation accuracy can be improved.

Since it is the same as the first embodiment except that the image analysis is performed as the preprocessing of the second model 52, the flowchart and other detailed explanations will be omitted in this modification.

(Embodiment 2)
In this embodiment, endovascular treatment will be described, and in particular, a mode for supporting the placement and expansion of a stent will be described. The contents overlapping with the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

FIG. 11 is an explanatory diagram of the learning model 50 according to the second embodiment. Similar to the first embodiment, the learning model 50 according to the present embodiment also includes the first model 51 and the second model 52, and generates a second medical image obtained by processing the object region and other treatment information. As shown in the upper right of FIG. 11, the treatment information includes stent information regarding the stent to be placed in the blood vessel of the patient, and the type, shape (diameter, length) and the like of the stent to be used are output.

In the present embodiment, the learning model 50 further includes a third model 53. The third model 53 is a machine learning model that estimates a target position in a blood vessel in which a stent should be placed and a target dilation diameter when a medical image before treatment is input. Specifically, the third model 53 takes a medical image as an input and detects an image region in which the stent should be placed and expanded. In addition to the stent information generated by the second model 52, the server 1 detects an image region in which the stent should be placed and expanded by the third model 53, generates a second medical image showing the detected region, and generates the stent information. It is displayed on the display device 23 as one of the above.

In the following description, for convenience, the area where the stent should be placed and expanded is referred to as the "target area".

For example, the server 1 uses Mask R-CNN as the third model 53. As described above, the Mask R-CNN is a CNN that detects a target image area from the input image, and is a model that can identify the target image area on a pixel-by-pixel basis. For example, the server 1 generates a third model 53 using the data with a label indicating the coordinate range of the stent placed in the blood vessel of the patient as training data for the fluoroscopic image of the patient who has been treated.

The third model 53 is not limited to the Mask R-CNN, and may be another CNN such as U-net or another machine learning model such as GAN.

For example, the third model 53 detects the target area as a rectangular bounding box as shown in FIG. The target medical image may be a tomographic image as well as a fluoroscopic image. Further, the target region may be any one as long as the stent is placed and the region to be expanded by the balloon is correctly indicated, and the shape of the target region is not limited to the rectangular shape.

The server 1 may directly input the fluoroscopic image acquired from the fluoroscopic image capturing device 22 to the third model 53, but as shown in FIG. 11, the first model 51 processes an object area such as a lesion. 2 It is preferable to input a medical image. As a result, the third model 53 can determine the target area in consideration of the position, shape, and the like of the lesion to be treated.

Note that, as in the modified example 2, the type and dimensions of the object such as the lesion may be specified from the second medical image, and the specified object information may be given to the third model 53.

For example, the server 1 should expand the lesion and the stent by superimposing a bounding box indicating the target region detected by the third model 53 on the second medical image obtained by processing the object region corresponding to the lesion. Generate a second medical image showing the range at the same time. The server 1 may generate an image surrounding only the target area as a second medical image. The server 1 outputs the generated second medical image as a fluoroscopic image 74 to the display device 23.

In the present embodiment, the first model 51 for detecting an object such as a lesion and the third model 53 for detecting a target area where a stent should be placed and expanded are separately prepared. May be the same model, and the object detection and the target area detection may be performed at the same time.

12 and 13 are explanatory views showing a display example of a second medical image according to the second embodiment. In the present embodiment, the display device 23 displays a second medical image showing the target area in which the stent should be placed and expanded, and displays the stent currently inserted in the blood vessel in a distinguishable manner in the second medical image. , Assist in stent placement and expansion. 12A to 13B show in chronological order how the stent is inserted into the blood vessel, reaches the target position, and expands.

FIGS. 12A and 12B show how the stent is inserted to the target region. The display device 23 identifiablely displays the object area 741 and the target area 742 in the perspective image 74 by a method such as color display. As described above, the object region 741 corresponds to the lesion and the target region 742 corresponds to the region within the blood vessel where the stent 743 should be placed.

Further, the display device 23 displays the stent 743 inserted into the blood vessel and the rectangular stent region 744 representing the current position of the stent 743. The server 1 detects at least the current position of the stent and the current expansion diameter (hereinafter referred to as “current diameter”) from the fluoroscopic image acquired from the fluoroscopic image capturing apparatus 22, and displays the stent in an identifiable manner by a method such as color display. Let me.

The stent 743 may be detected by using the first model 51 or by image recognition by pattern matching.

Further, the server 1 calculates the difference value between the current position of the detected stent 743 and the target position of the stent 743 indicated by the target area 742, and displays it in the upper left of the fluoroscopic image 74. The current position of the stent 743 may be, for example, the midpoint of the elongated stent 743 as the current position, or the tip of the stent 743 as the current position, and any point of the stent 743 can be detected as the current position. .. Further, the target position of the stent 743 may be, for example, the center of gravity of the rectangular target area 742 as the target position, or the midpoint of the short side of the target area 742 located on the opposite side of the stent 743 as the target position. , Any point in the target area 742 can be set as the target position. The server 1 sequentially detects the stent 743 from the fluoroscopic image, calculates the difference value between the current position and the target position, and displays it on the display device 23.

The server 1 determines whether or not the stent 743 has reached the target position based on the difference value between the current position and the target position. When it is determined that the target position has been reached, the server 1 calculates the expansion rate obtained by dividing the current diameter by the target expansion diameter based on the current diameter of the stent 743 and the target expansion diameter of the stent 743 indicated by the target region 742. , It is displayed in the upper left of the fluoroscopic image 74. The target expansion diameter is the width of the target region 742 in the direction orthogonal to the longitudinal direction of the blood vessel, and is the length of the short side of the rectangular target region 742.

FIGS. 13A and 13B show the state from when the stent 743 reaches the target position until the expansion is completed. When the stent 743 reaches the target position, the display device 23 switches the display on the upper left of the fluoroscopic image 74 to the expansion rate. The server 1 sequentially detects the current diameter of the stent 743, calculates the expansion rate, and displays it on the display device 23.

FIG. 14 is a flowchart showing a procedure for generating the learning model 50 according to the second embodiment.
The control unit 11 of the server 1 acquires training data for generating the learning model 50 (step S201). The training data according to the present embodiment includes a medical image of a patient who has undergone treatment, label data indicating an object area such as a lesion, and a label indicating an image area (target area) of a stent placed in a blood vessel. Includes data. The control unit 11 shifts the process to step S12.

After executing the process of step S13, the control unit 11 generates a third model 53 that detects the target region in which the stent should be placed and expanded when the medical image is input (step S202). Specifically, as described above, the control unit 11 generates the Mask R-CNN as the third model 53. For example, the control unit 11 inputs the second medical image obtained by processing the object area of the medical image for training according to the label data into the third model 53, and acquires the detection result of detecting the target area as an output. The control unit 11 compares the acquired target region with the label data of the correct answer, optimizes parameters such as weights between neurons so that the two approximate each other, and generates the third model 53. The control unit 11 ends a series of processes.

FIG. 15 is a flowchart showing a procedure for outputting treatment information according to the second embodiment. After generating the treatment information using the second model 52 (step S33), the server 1 executes the following processing.
The control unit 11 inputs the pre-treatment medical image of the patient into the third model 53 to detect the target region in which the stent should be placed (step S221). Specifically, as described above, the control unit 11 inputs the second medical image (perspective image) generated by using the first model 51 into the third model 53, and detects the target region.

The control unit 11 detects the stent inserted into the patient's blood vessel from the medical image. (Step S222). The control unit 11 generates a second medical image showing the detected stent and the target region, and outputs the second medical image to the display device 23 together with the treatment information generated in step S33 (step S223).

The control unit 11 calculates the difference value between the current position of the stent and the target position of the stent indicated by the target area, and outputs the difference value to the display device 23 (step S224). The control unit 11 determines whether or not the stent has reached the target position based on the difference value between the current position and the target position (step S225). If it is determined that the target position has not been reached (S225: NO), the control unit 11 returns the process to step S224. When it is determined that the target position has been reached (S225: YES), the control unit 11 calculates the expansion rate from the current diameter of the stent and the target expansion diameter, and outputs the expansion rate to the display device 23 (step S226).

The control unit 11 determines whether or not the expansion to the target expansion diameter is completed (step S227). If it is determined that the expansion is not completed (S227: NO), the control unit 11 returns the process to step S226. When it is determined that the expansion is completed (S227: YES), the control unit 11 ends a series of processes.

In the above, the type and shape of the stent to be used in the learning model 50 are automatically predicted, and the target position and expansion conditions of the stent are predicted, but this embodiment is limited to this. It's not something. For example, the server 1 receives a designated input for designating the type and shape of the stent used for endovascular treatment from the user, inputs the designated stent information (first stent information) into the learning model 50, and inputs the designated stent information (first stent information) to the learning model 50. Information such as a target position and expansion conditions (second stent information) may be predicted. For example, the server 1 inputs the first stent information specified by the user into the third model 53 together with the medical image, and detects the target area when the stent specified by the user is used. This makes it possible to support stent placement and expansion according to the user's request.

From the above, according to the second embodiment, it is possible to suitably support the placement and expansion of the stent during endovascular treatment.

(Modification example 3)
In the first modification, a mode in which the treatment information is output, the correction input is received, and the re-learning is performed has been described. Similarly, in the second embodiment, after the stent information is output, a correction input may be received and re-learning may be performed.

FIG. 16 is a flowchart showing a procedure for outputting treatment information according to the third modification. After outputting the treatment information including the stent information (step S223), the server 1 executes the following processing.
The control unit 11 of the server 1 receives the correction input of the output stent information (step S241). For example, the control unit 11 receives an input for correcting the type, shape, etc. of the stent displayed as the stent information. Further, the control unit 11 receives an input for correcting the coordinate range of the target region for the second medical image. The control unit 11 shifts the process to step S224.

When it is determined that the expansion of the stent is completed (S227: YES), the control unit 11 relearns using the original medical image input to the learning model 50 and the modified stent information as training data, and learns. Model 50 is updated (step S242). That is, the control unit 11 optimizes parameters such as weights between neurons so that the stent information output from the learning model 50 approximates the modified stent information, and regenerates the learning model 50. The control unit 11 ends a series of processes.

From the above, according to the present modification 3, the learning model 50 can be optimized through the operation of this system.

It should be considered that the embodiment disclosed this time is an example in all respects and is not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Server (information processing device)
11 Control unit 12 Main storage unit 13 Communication unit 14 Auxiliary storage unit P program 141 Medical care DB
50 Learning model 51 1st model 52 2nd model 53 3rd model 2 diagnostic imaging system 21 intravascular diagnostic imaging device 22 fluoroscopic imaging device 23 display device

Claims

Obtaining patient information about the patient undergoing treatment and a medical image of the patient's luminal organs,
When the patient information and the medical image are input, the acquired patient information and the medical image are input to the trained model to output the treatment information to support the treatment to be performed on the patient, and the treatment information is output. A program that causes a computer to perform processing.
The program according to claim 1, which outputs the treatment information regarding the lesion portion in the luminal organ.
The program according to claim 1 or 2, which outputs the therapeutic information regarding the therapeutic device to be inserted into the luminal organ.
The treatment is endovascular treatment
The program according to any one of claims 1 to 3, which outputs the treatment information regarding the stent to be placed in the blood vessel.
The program according to claim 4, which outputs the treatment information regarding the procedure to be performed before or after the placement of the stent.
The treatment is endovascular treatment
The program according to any one of claims 1 to 5, which outputs the treatment information regarding the balloon that dilates the blood vessel.
The medical image includes a fluoroscopic image of the luminal organ by X-ray.
The program according to any one of claims 1 to 6, which outputs the treatment information regarding the imaging conditions of the fluoroscopic image.
The program according to any one of claims 1 to 7, which outputs progress information in which the progress after treatment of the patient is estimated as the treatment information.
The program according to claim 8, which outputs the progress information regarding complications that may occur after treatment.
When the medical image is input, the acquired medical image is input to the first model trained to detect a predetermined object in the medical image to detect the object.
The detection result of the object by the first model and the acquired patient information are added to the second model trained to output the treatment information when the detection result of the object and the patient information are input. The program according to any one of claims 1 to 9, which is input and outputs the treatment information.
Based on the detection result of the object by the first model, a second medical image in which the image area corresponding to the object in the medical image is displayed in a display mode different from that of other image areas is generated.
The program according to claim 10, wherein the generated second medical image is input to the second model and the treatment information is output.
Based on the detection result of the object by the first model, the object information indicating the type and dimensions of the object is specified, and the object information is specified.
The program according to claim 10, wherein the object information is input to the second model and the treatment information is output.
A plurality of the medical images generated along the longitudinal direction of the luminal organ were acquired and obtained.
The program according to any one of claims 1 to 12, which inputs the plurality of medical images into the model and outputs the treatment information.
A tomographic image generated based on a signal detected by a catheter inserted into the luminal organ and a fluoroscopic image of the luminal organ are acquired.
The program according to any one of claims 1 to 13, which inputs the tomographic image and the fluoroscopic image into the model and outputs the treatment information.
The program according to any one of claims 1 to 14, wherein the medical image includes at least one of an ultrasonic tomographic image, an optical interference tomographic image, and an X-ray fluoroscopic image of the lumen organ.
Accepts the correction input to correct the output treatment information,
The program according to any one of claims 1 to 15, which relearns based on the patient information and the medical image input to the model and the modified treatment information, and updates the model.
Obtaining patient information about the patient undergoing treatment and a medical image of the patient's luminal organs,
When the patient information and the medical image are input, the acquired patient information and the medical image are input to the trained model to output the treatment information to support the treatment to be performed on the patient, and the treatment information is output. An information processing method that causes a computer to perform processing.
An acquisition unit that acquires patient information about a patient to be treated and a medical image of the patient's tract organs.
When the patient information and the medical image are input, the acquired patient information and the medical image are input to the trained model to output the treatment information to support the treatment to be performed on the patient, and the treatment information is output. An information processing device equipped with an output unit.
For the patient information about the patient who has been treated and the medical image which imaged the lumen organ of the patient, the training data in which the treatment information about the treatment which has been performed is given to the patient is acquired.
A model generation method in which a computer executes a process of generating a learned model that outputs the treatment information when the patient information and the medical image are input based on the training data.