WO2021193021A1

WO2021193021A1 - Program, information processing method, information processing device, and model generation method

Info

Publication number: WO2021193021A1
Application number: PCT/JP2021/009301
Authority: WO
Inventors: 陽井口; 悠介関; 雄紀坂口
Original assignee: テルモ株式会社
Priority date: 2020-03-27
Filing date: 2021-03-09
Publication date: 2021-09-30
Also published as: JPWO2021193021A1

Abstract

This program makes a computer execute processing for acquiring a first medical image on which a hollow organ of a patient before treatment is imaged and device information associated with a treatment device used for treating the hollow organ, and generating, when the first medical image and device information are input, a second medical image after treatment by inputting the acquired first medical image and device information to a model having learnt to generate the second medical image.

Description

Program, information processing method, information processing device and model generation method

The present invention relates to a program, an information processing method, an information processing device, and a model generation method.

Treatment is performed based on medical images that visualize the inside of the human body, such as ultrasonic images, optical coherence tomography (OCT) images, and X-ray images. Along with this, various techniques for processing medical images have been proposed so that image observers can preferably observe medical images.

For example, in Patent Document 1, it is an ultrasonic diagnostic apparatus that displays an ultrasonic image of a lesion portion of a subject, and generates an image in which an ultrasonic image before treatment and an ultrasonic image after treatment are superimposed to generate a lesion. An ultrasonic diagnostic apparatus for displaying a treated area and an untreated area of a part in different colors is disclosed.

Japanese Unexamined Patent Publication No. 2008-119071

However, the invention according to Patent Document 1 merely superimposes the image acquired before the treatment and the image acquired after the treatment, and does not generate the image after the treatment.

One aspect is to provide a program or the like that can obtain a medical image that predicts the state after treatment from a medical image before treatment.

A program according to one aspect acquires a first medical image of a patient's luminal organ before treatment and device information related to a therapeutic device used for the treatment of the luminal organ, and obtains the first medical image. A process of inputting the acquired first medical image and device information into a model trained to generate a second medical image after treatment when an image and device information are input to generate the second medical image. To the computer.

On one aspect, it is possible to obtain a medical image that predicts the state after treatment from the medical image before treatment.

It is explanatory drawing which shows the configuration example of the treatment support system. It is a block diagram which shows the configuration example of a server. It is explanatory drawing which shows the outline of Embodiment 1. FIG. It is explanatory drawing about the generative model. It is explanatory drawing which shows the display screen example of a medical image. It is explanatory drawing about the treatment stage of endovascular treatment. It is a flowchart which shows the procedure of the generation process of a generation model. It is a flowchart which shows the procedure of the 2nd medical image generation processing. It is explanatory drawing about the generation model which concerns on Embodiment 2. It is explanatory drawing which shows the display screen example of the medical image which concerns on Embodiment 2. It is a flowchart which shows the procedure of the 2nd medical image generation processing which concerns on Embodiment 3.

Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments thereof.
(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of a treatment support system. In the present embodiment, based on a medical image (hereinafter referred to as "first medical image") in which a patient's blood vessel is imaged before performing endovascular treatment, and device information regarding a therapeutic device used for endovascular treatment. A treatment support system that generates a medical image (hereinafter, "second medical image") that predicts the state of blood vessels after treatment will be described. The treatment support system includes an information processing device 1 and a diagnostic imaging system 2. The information processing device 1 and the diagnostic imaging system 2 are communication-connected to a network N such as a LAN (Local Area Network) or the Internet.

In the present embodiment, endovascular treatment will be described as an example, but the target luminal organ is not limited to a blood vessel, and may be another luminal organ such as a bile duct, pancreatic duct, bronchus, or intestine. ..

The diagnostic imaging system 2 includes an intravascular diagnostic imaging device 21, a fluoroscopic imaging device 22, and a display device 23. The intravascular image diagnostic device 21 is a device for imaging an intravascular tomographic image of a patient, for example, an IVUS (Intravascular Ultrasound) device that performs an ultrasonic examination using a catheter 211. Catheter 211 is a medical device that is inserted into a patient's blood vessel and comprises an imaging core that transmits ultrasonic waves and receives reflected waves from within the blood vessel. The intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image based on the signal of the reflected wave received by the catheter 211 and displays it on the display device 23.

In the present embodiment, the intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image. For example, an optical interference tomographic image is taken by an optical method such as OCT or OFDI (Optical Frequency Domain Imaging). May be good.

The fluoroscopic image capturing device 22 is a device unit for capturing a fluoroscopic image that sees through the inside of a patient, and is, for example, an angiography device that performs angiography examination. The fluoroscopic image capturing device 22 includes an X-ray source 221 and an X-ray sensor 222, and the X-ray sensor 222 receives the X-rays emitted from the X-ray source 221 to capture an X-ray fluoroscopic image of the patient.

In the above, ultrasonic tomography, optical interference tomography, and angiography are given as examples of medical images, but the medical images are computed tomography (CT) images and magnetic resonance imaging (MRI). It may be an image or the like.

The information processing device 1 is an information processing device capable of transmitting and receiving various types of information processing and information, such as a server computer and a personal computer. In the present embodiment, it is assumed that the information processing device 1 is a server computer, and in the following, it will be read as server 1 for the sake of brevity. The server 1 may be a local server installed in the same facility (hospital or the like) as the diagnostic imaging system 2, or may be a cloud server communicated and connected via the Internet or the like. The server 1 functions as a generator that generates a second medical image from the first medical image (ultrasonic tomographic image and X-ray perspective image) generated by the diagnostic imaging system 2, and the generated second medical image is image-diagnosed. Output to system 2.

Specifically, as will be described later, the server 1 performs machine learning to learn predetermined training data in advance, and generates a second medical image by inputting the first medical image and device information (FIG. 5). See 3rd grade). The device information will be described later. The server 1 acquires the first medical image and device information from the diagnostic imaging system 2 and inputs them to the generation model 50 to generate the second medical image and display it on the display device 23.

In the present embodiment, the second medical image is generated on the server 1 separate from the diagnostic imaging system 2, but the generation model 50 generated by the server 1 by machine learning is installed in the diagnostic imaging system 2. , The diagnostic imaging system 2 may be capable of generating a second medical image.

FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.
The control unit 11 has one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and other arithmetic processing units, and stores the program P stored in the auxiliary storage unit 14. By reading and executing, various information processing, control processing, etc. are performed. The main storage unit 12 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing related to communication, and transmits / receives information to / from the outside.

The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores a program P and other data necessary for the control unit 11 to execute processing. Further, the auxiliary storage unit 14 stores the generation model 50. The generative model 50 is a machine learning model in which training data has been trained as described above, and is a model that generates a second medical image by inputting a first medical image and device information. The generation model 50 is expected to be used as a program module constituting artificial intelligence software.

The auxiliary storage unit 14 may be an external storage device connected to the server 1. Further, the server 1 may be a multi-computer composed of a plurality of computers, or may be a virtual machine virtually constructed by software.

Further, in the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays an image, and the like. Further, the server 1 is provided with a reading unit for reading a portable storage medium 1a such as a CD (CompactDisk), a DVD (DigitalVersatileDisc), or a USB (UniversalSerialBus) memory, and reads a program P from the portable storage medium 1a. You may try to execute it. Alternatively, the server 1 may read the program P from the semiconductor memory 1b.

FIG. 3 is an explanatory diagram showing an outline of the first embodiment. FIG. 3 conceptually illustrates how a second medical image is generated from the first medical image and text information using the generative model 50. An outline of the present embodiment will be described with reference to FIG.

The generation model 50 generates a second medical image that predicts the state after the treatment by inputting the first medical image acquired by the diagnostic imaging system 2 before the treatment of the patient and the device information about the treatment device used for the treatment. It is a machine learning model to do. The first medical image is an ultrasonic tomographic image and an X-ray fluoroscopic image acquired by the intravascular image diagnostic apparatus 21 and the fluoroscopic imaging apparatus 22, respectively, and is an image in which the blood vessel of the patient is inspected before treatment.

The device information is information related to a device used for endovascular treatment, for example, information related to a stent, a balloon, etc. used for PCI (Percutaneous Coronary Intervention). For example, device information includes the length, diameter, type, placement position, balloon type, diameter after expansion by the balloon (hereinafter referred to as "expansion diameter"), expansion pressure, and expansion time of the stent to be placed in the patient's blood vessel. And so on. The therapeutic device is not limited to the stent and the balloon, and may be another device such as a rotablator.

The second medical image is an image that predicts the state in the blood vessel after the treatment using the therapeutic device, and is an ultrasonic tomographic image and an X-ray fluoroscopic image after the treatment. The server 1 may generate only one of the ultrasonic tomographic image and the X-ray perspective image as the second medical image, or may generate three or more types of images.

The server 1 learns the training data including the first medical image and device information for training and the second medical image, and generates the generation model 50. The training data is, for example, data of a patient who has undergone endovascular treatment, and is an ultrasonic tomographic image and an X-ray fluoroscopic image (first medical image) before treatment, and a device related to a therapeutic device used for treating the patient. Includes information and post-treatment ultrasound tomographic and fluoroscopic images. The server 1 performs machine learning using these data as training data, and generates a generation model 50 in advance.

The training data is not limited to the actual patient data, and may be virtual data inflated by a data generation means such as GAN (Generative Adversarial Network).

For example, the server 1 acquires the first medical image and device information from the diagnostic imaging system 2 at the time of treating the patient, generates the second medical image after the treatment, and displays it on the display device 23. Specifically, the server 1 receives a plurality of ultrasonic tomographic images (frame images) continuously imaged along the longitudinal direction of the blood vessel from the intravascular image diagnostic apparatus 21 during the ultrasonic examination using the catheter 211. Acquire sequentially. Further, the server 1 acquires an X-ray fluoroscopic image taken at the same time as the ultrasonic inspection from the fluoroscopic image capturing device 22. The server 1 sequentially inputs a plurality of continuous ultrasonic tomographic images and an X-ray fluoroscopic image at the time of acquisition of each ultrasonic tomographic image into the generation model 50 together with device information, and predicts the state after treatment. Images and fluoroscopic images are sequentially generated.

In the example of FIG. 3, as a second medical image after treatment, an ultrasonic tomographic image and an X-ray fluoroscopic image of a blood vessel after a stent is placed and expanded by a balloon are generated. In FIG. 3, the portion corresponding to the stent is shown by hatching. The server 1 predicts the state of the blood vessel after the treatment using the generative model 50 and presents it as a second medical image.

In the present embodiment, the second medical image is generated at the time of treatment and output to the diagnostic imaging system 2. However, the recorded first medical image is input to the generation model 50 after the fact and used for the second medical. Of course, an image may be generated.

FIG. 4 is an explanatory diagram of the generative model 50. In this embodiment, GAN is used as the generative model 50. The GAN includes a generator that generates output data from input data and a discriminator that discriminates the authenticity of the data generated by the generator, and the generator and the discriminator compete with each other for learning. Build a network by doing.

The generator related to GAN accepts random noise (latent variable) input and generates output data. The discriminator learns the truth of the data given by the generator by using the true data given for learning and the data given by the generator. In GAN, the network is constructed so that the loss function of the generator is finally minimized and the loss function of the discriminator is maximized.

The generator of the generation model 50 according to the present embodiment includes an encoder that converts input data into a latent variable and a decoder that generates output data from the latent variable, and includes a first medical image and a second medical image from device information. To generate. For example, as shown in FIG. 4, the generator of the generative model 50 includes three encoders that accept inputs of three types of input data: an ultrasonic tomographic image before treatment, an X-ray fluoroscopic image, and device information. The generator also comprises two decoders that generate a post-treatment ultrasound tomographic image and a fluoroscopic image. Further, the generative model 50 includes two classifiers for discriminating the authenticity of the data output from each decoder. The generator extracts the features of the input data in each encoder, inputs the latent variable that combines the features of each input data to each decoder, and predicts the state after treatment. Ultrasonic tomographic image and X-ray fluoroscopy. Generate an image. The two classifiers discriminate between the authenticity of the ultrasonic tomographic image and the fluoroscopic image, respectively.

The server 1 performs learning using the first medical image and device information given for training and the second medical image, and generates a generation model 50. For example, the server 1 first fixes the parameters (weights, etc.) of the generator, inputs the first medical image for training and the device information to the generator, and generates the second medical image. Then, the server 1 uses the second medical image generated by the generator as fake data, gives the second medical image for training as true data to the classifier, and optimizes the parameters of the classifier. Next, the server 1 fixes the parameter of the classifier to the optimum value and learns the generator. The server 1 optimizes the parameters of the generator so that the probability of authenticity is close to 50% when the second medical image generated by the generator is input to the classifier. As a result, the server 1 generates the generative model 50. When actually generating the second medical image from the first medical image, only the generator is used.

The generative model 50 is not limited to GAN, and may be a model based on a neural network such as VAE (Variational Autoencoder), CNN (for example, U-net), or another learning algorithm.

Further, the network structure of the generation model 50 shown in FIG. 4 is an example, and the present embodiment is not limited to this. For example, two images, an ultrasonic tomographic image and an X-ray perspective image, may be combined and regarded as a single image, and two images may be processed simultaneously by one type of encoder and decoder. Further, the encoder for extracting the feature amount of the device information is not prepared, and the device information (text data) may be encoded by a means such as One-hot encoding and directly mapped (combined) to the latent space. As described above, various changes can be considered in the network structure of the generative model 50.

As the second medical image for training, the server 1 uses a second medical image that displays an object of interest to the user, such as a stent, in a display mode different from that of other image areas, as illustrated in FIG. Suitable. For example, the server 1 uses an image in which an image area corresponding to an object such as a stent is labeled (processed) by color coding, edge enhancement, or the like. This allows the server 1 to generate a second medical image that allows the user to identify the object of interest.

The labeled object is not limited to a therapeutic device such as a stent, and may be another object such as a lesion (plaque or the like) in a blood vessel or a lumen boundary.

In addition, although the stent information is mentioned as input data other than the medical image in the above, other information related to endovascular treatment may also be input to the generation model 50. For example, the server 1 inputs lesion information regarding a lesion portion in a blood vessel into the generation model 50. The lesion information includes, for example, the type of lesion (plaque, calcified tissue, vascular stenosis, etc.), position, and properties (hardness of lesion, etc.). For example, the server 1 also inputs the lesion information into the encoder corresponding to the stent information and converts it into a latent variable. As a result, the generative model 50 can generate a second medical image while also referring to the nature and state of the lesion.

FIG. 5 is an explanatory diagram showing an example of a display screen of a medical image. FIG. 5 is an example of a screen displayed by the display device 23, and shows an example of a screen displaying a first medical image and a second medical image. The processing flow executed at the time of endovascular treatment will be described with reference to FIG.

For example, the display device 23 displays the ultrasonic tomographic image and the X-ray perspective image of the current blood vessel corresponding to the first medical image as the current tomographic image 501 and the current perspective image 502. Further, in order to accept the designated input of the device information, the display device 23 displays the designated

fields

503, 503, 503 ... In the menu bar on the left side of the screen. In addition, the display device 23 displays a status column 504 for accepting a designated input of the current treatment stage on the menu bar.

The display device 23 accepts a designated input of device information on the screen. Specifically, the display device 23 accepts the operation input for drawing a rectangular frame on the current fluoroscopic image 502, thereby accepting the designated input of the site (position of the lesion portion) in the blood vessel to be treated. Further, the display device 23 accepts designation inputs such as the length and diameter of the stent and the expansion diameter by the balloon in each designation field 503.

In the example of FIG. 5, the device information of each item is individually input in the plurality of designation fields 503. However, for example, by inputting the product name of the stent or balloon, each item can be specified at the same time. good.

Further, the display device 23 accepts the designated input of the treatment stage of the current endovascular treatment via the status column 504. The server 1 generates a second medical image according to the treatment stage specified in the status column 504 and outputs it to the display device 23.

FIG. 6 is an explanatory diagram regarding the treatment stage of endovascular treatment. FIG. 6 conceptually illustrates a general treatment flow when performing PCI. Generally, PCI is carried out by the following procedure.

First, a guide wire for guiding the catheter 211 is inserted into the patient's blood vessel. Next, the catheter 211 is inserted to obtain an ultrasonic tomographic image, and the condition of the lesion is confirmed together with the fluoroscopic image. The balloon is then vasodilated prior to placement, if necessary, so that the stent can be placed. Balloon expansion before stent placement may be omitted.

Next, a stent is placed in the blood vessel and the lesion is expanded with a balloon. After the stent is placed, the ultrasonic image is acquired again, and the condition of the lesion after expansion is confirmed together with the fluoroscopic image. After that, if the stent is not sufficiently dilated, the blood vessel is dilated again by the balloon. Balloon expansion after stent placement may be omitted. Finally, the condition of the lesion is confirmed on each image, and a series of treatments is completed.

In the following explanation, for the sake of brevity, balloon dilation before stent placement is referred to as "pre-dilation", and balloon dilation after stent placement is referred to as "post-dilation".

In the present embodiment, the server 1 generates a second medical image after the treatment from the first medical image before the treatment at each treatment stage by the generation model 50 in accordance with the above-mentioned pre-dilation, stent expansion, and post-dilation. do.

For example, the server 1 configures the generative model 50 as a model in which the class label of the input data can be input, such as the Conditional GAN. When the server 1 inputs the first medical image into the generative model 50, the label data indicating the class of the first medical image is used before the pre-expansion procedure, before the stent expansion, or after the post-expansion. Enter a value that indicates whether it is before the treatment of. Then, the server 1 generates a second medical image after any of the treatments according to the input label data. As a result, the user can confirm the second medical image that predicts the state after the treatment to be performed step by step.

FIG. 5 illustrates an example of a display screen when the stent is expanded. Server 1 produces a second medical image after stent expansion according to the treatment stage specified in status column 504. That is, the server 1 generates an ultrasonic tomographic image generated by the intravascular diagnostic imaging apparatus 21 before stent expansion, an X-ray perspective image captured by the perspective imaging apparatus 22, and device information specified by the user. It is input to the model 50 to generate an ultrasonic tomographic image and an X-ray perspective image that predict the state after stent expansion. In this case, for example, the server 1 generates an ultrasonic tomographic image and an X-ray perspective image showing the image region corresponding to the stent in a display mode different from that of other regions as a second medical image.

The server 1 outputs the generated second medical image and displays it on the display device 23. The display device 23 displays the ultrasonic tomographic image and the X-ray perspective image corresponding to the second medical image as the predicted tomographic image 505 and the predicted perspective image 506.

Although FIG. 5 shows a two-dimensional ultrasonic tomographic image and an X-ray fluoroscopic image as the second medical image, the server 1 reconstructs the two-dimensional ultrasonic tomographic image and the X-ray fluoroscopic image, and 3 A dimensional blood vessel image may be generated and displayed on the display device 23. As described with reference to FIG. 3, the server 1 sequentially inputs a plurality of ultrasonic tomographic images (transverse layer images) sequentially acquired from the intravascular diagnostic imaging apparatus 21 and an X-ray perspective image into the generation model 50, and then sequentially inputs the plurality of ultrasonic tomographic images (transverse layer images) to the generation model 50. A plurality of ultrasonic tomographic images predicting the state after treatment and an X-ray perspective image are generated as a second medical image. The server 1 aligns each ultrasonic tomographic image with the X-ray fluoroscopic image according to the position of the catheter 211 detected by means such as an X-ray opaque marker during the ultrasonic examination, and uses a known method to perform a three-dimensional image. Convert to. Thereby, the state in the blood vessel after the treatment can be expressed in three dimensions and presented to the user.

Further, in the example of FIG. 5, one pattern of designated input of device information is accepted and one pattern of the second medical image is displayed. However, a plurality of patterns of designated input of device information are accepted to support a plurality of patterns. A plurality of second medical images may be displayed. For example, the server 1 accepts designated inputs of a plurality of stents as candidates for stents used for treatment, separately generates a second medical image when each stent is used, and displays the preview on the display device 23. Thereby, the convenience of the user can be improved.

FIG. 7 is a flowchart showing the procedure of the generation process of the generation model 50. Based on FIG. 7, the processing content when the generation model 50 is generated by machine learning will be described.
The control unit 11 of the server 1 is data about a patient who has been treated, and is a first medical image which is a medical image before treatment, device information about a treatment device used for treatment, and a medical after treatment. The training data including the second medical image which is an image is acquired (step S11). Specifically, the control unit 11 acquires an ultrasonic tomographic image and an X-ray perspective image before treatment as a first medical image for a patient who has undergone endovascular treatment. In addition, the control unit 11 acquires information on the stent, balloon, and the like used for endovascular treatment as device information. In addition, the control unit 11 acquires an ultrasonic tomographic image and an X-ray perspective image after endovascular treatment as a second medical image.

Based on the training data, the control unit 11 generates a generation model 50 that generates a second medical image when the first medical image and device information are input (step S12). Specifically, the control unit 11 generates a GAN that inputs the ultrasonic tomographic image before the treatment, the X-ray fluoroscopic image, and the device information, and outputs the ultrasonic tomographic image and the X-ray fluoroscopic image after the treatment. .. The control unit 11 ends a series of processes.

FIG. 8 is a flowchart showing a procedure for generating a second medical image. Based on FIG. 8, the processing content when generating the second medical image will be described.
The control unit 11 of the server 1 acquires device information regarding a therapeutic device used for treating a patient's luminal organ from the diagnostic imaging system 2 (step S31). Further, the control unit 11 acquires the first medical image from the diagnostic imaging system 2 (step S32). Specifically, as described above, the control unit 11 acquires an ultrasonic tomographic image of the patient to be treated and an X-ray fluoroscopic image taken at the same time as the ultrasonic examination.

The control unit 11 determines whether or not pre-expansion before stent placement is necessary in response to an operation input from the user (step S33). When it is determined that the pre-expansion is not necessary (S33: NO), the control unit 11 shifts the process to step S36. When it is determined that the pre-expansion is necessary (S33: YES), the control unit 11 inputs the first medical image acquired in step S32 into the generation model 50 to generate the second medical image after the pre-expansion. , Output to the display device 23 (step S34).

The control unit 11 determines whether or not the pre-expansion is completed in response to the operation input from the user (step S35). When it is determined that the pre-expansion is not completed (S35: NO), the control unit 11 waits for processing.

When it is determined that the pre-expansion is completed (S35: YES), the control unit 11 generates a second medical image after the stent expansion and outputs it to the display device 23 (step S36).

The control unit 11 determines whether or not post-extension is necessary in response to an operation input from the user (step S37). When it is determined that post-extension is not necessary (S37: NO), the control unit 11 ends a series of processes. When it is determined that post-expansion is necessary (S37: YES), the control unit 11 reacquires the first medical image (step S38). The control unit 11 inputs the acquired first medical image into the generation model 50, generates the second medical image after post-expansion, and outputs the second medical image to the display device 23 (step S39). The control unit 11 ends a series of processes.

From the above, according to the first embodiment, it is possible to generate a second medical image that predicts the state after the treatment from the first medical image before the treatment and present it to the user.

Further, according to the first embodiment, it is possible to present a second medical image after each treatment according to a plurality of treatment stages.

Further, according to the first embodiment, it is possible to generate a second medical image after each of the pre-dilation, the stent dilation, and the post-dilation according to the endovascular treatment and present it to the user.

Further, according to the first embodiment, the device information can be preferably specified by accepting the designated input of the target portion on the fluoroscopic image.

Further, according to the first embodiment, the state after treatment is represented three-dimensionally from a plurality of ultrasonic tomographic images (cross-sectional layer images) sequentially generated using the generative model 50 and an X-ray perspective image. It is also possible to provide a modified image.

Further, according to the first embodiment, the generation accuracy of the second medical image can be improved by using the lesion information in addition to the device information for the input of the generation model 50.

(Embodiment 2)
In this embodiment, a mode in which only an ultrasonic tomographic image is treated as a first medical image and a second medical image will be described. The contents overlapping with the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

FIG. 9 is an explanatory diagram of the generative model 50 according to the second embodiment. The generation model 50 according to the present embodiment does not include an encoder and a decoder for processing an X-ray fluoroscopic image, and has two encoders that accept input of an ultrasonic tomographic image before treatment and device information, respectively, and a super after treatment. It has only a decoder that generates an ultrasonic tomographic image and a classifier that discriminates the authenticity of the ultrasonic tomographic image generated by the decoder. The server 1 uses the ultrasonic tomographic images before and after the treatment of the patient who has undergone endovascular treatment as the training data as the first medical image and the second medical image for training, and generates the generative model 50 shown in FIG. ..

FIG. 10 is an explanatory diagram showing an example of a display screen of a medical image according to the second embodiment. In the present embodiment, the display device 23 displays the current tomographic image 501 corresponding to the first medical image, as in the first embodiment. In the present embodiment, the display device 23 further displays a vertical tomographic image 521 in which a plurality of ultrasonic tomographic images (cross-layer images) are reconstructed.

The vertical tomographic image 521 is a vertical cross-sectional image obtained by reconstructing a plurality of cross-sectional layer images continuously imaged by the intravascular image diagnostic apparatus 21 along the longitudinal direction of the blood vessel. The server 1 reconstructs the longitudinal tomographic image 521 from a plurality of transverse layer images (frame images) continuously imaged along the longitudinal direction of the blood vessel according to the scanning of the catheter 211, and displays the longitudinal tomographic image 521 on the display device 23. In the present embodiment, the display device 23 receives the drawing input of the rectangular frame on the vertical tomographic image 521, and thereby receives the designated input of the part in the blood vessel to be treated. In addition, the display device 23 receives the designated input of other device information in the designated field 503 as in the first embodiment.

The server 1 inputs the ultrasonic tomographic image corresponding to the first medical image and the device information into the generation model 50, and generates the ultrasonic tomographic image corresponding to the second medical image. Then, the server 1 displays the generated ultrasonic tomographic image as a predicted tomographic image 505. Of course, as the predicted tomographic image 505, not only the transverse layer image but also the vertical tomographic image may be displayed.

Note that, as in the first embodiment, a plurality of ultrasonic tomographic images sequentially generated by the generation model 50 may be reconstructed to generate a three-dimensional image and display it on the display device 23.

Since the same as the first embodiment except for the above points, the flowchart and other detailed explanations will be omitted in the present embodiment.

(Embodiment 3)
In the present embodiment, an embodiment in which the input of the correction information of the second medical image generated by the generation model 50 is accepted and the re-learning is performed based on the input correction information will be described.

For example, the display device 23 receives from the user input of correction information for correcting the second medical image generated by the generation model 50 on the display screen illustrated in FIG. Specifically, the display device 23 receives a correction input for correcting the position, length, diameter (width), etc. of the stent (therapeutic device) displayed in color. For example, the display device 23 receives a drawing input for newly drawing the boundary (edge) of the image region corresponding to the stent with respect to the predicted tomographic image 505 or the predicted perspective image 506. The display device 23 transmits the second medical image on which the image area corresponding to the stent is drawn by the user to the server 1 as correction information, and causes the server 1 to perform re-learning.

In the present embodiment, the input of the correction information for the second medical image after the stent expansion is accepted, but the input of the correction information for the second medical image after the pre-expansion or the post-expansion may be accepted. ..

The server 1 relearns based on the correction information acquired from the display device 23, and updates the generation model 50. Specifically, the server 1 uses the first medical image and device information input when the second medical image is generated as training data for re-learning, and the second medical image in which the image area of the stent is drawn. Is given to the generation model 50, and the parameters of the generator and the classifier are updated.

FIG. 11 is a flowchart showing a procedure for generating a second medical image according to the third embodiment. After generating the second medical image after the stent placement and outputting it to the display device 23 (step S36), the server 1 executes the following processing.
The control unit 11 of the server 1 accepts the input of the correction information of the output second medical image (step S201). For example, the control unit 11 receives a correction input for correcting the position, length, diameter, etc. of the stent on the second medical image. The control unit 11 shifts the process to step S37.

After executing NO or the process of step S39 in step S37, the control unit 11 updates the generation model 50 based on the correction information input in step S201 (step S202). Specifically, the control unit 11 uses the first medical image and device information input when the second medical image was generated in step S36 as input data for training, and uses the second medical image generated in step S36 as input data. The corrected image is retrained as output data for training, and the parameters of the generator and the classifier are updated. The control unit 11 ends a series of processes.

From the above, according to the third embodiment, the accuracy of generating the second medical image can be improved through the operation of this system.

It should be considered that the embodiment disclosed this time is an example in all respects and is not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Server (information processing device)
1a Portable storage medium 1b Semiconductor memory 11 Control unit 12 Main storage unit 13 Communication unit 14 Auxiliary storage unit N network P program 2 Image diagnosis system 21 Intravascular image diagnosis device 211 Catheter 22 Perspective imaging device 221 X-ray source 222 X-ray Sensor 23 Display device 501 Current tomographic image 502 Current perspective image 503 Designation column 504 Status column 505 Predicted tomographic image 506 Predicted perspective image 521 Vertical tomographic image

Claims

A first medical image of the patient's luminal organ before treatment and device information related to the therapeutic device used for the treatment of the luminal organ are acquired.
When the first medical image and device information are input, the acquired first medical image and device information are input to a model trained to generate a second medical image after treatment, and the second medical image is input. A program that causes a computer to perform the process of generating.
The first medical image before the treatment of each treatment stage is acquired according to a plurality of treatment stages.
The program according to claim 1, wherein the acquired first medical image is input to the model to generate the second medical image after the treatment of each treatment stage.
The treatment is endovascular treatment using a catheter.
Claim 2 to obtain a first medical image before treatment of at least two or more treatment stages of balloon expansion before stent placement, expansion of the stent, and balloon expansion after stent placement to generate the second medical image. The program described in.
The first medical image includes a fluoroscopic image of the patient's body.
The fluoroscopic image is displayed on the display unit, and the fluoroscopic image is displayed on the display unit.
For the fluoroscopic image, a designated input for designating a site in the luminal organ to be treated and the therapeutic device used for the treatment of the site is accepted.
The program according to any one of claims 1 to 3, wherein the device information indicating the designated site and the therapeutic device is input to the model.
The first medical image includes a cross-layer image within the luminal organ.
A longitudinal tomographic image is generated from a plurality of transverse image images continuously imaged along the longitudinal direction of the luminal organ and displayed on the display unit.
For the longitudinal tomographic image, a designated input for designating a site in the luminal organ to be treated and the therapeutic device used for the treatment of the site is accepted.
The program according to any one of claims 1 to 3, wherein the device information indicating the designated site and the therapeutic device is input to the model.
The plurality of cross-layer images are input to the model to generate the plurality of cross-layer images after treatment.
The program according to claim 5, wherein a three-dimensional second medical image is generated from the generated plurality of cross-layer images after treatment.
Accepts input of lesion information indicating the properties of the lesion in the luminal organ,
The program according to any one of claims 1 to 6, wherein the lesion information is input to the model to generate the second medical image.
Accepting the input of the correction information of the generated second medical image,
The program according to any one of claims 1 to 7, which updates the model based on the modified information.
A first medical image of the patient's luminal organ before treatment and device information related to the therapeutic device used for the treatment of the luminal organ are acquired.
When the first medical image and device information are input, the acquired first medical image and device information are input to a model trained to generate a second medical image after treatment, and the second medical image is input. An information processing method in which a computer executes the process of generating information.
Input the acquisition unit for acquiring the first medical image of the patient's luminal organ before treatment and the device information related to the therapeutic device used for the treatment of the luminal organ, and the first medical image and device information. In this case, information processing including a generation unit for inputting the acquired first medical image and device information into a model trained to generate a second medical image after treatment and generating the second medical image. Device.
Acquisition of training data including first and second medical images imaging the luminal organs of each patient before and after treatment, and device information related to the therapeutic device used for the treatment of the luminal organs. death,
A model generation method in which a computer executes a process of generating a learned model that generates the second medical image when the first medical image and device information are input based on the training data.