JP7309377B2 - X-ray diagnostic device, medical information processing device and program - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to an X-ray diagnostic apparatus, a medical information processing apparatus, and a program.

X-ray diagnostic equipment is used not only for examination but also for treatment. For example, a method called coil embolization is known as a treatment for an aneurysm. In coil embolization, while an operator visually recognizes an X-ray image displayed in real time, a catheter and a guide wire are inserted into the blood vessel and advanced to the aneurysm, and the aneurysm is embolized by filling the aneurysm with a coil. As a result, the inside of the aneurysm is thrombosed to prevent rupture of the aneurysm.

In such coil embolization, the coil filling rate is an important index. The coil filling ratio means the volume occupied by the coil with respect to the volume of the aneurysm, and is also called volume embolization ratio (VER). The operator ends coil filling when, for example, the coil filling rate reaches 20% or more during coil filling. This completes the coil embolization.

U.S. Patent Application Publication No. 2016/0192894

The problem to be solved by the present invention is to support the operator in coil embolization using a coil filling factor.

A medical information processing apparatus according to an embodiment includes a first acquisition unit, a second acquisition unit, a processing unit, and an image generation unit.
The first acquisition unit acquires a three-dimensional medical image of an aneurysm of a subject captured by a medical image diagnostic apparatus.
The second acquiring unit acquires an X-ray image of the aneurysm of the subject, which is captured by an X-ray diagnostic apparatus.
The processing unit is based on a trained model functioned to output an index related to a coil filling factor for the aneurysm from a three-dimensional medical image of the aneurysm and an X-ray image of the aneurysm. Then, from the acquired three-dimensional medical image, the acquired X-ray image, and the learned model, an index related to the coil filling factor for the aneurysm on the acquired X-ray image is output.

The image generation unit generates an image showing a coil filling state based on the output index.

FIG. 1 is a block diagram showing the configuration of a medical information processing system including an X-ray diagnostic apparatus and a medical information processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing the configuration of the X-ray diagnostic apparatus according to the first embodiment. FIG. 3 is a schematic diagram for explaining the data creation function of the X-ray diagnostic apparatus and medical information processing apparatus according to the first embodiment. FIG. 4 is a schematic diagram for explaining another data creation function of the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 5 is a schematic diagram for explaining the learning control function of the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 6 is a schematic diagram for explaining another learning control function of the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 7 is a schematic diagram for explaining functions executed by the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 8 is a schematic diagram for explaining other functions executed by the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 9 is a schematic diagram for explaining still another execution function of the X-ray diagnostic apparatus and the medical information processing apparatus according to the first embodiment. FIG. 10 is a schematic diagram for explaining an image showing a coil filling situation in the first embodiment. FIG. 11 is a block diagram showing the configuration of the medical information processing apparatus according to the first embodiment. FIG. 12 is a flow chart for explaining the operation in the first embodiment. FIG. 13 is a schematic diagram for explaining a modification of the data creation function in the first embodiment. FIG. 14 is a schematic diagram for explaining a modification of the execution function in the first embodiment;

Hereinafter, each embodiment will be described with reference to the drawings. Each embodiment will be described using a case where the coil filling factor is used as an index for the coil filling factor as a representative example. However, the index related to the coil filling rate may not be the coil filling rate itself as long as it is an index corresponding to the coil filling rate. For example, instead of the coil filling rate, an unfilled rate (=100-coil filling rate [%]) may be used as an index related to the coil filling rate, or another index corresponding to the coil filling rate may be used. may Other indicators can be used, for example, the number of filled coils or the number of remaining coils. In any case, the index for the coil filling rate is not limited to the coil filling rate, but may be an index that can be calculated from the coil filling rate (e.g. unfilled rate) or an index corresponding to the coil filling rate (e.g. number of coils). Any index can be used.

<First Embodiment>
FIG. 1 is a block diagram showing the configuration of a medical information processing system including an X-ray diagnostic apparatus and a medical information processing apparatus according to the first embodiment, and FIG. 2 is a block diagram showing the configuration of the X-ray diagnostic apparatus. be. 3 to 9 are schematic diagrams for explaining the data creation function, learning control function, execution function, and acquisition function of the X-ray diagnostic apparatus and the medical information processing apparatus, and FIG. 10 is an image showing the state of coil filling. It is a schematic diagram for explaining. FIG. 11 is a block diagram showing the configuration of a medical information processing apparatus. The medical information processing system shown in FIG. 1 includes an X-ray diagnostic apparatus 1, a medical information processing apparatus 90, an X-ray CT apparatus 200, and an MRI apparatus 210 that can communicate with each other via a network Nw. This medical information processing system is not limited to this, and can be modified to include at least one of the medical information processing apparatus 90, the X-ray CT apparatus 200, and the MRI apparatus 210 in addition to the X-ray diagnostic apparatus 1. is. In addition, although the X-ray diagnostic apparatus 1 has a biplane structure as an example, the X-ray diagnostic apparatus 1 is not limited to this, and may have a single plane structure. It has 70. The imaging device 10 includes a high voltage generator 3, a first X-ray tube 5, a first X-ray detector 7, a C-arm 9, a first collimator 11, a second X-ray tube 15, a second X-ray detector 17, an Ω arm 19, A second collimator 21 , a top plate 24 , a bed 25 and a drive unit 27 are provided.

The high voltage generator 3 generates a tube current to be supplied to the first X-ray tube 5 and the second X-ray tube 15 and a tube voltage to be applied to the first X-ray tube 5 and the second X-ray tube 15 . The high voltage generator 3 is controlled by the processing circuit 74 to supply a tube current to the first X-ray tube 5 and the second X-ray tube 15 and apply a tube voltage to the first X-ray tube 5 and the second X-ray tube 15. do.

Based on the tube current supplied from the high voltage generator 3 and the tube voltage applied by the high voltage generator 3, the first X-ray tube 5 generates a X-rays (hereinafter referred to as first X-rays) are generated. A subject P is irradiated with the first X-rays generated from the first focal point through an X-ray radiation window provided on the front surface of the first X-ray tube 5 . Part of the first X-rays generated from the first focal point is shielded by the first collimator 11 provided between the first X-ray tube 5 and the X-ray radiation window. That is, the first collimator 11 is moved according to the first irradiation range input from the input interface 73, and the irradiation range of the first X-rays is limited.

The first X-ray detector 7 detects the first X-rays emitted from the first X-ray tube 5 and transmitted through the subject P. FIG. For example, the first X-ray detector 7 has a flat panel detector (Flat Panel Detector: hereinafter referred to as first FPD). The first FPD has a plurality of semiconductor detection elements. There are direct conversion type and indirect conversion type semiconductor detection elements. The direct conversion type is a type in which incident X-rays are directly converted into electric signals. The indirect conversion type is a type in which incident X-rays are converted into light by a phosphor and the light is converted into an electric signal.

A projection data generation circuit and a projection data storage circuit (not shown) are provided at the stage subsequent to the first X-ray detector 7 . The projection data generation circuit includes a charge/voltage converter that converts the charges read out in parallel from the first FPD on a row-by-row or column-by-column basis into a voltage, and an A/ It has a D converter and a parallel/serial converter that converts a digital-converted parallel signal into a time-series serial signal. The projection data generation circuit supplies this serial signal as time-series projection data to the projection data storage circuit. The projection data storage circuit sequentially stores the time-series projection data supplied from the projection data generation circuit to generate two-dimensional projection data. This two-dimensional projection data is stored in the memory 71 .

The C-arm 9 holds the first X-ray tube 5 and the first X-ray detector 7 so as to face each other with the subject P and the top plate 24 interposed therebetween, thereby performing X-ray imaging of the subject P on the top plate 24 . It has a configuration that can perform The top plate 24 is provided on the bed 25 so as to be movable along the longitudinal direction, the minor axis direction and the vertical direction of the top plate 24 .

Specifically, the C-arm 9 is movable along the major axis direction and the minor axis direction of the top plate 24 . Also, the C-arm 9 is supported by a support arm via a holding portion. The support arm has a substantially arcuate shape and has a base end attached to a moving mechanism for a rail provided on the ceiling. The C-arm 9 is held by the holding portion so as to be rotatable about an X-direction axis orthogonal to both the Z-direction perpendicular to the top plate 24 and the Y-direction along the longitudinal direction of the top plate 24. . Further, the C-arm 9 has a substantially circular arc shape centered on the Y-direction axis, and is held by the holding portion so as to be slidable along the substantially circular arc shape. That is, the C-arm 9 can perform a sliding motion about the Y-direction axis. In addition, the C-arm 9 can rotate about the axis in the X direction centering on the holding portion (hereinafter referred to as "main rotation operation"). Allows observation of X-ray images from any direction. Furthermore, the C-arm 9 can also rotate about the axis in the Z direction, so that, for example, the central axis of rotation of the slide operation described above can be in the X direction. The imaging axis passing through the first focal point of the first X-ray tube 5 and the center of the detection surface of the first X-ray detector 7 intersects the rotation center axis of the slide movement and the rotation center axis of the main rotation movement at one point. is designed to The intersection point is generally called the isocenter. The isocenter is not displaced even when the C-arm 9 performs the above-described sliding motion and main rotating motion. Therefore, when a site of interest (for example, an aneurysm) is positioned at the isocenter, the site of interest can be easily observed in the moving image of the medical image obtained by the sliding motion or main rotating motion of the C-arm 9 .

Based on the tube current supplied from the high voltage generator 3 and the tube voltage applied by the high voltage generator 3, the second X-ray tube 15 generates a X-rays (hereinafter referred to as second X-rays) are generated. A subject P is irradiated with the second X-rays generated from the second focal point through an X-ray radiation window provided in front of the second X-ray tube 15 . A part of the second X-rays generated from the second focal point is shielded by the second collimator 21 provided between the second X-ray tube 15 and the X-ray radiation window. That is, the second collimator 21 is moved according to the second irradiation range input from the input interface 73, and the irradiation range of the second X-rays is limited.

The second X-ray detector 17 detects X-rays that are generated from the second X-ray tube 15 and pass through the subject P. FIG. For example, the second X-ray detector 17 has a second FPD. A projection data generation circuit and a projection data storage circuit (not shown) are provided at the stage subsequent to the second X-ray detector 17 . The projection data generation circuit includes a charge/voltage converter that converts the charges read out in parallel from the second FPD on a row-by-row or column-by-column basis into a voltage, and an A/ It has a D converter and a parallel/serial converter that converts a digital-converted parallel signal into a time-series serial signal. The projection data generation circuit supplies this serial signal as time-series projection data to the projection data storage circuit. The projection data storage circuit sequentially stores the time-series projection data supplied from the projection data generation circuit to generate two-dimensional projection data. This two-dimensional projection data is stored in the memory 71 .

The Ω arm 19 holds the second X-ray tube 15 and the second X-ray detector 17 so as to face each other with the subject P on the top plate 24 therebetween, thereby performing X-ray imaging of the subject P. It has a configuration that can

Specifically, the Ω arm 19 is movable along the long axis direction and the short axis direction of the top plate 24 . Also, the Ω arm 19 is supported by a support arm via a holding portion. The support arm has a substantially arcuate shape and has a base end attached to a moving mechanism for a rail provided on the ceiling. The Ω arm 19 is held by the holding portion so as to be rotatable about axes in the Z direction perpendicular to the top plate 24 and the Y direction along the longitudinal direction of the top plate 24 . The Ω arm 19 has a substantially circular arc shape centered on the Y-direction axis, and is held by the holding portion so as to be slidable along the substantially circular arc shape. That is, the Ω-arm 19 can perform a sliding motion about the Y-direction axis. In addition, the Ω arm 19 can rotate about the axis in the Z direction around the holding portion (hereinafter referred to as "main rotation operation"). Allows observation of X-ray images from any direction. The imaging axis passing through the second focal point of the second X-ray tube 15 and the center of the detection surface of the second X-ray detector 17 intersects the rotation center axis of the slide movement and the rotation center axis of the main rotation movement at one point. is designed to The intersection point is generally called the isocenter. The isocenter is not displaced even if the Ω-arm 19 performs the above-described sliding motion or main rotating motion. Therefore, when a site of interest (for example, an aneurysm) is located at the isocenter, the site of interest can be easily observed in the moving image of the medical image obtained by the sliding motion or main rotating motion of the Ω-arm 19 .

Such a C-arm 9 and Ω-arm 19 are provided at appropriate locations where a plurality of power sources are applicable to realize motions related to the support arm under the rail, the X-direction axis, the Y-direction axis and the Z-direction axis. provided for. These power sources constitute the driving section 27 .

The drive unit 27 reads a drive signal from the drive control function 742 and causes each of the C-arm 9 and the Ω-arm 19 to slide, rotate, and linearly move. Furthermore, each of the C-arm 9 and the Ω-arm 19 is provided with a state detector for detecting information on its angle (imaging angle), posture, and position. The state detector is composed of, for example, a potentiometer that detects a rotation angle and a movement amount, an encoder that is a position detection sensor, and the like. As the encoder, a so-called absolute encoder such as a magnetic system, a brush system, or a photoelectric system can be used. Various types of position detection mechanisms, such as a rotary encoder that outputs rotational displacement as a digital signal or a linear encoder that outputs linear displacement as a digital signal, can be appropriately used as the state detector.

The console device 70 has a memory 71 , a display 72 , an input interface 73 , a processing circuit 74 and a network interface 76 .

The memory 71 includes a memory body for recording electrical information such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive) and an image memory, and a memory controller and a memory interface attached to the memory body. and peripheral circuits. The memory 71 stores, for example, programs executed by the processing circuit 74, detection data (projection data) received from the first X-ray detector 7 and the second X-ray detector 17, medical images generated by the processing circuit 74, machine A learning model, a learned model, data used for processing by the processing circuit 74, various tables, data during processing, data after processing, and the like are stored. Medical images include, for example, CT images (volume data), MRI images (volume data), X-ray images (X-ray fluoroscopic images), contrast-enhanced vascular X-ray images (DSA images), and superimposed images. The trained model is functionalized to output an index related to the coil filling factor for the aneurysm based on the three-dimensional medical image (volume data) of the aneurysm and the X-ray image of the aneurysm. It is The program, for example, provides a computer with a first acquisition function of acquiring a three-dimensional medical image of an aneurysm of a subject captured by a medical image diagnostic apparatus, an X-ray image of an aneurysm of the subject captured by an X-ray diagnostic apparatus, and A second acquisition function for acquiring a line image, a function to output an index related to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray image of the aneurysm. Based on the attached trained model, from the acquired three-dimensional medical image, the acquired X-ray image, and the learned model, an index related to the coil filling rate for the aneurysm on the acquired X-ray image and an image generation function for generating an image showing the coil filling status based on the output index. Supplementally, as such a program, for example, a program that is installed in the computer in advance from a network or a non-transitory computer-readable storage medium and causes the computer to implement each function of the medical information processing apparatus 77 is used. .

The display 72 is composed of a display body that displays various information such as medical images, an internal circuit that supplies display signals to the display body, and peripheral circuits such as connectors and cables that connect the display body and the internal circuits. there is The internal circuit superimposes incidental information such as object information and projection data generation conditions on the image data supplied from the processing circuit 74 to generate display data, and D/A converts and TV formats the obtained display data. Converts and displays on the display itself. For example, the display 72 outputs a medical image generated by the processing circuit 74, a GUI (Graphical User Interface) for accepting various operations from the operator, and the like. For example, the display 72 is a liquid crystal display or a CRT (Cathode Ray Tube) display. Also, the display 72 is an example of a display unit. The display 72 may be of a desktop type, or may be configured by a tablet terminal or the like capable of wireless communication with the main body of the console device 70 . The display 72 is an example of a display section.

The input interface 73 is used for inputting subject information, setting X-ray conditions, and inputting various command signals. The subject information includes, for example, subject ID, subject name, date of birth, age, weight, sex, examination site, and the like. Note that the subject information may include the height of the subject. The input interface 73 is, for example, a trackball for instructing movement of the C-arm 9 and the Ω-arm 19, setting a region of interest (ROI), a switch button, a mouse, a keyboard, and performing input operations by touching the operation surface. It is realized by a touch pad, a touch panel display in which a display screen and a touch pad are integrated, or the like. The input interface 73 is connected to the processing circuit 74 , converts an input operation received from the operator into an electrical signal, and outputs the electrical signal to the processing circuit 74 . Also, the input interface 73 may be composed of a tablet terminal or the like capable of wireless communication with the main body of the console device 70 . Also, in this specification, the input interface 73 is not limited to having physical operation parts such as a mouse and a keyboard. For example, the input interface 73 also includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to the processing circuit 74. .

By calling and executing the programs in the memory 71, the processing circuit 74 performs a system control function 741, a drive control function 742, an imaging control function 743, an image processing function 744, a data creation function 745, and a learning control function 746 corresponding to the programs. , an execution function 747 , an image generation function 748 and a display control function 749 . Note that the term "processor" used in the description of the processing circuit 74 means, for example, a CPU, a GPU, an application specific integrated circuit (ASIC), a programmable logic device or the like. Programmable logic devices include, for example, Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and Field Programmable Gate Arrays (FPGAs). Instead of calling and executing a program in memory 71, such a processor may be configured to incorporate the program directly into the circuitry of the processor. A program here includes a trained model. That is, the processing circuit 74 may be configured to call and execute the learned model from the memory 71, or may be configured to implement the learned model as a circuit such as ASIC or FPGA. 2, a single processing circuit 74 includes a system control function 741, a drive control function 742, an imaging control function 743, an image processing function 744, a data creation function 745, a learning control function 746, an execution function 747, and an image generation function. Although the function 748 and the display control function 749 have been described as being implemented, a processing circuit may be configured by combining a plurality of independent processors, and each function may be implemented by executing a program by each processor. . A system control function 741, a drive control function 742, an imaging control function 743, an image processing function 744, a data creation function 745, a learning control function 746, an execution function 747, an image generation function 748, and a display control function 749 are each a system control function. They may also be called circuits, drive control circuit, imaging control circuit, image processing circuit, data generation circuit, learning control circuit, execution circuit, image generation circuit, and display control circuit, and may be implemented as individual hardware circuits.

The system control function 741 , for example, temporarily stores information such as a command signal by an operator input from the input interface 73 and various initial setting conditions, and then transmits this information to each processing function of the processing circuit 74 .

The drive control function 742 controls the drive section 27 and the bed 25 using, for example, information on driving the C-arm 9 , the Ω-arm 19 and the tabletop 24 input from the input interface 73 . For example, the drive control function 742 controls movement and rotation of the C-arm 9 and Ω-arm 19 and movement and tilt of the top plate 24 .

The imaging control function 743 , for example, reads information from the system control function 741 and controls X-ray conditions such as tube voltage, tube current, and irradiation time in the high voltage generator 3 . X-ray conditions may include the product of tube current and irradiation time (mAs).

The image processing function 744 performs image processing such as filtering processing on the projection data in the memory 71 to generate an X-ray image, and stores the X-ray image in the memory 71 . Here, the X-ray image based on the output of the first X-ray detector 7 is called the first X-ray image, and the X-ray image based on the output of the second X-ray detector 17 is called the second X-ray image. The first X-ray image and the second X-ray image are taken at two different imaging angles. Furthermore, the image processing function 744 performs synthesis processing, subtraction processing, etc. on the obtained multiple X-ray image data, and stores the obtained X-ray image in the memory 71 . An X-ray image obtained by subtracting an X-ray image obtained without using a contrast agent from an X-ray image obtained using a contrast agent is a contrast-enhanced vascular X-ray image or DSA (Digital Subtraction Angiography) image.

The data creation function 745 creates learning data including a three-dimensional medical image of the aneurysm, an X-ray image of the aneurysm, and an index related to the coil filling factor for the aneurysm. In this embodiment, as described above, the coil filling factor is used as an index for the coil filling factor. Such learning data may be created, for example, for each vessel with an aneurysm (anterior communicating artery, internal carotid artery, middle cerebral artery, basilar artery, vertebral artery, etc.) from the viewpoint of improving accuracy. It is preferable to create for each imaging angle of the C-arm 9 or Ω-arm 19 . In the case of a biplane structure, such a data creation function 745 generates a three-dimensional medical image, a first X-ray image and a second X-ray image as X-ray images, and a coil filling factor, as shown in FIG. Create training data containing The three-dimensional medical image (volume data) may be, for example, a CT image captured by the X-ray CT device 200 or an MRI image captured by the MRI device 210 . The first X-ray image and the second X-ray image are taken at two different imaging angles. The data creation function 745 may set an index related to the coil filling factor corresponding to the X-ray image to the learning data by the operation of the input interface 73 by the operator. In addition, in the case of a single-plane structure, the data creation function 745 creates learning data including a three-dimensional medical image, a first X-ray image, and a coil filling factor, for example, as shown in FIG. Training data is created that includes the medical image, the second X-ray image, and the coil filling factor. Note that the case of the single plane structure here includes the case where the X-ray irradiation of the first X-ray tube 5 or the second X-ray tube 15 of the biplane structure is stopped. In any case, the data creation function 745 outputs the created learning data to the memory 71 . Note that the data creation function 745 is an example of a data creation unit.

The learning control function 746, as shown in FIG. A trained model M2b, which is a model, is created. The learning control function 746 writes the biplane learned model M2b (first learned model) to the memory 71 . As a result, the learned model M2b is stored in the memory 71. FIG. The learned model M2b relates to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray images (first X-ray image and second X-ray image) of the aneurysm. It is functionalized to output the index. In this example, the learning data includes input data including three-dimensional medical images and X-ray images (first and second X-ray images) and output data including coil filling factors. The X-ray images used for biplane learning data, the machine learning model M1b, and the trained model M2b are both the first X-ray image and the second X-ray image captured in parallel at two different imaging angles. be.

In addition, as shown in FIG. 6, the learning control function 746 causes the single-plane machine learning model M1s to perform machine learning based on the learning data obtained by the data creation function 745. A trained model M2s, which is a machine learning model, is created. The learning control function 746 writes the single-plane trained model M2s (second trained model) to the memory 71 . As a result, the learned model M2s is stored in the memory 71. FIG. The learned model M2s relates to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray image (first X-ray image or second X-ray image) of the aneurysm. It is functionalized to output the index. In this example, the learning data has input data including a three-dimensional medical image and an X-ray image (first X-ray image or second X-ray image) and output data including a coil filling factor. The X-ray images used for the single-plane learning data, the machine learning model M1s, and the learned model M2s are at least one of a first X-ray image and a second X-ray image captured in parallel at two imaging angles different from each other. is. Also, the X-ray images used for the single-plane learning data, the machine learning model M1s, and the trained model M2s are a series of X-ray images captured by irregularly switching the imaging angle. The situation in which the imaging angle is irregularly switched is, for example, when coil embolization is performed by stopping the X-ray irradiation of one of the first X-ray tube 5 and the second X-ray tube 15 having a biplane structure, There are situations in which the ray tube 5 and the second X-ray tube 15 are used interchangeably. Also, when coil embolization is performed by stopping the X-ray irradiation of one of the first X-ray tube 5 and the second X-ray tube 15, the angle of X-ray irradiation by the X-ray tube is changed. I have a situation. Alternatively, when performing coil embolization using a single-plane X-ray tube, there is a situation in which the angle of X-ray irradiation by the X-ray tube is changed.

In any case, the machine learning models M1b and M1s use a three-dimensional medical image and an X-ray image as input, and output the coil filling factor (indicator related to) for the aneurysm in the X-ray image. A plurality of functions are synthesized. is a parameterized composite function. A parameterized composite function is defined by a combination of multiple adjustable functions and parameters. The machine learning model according to the present embodiment may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multi-layered network model (hereinafter referred to as a multi-layered network). The trained models M2b and M2s using a multi-layered network include an input layer for inputting a three-dimensional medical image and an X-ray image, an output layer for outputting an index related to the coil filling factor in the X-ray image, an input layer and an output layer. and at least one intermediate layer provided between. The input layer may have, for example, units corresponding to the total number of voxels of the three-dimensional medical image and the number of pixels of the X-ray image. The output layer may, for example, have one unit for outputting an indication about the coil fill factor. Also, the input layer may input incidental information indicating the type (material and shape) of the coil in addition to the three-dimensional medical image and the X-ray image. The trained model M2 is assumed to be used as a program module that is part of artificial intelligence software. As such a multi-layered network, for example, a deep neural network (DNN), which is a multi-layered neural network targeted for deep learning, is used. As the DNN, for example, a recurrent neural network (RNN) may be used for moving images, and a convolutional neural network (CNN) may be used for still images. A RNN may include a Long Short-Term Memory (LSTM). The following description mainly takes as an example the case where the image of the specified region in the X-ray image is a moving image. The above description of the multilayered network also applies to all machine learning models and trained models below.

The execution function 747 acquires from the memory 71 a three-dimensional medical image of the subject's aneurysm captured by the medical image diagnostic apparatus, and an X-ray image of the subject's aneurysm captured by the X-ray diagnostic apparatus 1 ( the first X-ray image and/or the second X-ray image) are obtained from memory 71 . At this time, as shown in FIG. 7, for example, the execution function 747 performs Then, the coil filling factor (indicator relating to it) for the aneurysm on the acquired X-ray image is output. Alternatively, the execution function 747, for example, as shown in FIG. 8, performs , outputs (indices relating to) the coil filling factor for the aneurysm on the acquired X-ray image. Note that the X-ray image is at least one of the first X-ray image and the second X-ray image captured in parallel at two imaging angles different from each other. The trained models executable by the execution function 747 are a trained model M2b (first trained model) when the X-ray images are the first X-ray image and the second X-ray image, and a trained model M2b (first trained model) when the X-ray image is the first X-ray image or a trained model M2s (second trained model) in the case of the second X-ray image. Here, as shown in FIG. 9, the execution function 747 selects the learned model M2b when the acquired X-ray images are the first X-ray image and the second X-ray image, and is the first X-ray image or the second X-ray image, the trained model M2s may be selected. Also, the execution function 747 outputs an index related to the coil filling rate based on the selected learned model M2b or M2s, the acquired X-ray image, and the acquired three-dimensional medical image. good too. That is, the execution function 747 may switch and execute the trained models M2b and M2s. Alternatively, execution function 747 may execute trained models M2b and M2s in parallel. The execution function 747 is an example of a first acquisition unit, a second acquisition unit, and a processing unit.

The image generation function 748 generates images 72a, 72b, 72c showing the coil filling status based on the coil filling rate (indices related to) output by the execution function 747, as shown in FIG. The generated images 72a, 72b, and 72c are displayed on the display 72 by the display control function 749 together with the first X-ray image g1 and the second X-ray image g2. However, the images 72a, 72b, and 72c do not have to be displayed on the display 72 together with the first X-ray image g1 and the second X-ray image g2, and may be displayed on a display different from the display that displays the X-ray images. Here, the coil filling state is a state based on an index related to the coil filling rate, and may be, for example, a numerical value of the index, a graph corresponding to the index, or the number of filled or remaining coils corresponding to the index. For example, if the indicator is the coil fill factor, the image 72a shows the coil fill status in terms of the current coil fill factor values. Image 72b shows the coil fill status with a bar graph showing the current coil fill (7%) relative to the target coil fill (20%). The image 72c shows the remaining coils obtained by subtracting the current number of filled coils (1) from the number of coils (3) corresponding to the target coil filling rate, such that the coil filling status is "more". A message indicating the number of coils (2) and a bar indicating the number of each coil (3 in total, 2 remaining, 1 already filled) indicate the coil filling status. The image showing the coil filling status is not limited to these examples, and any image presenting the coil filling status based on at least the index of the current coil filling rate can be used as appropriate. For example, as an image showing the coil filling status, a graph of another type such as a pie chart may be used instead of the bar graph described above. Also, the image generation function 748 does not need to generate all the images 72a, 72b, and 72c, but only needs to generate at least one image. Image generator function 748 is an example of an image generator.

The display control function 749 reads a signal from the system control function 741, acquires desired medical image data from the memory 71, and displays the data on the display 72, for example. Also, the display control function 749 causes the display 72 to display images 72 a to 72 c generated by the image generation function 748 . The display control function 749 is an example of a display control unit.

The network interface 76 is a circuit for connecting the console device 70 to the network Nw and communicating with other devices such as the medical information processing device 90, the X-ray CT device 200, the MRI device 210, and the like. As the network interface 76, for example, a network interface card (NIC) can be used. In the following description, the description that the network interface 76 intervenes in communication with other devices is omitted.

The memory 71 , display 72 , input interface 73 , data creation function 745 , learning control function 746 , execution function 747 , image generation function 748 and display control function 749 of the processing circuit 74 constitute a medical information processing apparatus 77 . . The medical information processing device 77 may further include an image processing function 744 . The medical information processing apparatus 77 may be built in the X-ray diagnostic apparatus 1 or may be provided outside the X-ray diagnostic apparatus 1 as a device separate from the X-ray diagnostic apparatus 1 .

On the other hand, the medical information processing apparatus 90 includes a memory 91, a display 92, an input interface 93, a processing circuit 94 and a network interface 96, as shown in FIG.

The memory 91 includes memory bodies such as ROM, RAM, HDD, and image memory for recording electrical information, and peripheral circuits such as memory controllers and memory interfaces attached to the memory bodies. The memory 91 stores, for example, a program executed by the processing circuit 94, a medical image generated by the processing circuit 94, a machine learning model, a trained model, data used for processing by the processing circuit 94, various tables, data in process, and Data and the like after processing are stored. Medical images include, for example, CT images (volume data), MRI images (volume data), X-ray images (X-ray fluoroscopic images), contrast-enhanced vascular X-ray images (DSA images), and superimposed images. The trained model is functionalized to output an index related to the coil filling factor for the aneurysm based on the three-dimensional medical image (volume data) of the aneurysm and the X-ray image of the aneurysm. It is The program, for example, provides a computer with a first acquisition function of acquiring a three-dimensional medical image of an aneurysm of a subject captured by a medical image diagnostic apparatus, an X-ray image of an aneurysm of the subject captured by an X-ray diagnostic apparatus, and A second acquisition function for acquiring a line image, a function to output an index related to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray image of the aneurysm. Based on the attached trained model, from the acquired three-dimensional medical image, the acquired X-ray image, and the learned model, an index related to the coil filling rate for the aneurysm on the acquired X-ray image and an image generation function for generating an image showing the coil filling status based on the output index. Supplementally, as such a program, for example, a program that is installed in the computer in advance from a network or a non-transitory computer-readable storage medium and causes the computer to implement each function of the medical information processing apparatus 90 is used. .

The display 92 is composed of a display body that displays various information such as medical images, an internal circuit that supplies display signals to the display body, and peripheral circuits such as connectors and cables that connect the display body and the internal circuits. there is The internal circuit superimposes incidental information such as subject information and projection data generation conditions on the image data supplied from the processing circuit 94 to generate display data. Converts and displays on the display itself. For example, the display 92 outputs a medical image emphasized by the processing circuit 94, a GUI for accepting various operations from the operator, and the like. For example, the display 92 is a liquid crystal display or CRT display. Also, the display 92 is an example of a display unit. The display 92 may be of a desktop type, or may be configured by a tablet terminal or the like capable of wireless communication with the main body of the medical information processing apparatus 90 . The display 92 is an example of a display section.

The input interface 93 inputs object information and various command signals. The subject information includes, for example, subject ID, subject name, date of birth, age, weight, sex, examination site, and the like. Note that the subject information may include the height of the subject. The input interface 93 includes, for example, a trackball, a switch button, a mouse, a keyboard, and an input operation by touching an operation surface for instructing medical information processing such as machine learning and image processing, setting a region of interest (ROI), and the like. It is realized by a touch pad, a touch panel display in which a display screen and a touch pad are integrated, or the like. The input interface 93 is connected to the processing circuit 94 , converts an input operation received from the operator into an electrical signal, and outputs the electrical signal to the processing circuit 94 . Also, the input interface 93 may be composed of a tablet terminal or the like capable of wireless communication with the main body of the medical information processing apparatus 90 . Also, in this specification, the input interface 93 is not limited to having physical operation parts such as a mouse and a keyboard. For example, the input interface 93 also includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to the processing circuit 94. .

The processing circuit 94 is a processor that implements a data creation function 945, a learning control function 946, an execution function 947, an image generation function 948, and a display control function 949 corresponding to the programs by calling and executing the programs in the memory 91. . The term "processor" used in the description of the processing circuit 94 means, for example, a CPU, a GPU, an application specific integrated circuit (ASIC), a programmable logic device or the like. Programmable logic devices include, for example, Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and Field Programmable Gate Arrays (FPGAs). Instead of calling and executing a program in memory 91, such a processor may be configured to incorporate the program directly into the circuitry of the processor. A program here includes a trained model. That is, the processing circuit 74 may have a configuration in which the trained model is implemented as a circuit such as ASIC or FPGA. 11, the data creation function 945, the learning control function 946, the execution function 947, the image generation function 948, and the display control function 949 are realized by the single processing circuit 94. Such processors may be combined to form a processing circuit, and each function may be realized by each processor executing a program. Also, the data creation function 945, the learning control function 946, the execution function 947, the image generation function 948, and the display control function 949 may be called a data creation circuit, a learning control circuit, an execution circuit, an image generation circuit, and a display control circuit, respectively. , may be implemented as separate hardware circuits. Also, the data creation function 945, the learning control function 946, the execution function 947, the image generation function 948, and the display control function 949 in the medical information processing apparatus 90 correspond to the data creation function 745 and the learning control function 746 in the X-ray diagnostic apparatus 1. , execution function 747 , image generation function 748 and display control function 749 . For this reason, the following descriptions of the data creation function 945, the learning control function 946, the execution function 947, the image generation function 948, and the display control function 949 will be described by omitting overlapping portions as appropriate.

The data creation function 945 includes a three-dimensional medical image of the aneurysm, an X-ray image of the aneurysm, and an index related to the coil filling rate for the aneurysm (e.g., coil filling rate) for learning. Create data. Such learning data may be created, for example, for each vessel with an aneurysm (anterior communicating artery, internal carotid artery, middle cerebral artery, basilar artery, vertebral artery, etc.) from the viewpoint of improving accuracy. It is preferable to create for each imaging angle of the C-arm 9 or Ω-arm 19 . In the case of a biplane structure, such a data creation function 945 includes, for example, a three-dimensional medical image, a first X-ray image and a second X-ray image as X-ray images, and a coil filling factor, as shown in FIG. Create training data containing The three-dimensional medical image (volume data) may be, for example, a CT image captured by the X-ray CT device 200 or an MRI image captured by the MRI device 210 . The first X-ray image and the second X-ray image are taken at two different imaging angles. The data creation function 945 may set an index related to the coil filling factor corresponding to the X-ray image to the learning data by the operation of the input interface 93 by the operator. In addition, in the case of a single-plane structure, the data creation function 945 creates learning data including a three-dimensional medical image, a first X-ray image, and a coil filling factor, for example, as shown in FIG. Learning data is created that includes the original medical image, the second X-ray image, and the coil filling factor. Note that the case of the single plane structure here includes the case where the X-ray irradiation of the first X-ray tube 5 or the second X-ray tube 15 of the biplane structure is stopped. In any case, the data creation function 945 outputs the created learning data to the memory 91 . Note that the data creation function 945 is an example of a data creation unit.

The learning control function 946, as shown in FIG. A trained model M2b, which is a learning model, is created. The learning control function 946 writes the biplane learned model M2b (first learned model) to the memory 91 . As a result, the learned model M2b is stored in the memory 91. FIG. The learned model M2b relates to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray images (first X-ray image and second X-ray image) of the aneurysm. It is functionalized to output the index. In this case, the learning data includes input data including three-dimensional medical images and X-ray images (first and second X-ray images) and output data including coil filling factors. The X-ray images used for biplane learning data, the machine learning model M1b, and the trained model M2b are both the first X-ray image and the second X-ray image captured in parallel at two different imaging angles. be.

In addition, as shown in FIG. 6, the learning control function 946 causes the single-plane machine learning model M1s to perform machine learning based on the learning data obtained by the data creation function 945. create a trained model M2s, which is a machine learning model of The learning control function 946 writes the single-plane trained model M2s (second trained model) to the memory 91 . As a result, the learned model M2s is stored in the memory 91. FIG. The learned model M2s relates to the coil filling factor for the aneurysm based on the three-dimensional medical image of the aneurysm and the X-ray image (first X-ray image or second X-ray image) of the aneurysm. It is functionalized to output the index. In this example, the learning data has input data including a three-dimensional medical image and an X-ray image (first X-ray image or second X-ray image) and output data including a coil filling factor. The X-ray images used for the single-plane learning data, the machine learning model M1s, and the learned model M2s are at least one of a first X-ray image and a second X-ray image captured in parallel at two imaging angles different from each other. is. Also, the X-ray images used for the single-plane learning data, the machine learning model M1s, and the trained model M2s are a series of X-ray images captured by irregularly switching the imaging angle. The situation in which the imaging angle is irregularly switched is as described above.

In any case, the machine learning models M1b and M1s use a three-dimensional medical image and an X-ray image as input, and output the coil filling factor (indicator related to) for the aneurysm in the X-ray image. A plurality of functions are synthesized. is a parameterized composite function. Composite functions with parameters are described above.

The execution function 947 acquires from the memory 91 a three-dimensional medical image of the subject's aneurysm captured by the medical image diagnostic apparatus, and an X-ray image of the subject's aneurysm captured by the X-ray diagnostic apparatus 1 ( the first X-ray image and/or the second X-ray image) are obtained from memory 91 . At this time, the execution function 947, for example, as shown in FIG. Based on this, the coil filling factor for the aneurysm on the acquired X-ray image is output. Alternatively, the execution function 947, for example, as shown in FIG. 8, performs to output the coil filling factor for the aneurysm on the acquired X-ray image. Note that the X-ray image is at least one of the first X-ray image and the second X-ray image captured in parallel at two imaging angles different from each other. The trained models that can be executed by the execution function 947 are a trained model M2b (first trained model) when the X-ray images are the first X-ray image and the second X-ray image, and a or a trained model M2s (second trained model) in the case of the second X-ray image. Here, as shown in FIG. 9, the execution function 947 selects the learned model M2b when the acquired X-ray images are the first X-ray image and the second X-ray image, and The trained model M2s may be selected if the image is the first X-ray image or the second X-ray image. Also, the execution function 947 outputs (an index relating to) the coil filling factor based on the selected trained model M2b or M2s, the acquired X-ray image, and the acquired three-dimensional medical image. can be That is, the execution function 947 may switch and execute the trained models M2b and M2s. Alternatively, execution function 947 may execute trained models M2b and M2s in parallel. Execution function 947 is an example of a first acquisition unit, a second acquisition unit, and a processing unit.

The image generation function 948 generates images 92a, 92b, 92c showing the coil filling status based on the index (eg, coil filling factor) output by the execution function 947, as shown in FIG. Details of the images 92a, 92b, and 92c are the same as those of the images 72a, 72b, and 72c described above. The image generation function 948 does not have to generate all the images 92a, 92b, 92c, but only needs to generate at least one image. The image generator function 948 is an example of an image generator.

The display control function 949 reads a signal from the input interface 93, acquires desired medical image data from the memory 91, and displays the data on the display 92, for example. Also, the display control function 949 causes the display 72 to display the images 92 a to 92 c generated by the image generation function 948 . The display control function 949 is an example of a display control unit.

The network interface 96 is a circuit for connecting the medical information processing apparatus 90 to the network Nw and communicating with other apparatuses such as the X-ray diagnostic apparatus 1, the X-ray CT apparatus 200, the MRI apparatus 210, and the like. As the network interface 96, for example, a network interface card (NIC) can be used. In the following description, the description that the network interface 96 intervenes in communication with other devices is omitted.

The data creation function 945, the learning control function 946, the execution function 947, the image generation function 948, and the display control function 949 in the medical information processing apparatus 90 correspond to the data creation function 745 and the learning control function 746 in the X-ray diagnostic apparatus 1. , execution function 747 , image generation function 748 and display control function 749 . That is, as the medical information processing system, either the functions 945 to 949 in the medical information processing apparatus 90 or the functions 745 to 749 in the X-ray diagnostic apparatus 1 are executed. Note that the processing circuit 94 of the medical information processing apparatus 90 may further implement an image processing function similar to the image processing function 744 within the X-ray diagnostic apparatus 1 .

The X-ray CT apparatus 200 is an example of a medical image diagnostic apparatus that captures a medical image of the subject P. As shown in FIG. The X-ray CT apparatus 200 performs CT imaging of the subject P and transmits the obtained CT image (volume data) to the X-ray diagnostic apparatus 1 or the medical information processing apparatus 90 .

The MRI apparatus 210 is another example of a medical image diagnostic apparatus that captures medical images of the subject P. FIG. The MRI apparatus 210 performs MRI imaging of the subject P and transmits the obtained MRI image (volume data) to the X-ray diagnostic apparatus 1 or the medical information processing apparatus 90 .

Next, the operation of the medical information processing apparatus configured as described above will be described with reference to the flowchart of FIG. The processing circuit 74 of the X-ray diagnostic apparatus 1 and the processing circuit 94 of the medical information processing apparatus 90 perform substantially the same operations with respect to the data creation function, learning control function, execution function, image generation function, and display control function. do. Accordingly, from the viewpoint of avoiding repetition of redundant words and facilitating understanding, in the following description, the processing circuit 94 of the medical information processing apparatus 90 will be used as a representative example of the operation of each function. The description of such representative examples can be applied to the description of the operation of the processing circuit 74 of the X-ray diagnostic apparatus 1 by appropriately replacing the device names and reference numerals. This also applies to each of the following embodiments. It is also assumed that the memory 91 of the medical information processing apparatus 90 stores learned models M2b and M2s. Further, the following description will be given by exemplifying a case where the index related to the coil filling rate is the coil filling rate.

First, the X-ray CT apparatus 200 performs CT imaging of the aneurysm of the subject P and transmits the obtained three-dimensional CT image (volume data) to the medical information processing apparatus 90 . The processing circuit 94 of the medical information processing apparatus 90 stores the received CT image in the memory 91 . A three-dimensional MRI image (volume data) transmitted by the MRI apparatus 210 may be stored in the memory 91 instead of the CT image. In the following description, a case where a CT image is saved will be described as a typical example.

In step ST10, the processing circuit 94 of the medical information processing apparatus 90 reads a three-dimensional CT image (volume data) as a three-dimensional medical image from the memory 91 according to the operation of the input interface 93 by the operator. As a result, a three-dimensional medical image of the aneurysm of the subject is obtained. This three-dimensional medical image is an image before performing coil embolization, and the aneurysm is not filled with a coil.

In step ST20, the processing circuit 74 of the X-ray diagnostic apparatus 1 controls the imaging device 10 according to the operation of the input interface 73 by the operator to adjust the imaging angles of the C arm 9 and the Ω arm 19, Start fluoroscopic imaging. As a result, an X-ray image of the subject P is obtained as a moving image. This X-ray image is, for example, a first X-ray image and a second X-ray image, and is transmitted from the X-ray diagnostic apparatus 1 to the medical information processing apparatus 90 during X-ray fluoroscopic imaging. When coil embolization is performed by stopping the X-ray irradiation of one of the first X-ray tube 5 and the second X-ray tube 15, the transmitted X-ray image is the first X-ray image or the second X-ray image. becomes. In any case, the processing circuit 94 of the medical information processing apparatus 90 stores the received X-ray image in the memory 91 and displays it on the display 92 . Under this X-ray fluoroscopy, coil embolization is started. In coil embolization, an operator visually recognizes an X-ray image displayed in real time while inserting a catheter and a guide wire into a blood vessel to advance it to an aneurysm, thereby filling the aneurysm with a coil. It should be noted that the display in real time does not strictly mean the process of displaying the images at the moment the images are captured. This means a process of sequentially displaying the X-ray images on the device 90 side.

After step ST20, the processing circuit 94 of the medical information processing apparatus 90 acquires an X-ray image (first X-ray image and/or second X-ray image) from the memory 71 in step ST30.
Further, the processing circuit 94, based on the three-dimensional medical image acquired in step ST10, the acquired X-ray image (the first X-ray image and/or the second X-ray image), and the learned models M2b and M2s, A coil filling factor for the aneurysm on the acquired X-ray image is output.

Supplementally, in coil embolization, when X-ray irradiation is performed from both the first X-ray tube 5 and the second X-ray tube 15, as shown in FIG. A line image is acquired, and a learned model M2b for biplane is selected as a learned model. At this time, as shown in FIG. 9, the single-plane trained model M2s may be executed in parallel.

When performing coil embolization by stopping X-ray irradiation from one of the first X-ray tube 5 and the second X-ray tube 15, as shown in FIG. A line image is acquired, and a trained model M2s for single plane is selected as a trained model. In this case, the X-ray images are a series of X-ray images captured by irregularly switching the imaging angle.

In any case, the processing circuit 94 executes the learned models M2b and M2s selected according to the number of X-ray images, and outputs the coil filling factor for the aneurysms on the X-ray images.

After step ST30, in step ST40, the processing circuit 94 generates, for example, an image 92b showing the coil filling status based on the output coil filling rate.

After step ST40, in step ST50, the processing circuit 94 causes the display 92 to display the generated image 92b together with the first X-ray image g1 and the second X-ray image g2. Therefore, the operator proceeds with coil filling while visually recognizing the image 92b indicating the coil filling status displayed in real time together with the first X-ray image g1 and the second X-ray image g2. In this real-time display, as described above, in parallel with the process of sequentially acquiring X-ray images on the X-ray diagnostic apparatus 1 side, the medical information processing apparatus 90 displays the coil filling status generated from the X-ray images. This means a process in which the indicated images are sequentially displayed.

After step ST50, in step ST60, the operator visually checks the display 92 to determine whether or not the coil filling is completed. If the result of this determination is No (ST60: No), the process returns to step ST30, and the operations of steps ST30 to ST60 are continued. On the other hand, as a result of the determination in step ST60, if the coil filling is completed (ST60: Yes), coil embolization is terminated. As a result, the inside of the aneurysm can be thrombosed to prevent rupture of the aneurysm.

As described above, according to the first embodiment, a three-dimensional medical image of an aneurysm in a subject is captured by a medical image diagnostic apparatus. An X-ray image of the aneurysm of the subject is acquired by an X-ray diagnostic apparatus. Based on the learned model functioned to output an index related to the coil filling factor for the aneurysm from the three-dimensional medical image of the aneurysm and the X-ray image of the aneurysm, the acquired An index related to the coil filling factor for the aneurysm on the acquired X-ray image is output from the acquired three-dimensional medical image, the acquired X-ray image, and the learned model. An image showing the coil filling status is generated based on the output index. In this way, with a configuration that generates a screen showing the coil filling status based on the index related to the coil filling rate, it is possible to assist the operator in coil embolization using the coil filling rate.

Supplementally, in the X-ray image of the X-ray diagnostic apparatus 1, the degree of coil overlap is difficult to visually recognize, but there is a difference in the X-ray image data. For this reason, an index related to the coil filling rate is output from the three-dimensional medical image (shape and volume of the aneurysm lumen) and changes in coil shape and density on the X-ray image, and the coil filling status is displayed according to the index. do. In this way, coil embolization can be assisted by displaying the coil filling status.

Further, according to the first embodiment, the X-ray images may be a first X-ray image and a second X-ray image that are imaged in parallel at two imaging angles different from each other. In this case, an index related to the coil filling factor is obtained based on the first X-ray image and the second X-ray image at two imaging angles, so in addition to the effects described above, the accuracy of the obtained index can be improved.

Further, according to the first embodiment, the X-ray image may be a series of X-ray images captured by irregularly switching the imaging angle. In this case, since a series of X-ray images are taken by switching the imaging angle as necessary, the exposure dose to the subject can be reduced compared to when using the first X-ray image and the second X-ray image taken in parallel. can.

Further, according to the first embodiment, the X-ray image is at least one of the first X-ray image and the second X-ray image captured in parallel at two imaging angles different from each other. The trained models include a first trained model when the X-ray images are the first X-ray image and the second X-ray image, and a second trained model when the X-ray image is the first X-ray image or the second X-ray image. Includes ready-made models. Further, when the acquired X-ray images are the first X-ray image and the second X-ray image, the first trained model is selected, and the acquired X-ray image is the first X-ray image or the second X-ray image. In some cases, a second trained model may be selected, and an index may be output based on the selected trained model, the acquired X-ray image, and the acquired three-dimensional medical image. In such a case, in the middle of coil embolization, when obtaining an index with high accuracy using the first X-ray image and the second X-ray image, and using the first X-ray image or the second X-ray image, the exposure dose is calculated. It is possible to switch between when and when to reduce.

<Second embodiment>
Next, a medical information processing apparatus according to a second embodiment will be described. In the following description, the same reference numerals are assigned to the same parts as in the above-described drawings, and detailed description thereof will be omitted, and mainly different parts will be described. This also applies to each of the following embodiments.

The second embodiment relates to creation of learning data and learning of a machine learning model corresponding to the first embodiment.

Along with this, the processing circuit 94 executes the operations of the data creation function 945 and the learning control function 946 described above, as shown in FIGS.

Other configurations are similar to those of the first embodiment.

Next, operation of the medical information processing apparatus according to the second embodiment will be described. The following description will be given in the order of operations related to creation of learning data (FIGS. 3 and 4) and operations related to learning control of the machine learning model (FIGS. 5 and 6). In the following description, as in the case described above, the coil filling rate is used as an index for the coil filling rate.

(Operation related to creation of learning data)
The processing circuit 94 of the medical information processing apparatus 90 is operated by the operator on the input interface 93 to display the three-dimensional medical image of the aneurysm when coil embolization was performed in the past on the X-ray CT apparatus 200 or the MRI apparatus 210. request. Thereby, the processing circuit 94 stores the three-dimensional medical image received from the X-ray CT apparatus 200 or the MRI apparatus 210 in the memory 91 . Similarly, the processing circuit 94 of the medical information processing apparatus 90 instructs the X-ray diagnostic apparatus 1 to perform coil embolization on the same aneurysm as the aneurysm in the three-dimensional medical image by the operation of the input interface 93 by the operator. A first X-ray image and a second X-ray image taken are requested. The processing circuit 94 stores the first X-ray image and the second X-ray image received from the X-ray diagnostic apparatus 1 in the memory 91 .

The three-dimensional medical image used to create the learning data is an image before coil embolization, and an aneurysm before coil filling is imaged. The first X-ray image and the second X-ray image used for creating learning data are moving images during coil embolization, and an aneurysm is imaged from before coil filling to after coil filling.

The processing circuit 94 causes the display 92 to display the first X-ray image and the second X-ray image in the memory 91 according to the operation of the operator. In addition, the processing circuit 94 calculates indices related to the coil filling rate for the aneurysm on the first X-ray image and the second X-ray image in accordance with the operation of the operator at a plurality of points in time between before coil filling and after coil filling. set. Note that the processing circuit 94 can pause, play, reverse play, frame-by-frame, and frame-rewind the X-ray image, which is a moving image, in accordance with the operation of the operator when setting the index related to the coil filling rate. is preferred.

Here, methods (a) to (d), for example, are appropriately available as methods for setting an index related to the coil filling factor.
(a) An index related to the coil filling factor estimated by the operator is set for the first X-ray image and the second X-ray image being displayed.

(b) setting an index related to the coil filling rate calculated based on the three-dimensional medical image before coil filling and the three-dimensional medical image after coil filling; When the indicator is the coil filling rate, the volume of the aneurysm lumen is calculated, for example, based on the number of voxels of the aneurysm lumen in the three-dimensional medical image before coil filling. Also, the volume of the filled coil is calculated based on the number of voxels of the coil filled in the lumen of the aneurysm in the three-dimensional medical image after coil filling. The coil fill factor is then calculated by dividing the volume of the coil by the volume of the aneurysm lumen. The coil filling factor may be calculated by dividing the number of voxels of the coil by the number of voxels of the aneurysm lumen. In addition to the three-dimensional image after coil filling, a three-dimensional medical image during coil filling may be used to calculate the coil filling rate during coil filling. Further, the calculated coil filling rate may be appropriately converted into another index such as an unfilling rate.

(c) This is a modification of (b) above, in which an index related to the coil filling rate calculated based only on the three-dimensional medical image after coil filling is set without using the three-dimensional medical image before coil filling. In this case, in the above (b), the coil filling rate can be similarly calculated by using the three-dimensional medical image after coil filling instead of the three-dimensional medical image before coil filling. Similarly, in addition to the three-dimensional image after coil filling, a three-dimensional medical image during coil filling may be used to calculate the coil filling rate during coil filling. Similarly, the calculated coil filling rate may be appropriately converted into another index such as an unfilling rate.

(d) Set an index related to the coil filling rate calculated based on the three-dimensional medical image before coil filling and the number of filled coils. When the indicator is the coil filling rate, the volume of the aneurysm lumen is calculated, for example, based on the number of voxels of the aneurysm lumen in the three-dimensional medical image before coil filling. Also, based on the number of coils filled during and after coil filling, the volume of the filled coils is calculated. The coil fill factor is then calculated by dividing the volume of the coil by the volume of the aneurysm lumen. The coil filling factor may be calculated by dividing the number of coils by the number of voxels in the aneurysm lumen. Similarly, the calculated coil filling rate may be appropriately converted into another index such as an unfilling rate.

In any case, it is assumed that setting of indices related to the coil filling rate based on the first X-ray image and the second X-ray image is completed at a plurality of time points between before coil filling and after coil filling. At this time, as shown in FIG. 3, the processing circuit 94 converts the three-dimensional medical image, the first X-ray image, the second X-ray image, and the coil filling factor (indices related to) to the learned model. Output as learning data for use. The learning data includes input side data, which are the three-dimensional medical image, the first X-ray image, and the second X-ray image, and output side data, which is the coil filling factor. Alternatively, the processing circuit 94 uses the three-dimensional medical image, the first X-ray image or the second X-ray image, and the coil filling factor (indices related to) in the trained model, as shown in FIG. output as learning data for The learning data includes input side data, which is a three-dimensional medical image, first X-ray image, or second X-ray image, and output side data, which is a coil filling factor. In any case, the processing circuit 94 stores the created learning data in the memory 91 . This completes the creation of learning data for one aneurysm subjected to coil embolization. Similarly, by repeatedly executing the above-described operations, the amount of learning data used for machine learning is created.

(Operation related to learning control)
The processing circuit 94 causes the machine learning models M1b and M1s to perform machine learning by sequentially using the amount of learning data used for machine learning, as shown in FIG. 5 or 6 . At this time, the machine learning models M1b and M1s output the coil filling factor (index related to) for the aneurysm on the X-ray image based on the three-dimensional medical image and the X-ray image that are input from the learning data. do. The processing circuit 94 also controls the machine learning models M1b and M1s so that the coil filling factors (indicators relating to them) output from the machine learning models M1b and M1s approach the relevant coil filling factors (indicators relating to them) in the learning data. are adjusted, and machine learning is performed on the machine learning models M1b and M1s.

Upon completion of machine learning, the processing circuit 94 creates learned models M2b and M2s as machine learning models M1b and M1s whose parameters and the like have been adjusted. The created trained models M2b and M2s are functioned to output an index related to the coil filling factor for the aneurysm on the X-ray image based on the medical three-dimensional image and the X-ray image. The processing circuit 94 stores the created trained models M2b and M2s in the memory 91 .

As described above, according to the second embodiment, in coil embolization, it is possible to create a trained model for outputting an index related to the coil filling rate from an X-ray image and data for learning thereof.

<Third Embodiment>
The third embodiment is a modification of the first and second embodiments, in which learning data is obtained from three-dimensional medical images instead of three-dimensional medical images in order to improve the efficiency of machine learning. It is configured to include shape and volume data of the aneurysm lumen that is used. Here, as the shape/volume data, for example, two-dimensional image data representing the contour shape of the aneurysm lumen extracted from the three-dimensional medical image according to the imaging angle of the X-ray image, and the two-dimensional image data extracted from the three-dimensional medical image. A volume value based on the number of voxels in the aneurysm lumen is available. In the same manner as described above, the case where the index related to the coil filling rate is the coil filling rate will be described as an example.

Along with this, as an example shown in FIG. 13, the data generation function 945 of the processing circuit 94 has the shape and volume of the aneurysm lumen obtained from the three-dimensional medical image instead of the three-dimensional medical image described above. Create training data containing data. The data creation function 945 creates shape/volume data of the lumen of the aneurysm by specifying the aneurysm from the three-dimensional medical image in the memory 91, for example, in response to the operation of the input interface 93 by the operator. do. As a result, the data creation function 945 generates the shape/volume data of the lumen of the aneurysm, the input side data that is the first X-ray image and the second X-ray image, and the output side data that is the coil filling factor. Output training data containing Although not shown, this is also applicable to single-plane learning data as shown in FIG. That is, the data creation function 945 outputs learning data including the shape/volume data of the aneurysm lumen, the input side data that is the first X-ray image, and the output side data that is the coil filling factor. may Similarly, the data creation function 945 outputs learning data including the shape/volume data of the aneurysm lumen, the input side data that is the second X-ray image described above, and the output side data that is the coil filling factor described above. You may The term "aneurysm lumen" may be read as "aneurysm".

Further, the learning control function 946 of the processing circuit 94 sequentially uses the learning data according to the present embodiment to perform machine learning on the machine learning models M1b and M1s. Create M2s. The created trained models M2b and M2s are functioned to output the coil filling factor for the aneurysm on the X-ray image based on the aneurysm lumen shape/volume data and the X-ray image. . The processing circuit 94 stores the created trained models M2b and M2s in the memory 91 .

Further, as shown in FIG. 14, the execution function 947 of the processing circuit 94 identifies an aneurysm from the three-dimensional medical image in the memory 91 in response to the operation of the input interface 93 by the operator, and thereby extracts the artery. Create shape and volume data of the aneurysm lumen. Further, the execution function 947 of the processing circuit 94 performs artery analysis on the first X-ray image and the second X-ray image based on the created shape/volume data, the first X-ray image and the second X-ray image, and the learned model. Output the coil filling factor for the aneurysm. Although not shown, this is also applicable to the trained model M2s for a single plane as shown in FIG. 8 or FIG. That is, the data creation function 945 outputs the above-described coil filling factor based on the aneurysm lumen shape/volume data, the above-described first X-ray image or second X-ray image, and the learned model M2s. good too.

Other configurations are similar to those of the first and second embodiments.

According to the configuration as described above, the learning data includes shape and volume data of the aneurysm lumen obtained from the three-dimensional medical image instead of the three-dimensional medical image. The number of layer units can be reduced. For example, the number of input layer units used for inputting 3D medical images (the number of voxels in a 3D medical image) is changed to the number of input layer units used for inputting shape/volume data (the number of pixels in 2D image data). and the number of volume values). Accordingly, it is possible to simplify the configurations of the machine learning models M1b and M1s, thereby improving the efficiency of machine learning.

Similarly, the learned models M2b and M2s, instead of the three-dimensional medical images, are based on the shape and volume data of the aneurysm lumen obtained from the three-dimensional medical images and the X-ray images, and are indices related to the coil filling rate. is output, the configuration of the learned models M2b and M2s can be simplified, and the speed of calculation of the coil filling factor can be increased. For example, in the trained models M2b and M2s, it can be expected to omit the extraction of the feature quantity corresponding to the shape/volume data from the three-dimensional medical image, so the calculation of the coil filling factor can be speeded up. Alternatively, in the trained models M2b and M2s, since the number of units in the input layer is reduced, simplification of the calculation of the intermediate layer can be expected, so the speed of calculation of the coil filling factor can be increased.

According to at least one of the embodiments described above, a three-dimensional medical image of an aneurysm in a subject is captured by a medical image diagnostic apparatus. An X-ray image of the aneurysm of the subject is acquired by an X-ray diagnostic apparatus. Based on the learned model functioned to output an index related to the coil filling factor for the aneurysm from the three-dimensional medical image in which the aneurysm is imaged and the X-ray image in which the aneurysm is imaged, the acquired An index related to the coil filling factor for the aneurysm on the acquired X-ray image is output from the acquired three-dimensional medical image, the acquired X-ray image, and the learned model. An image showing the coil filling status is generated based on the output index. In this way, with the configuration that generates a screen showing the coil filling status based on the coil filling rate, it is possible to assist the operator in coil embolization using the coil filling rate.

The term "processor" used in the above description is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an application specific integrated circuit (ASIC), a programmable logic device (e.g., Circuits such as Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), Field Programmable Gate Array (FPGA), etc. A processor is a memory circuit. The function is realized by reading and executing the program stored in.In addition, instead of storing the program in the memory circuit, it may be configured so that the program is directly incorporated in the circuit of the processor.In this case, the processor The function is realized by reading and executing the program embedded in the circuit.In addition, each processor in this embodiment is not limited to being configured as a single circuit for each processor, but may be configured as a plurality of independent circuits. 2 or 7 may be integrated into one processor to realize its function. .

It should be noted that although several embodiments of the invention have been described, these embodiments are provided by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

1 X-ray diagnostic device 3 High voltage generator 5 First X-ray tube 7 First X-ray detector 9 C arm 10 Imaging device 11 First collimator 15 Second X-ray tube 17 Second X-ray detector 19 Ω arm 21 Second collimator 24 Top plate 25 bed 27 drive unit 70 console device 71, 91 memory 72, 92 display 73, 93 input interface 74, 94 processing circuit 741 system control function 742 drive control function 743 photographing control function 744 image processing function 745, 945 data creation function 746,946 Learning control function 747,947 Execution function 748,948 Image generation function 749,949 Display control function 76,96 Network interface 90 Medical information processing apparatus M1b, M1s Machine learning model M2b, M2s Trained model g1, g2 X-ray Images 72a-72c, 92a-92c Images showing coil filling status

Claims (7)

a first acquisition unit that acquires a three-dimensional medical image of an aneurysm of a subject captured by a medical image diagnostic device;
a second acquisition unit that acquires an X-ray image of the aneurysm and the coil captured by an X-ray diagnostic apparatus during coil embolization in which the aneurysm of the subject is filled with the coil;
a display control unit for displaying the acquired X-ray image on a display in real time;
Based on a trained model functioned to output an index related to the coil filling factor for the aneurysm from a three-dimensional medical image of the aneurysm and an X-ray image of the aneurysm and the coil , A process of outputting, from the acquired three-dimensional medical image, the acquired X-ray image, and the trained model, an index related to the coil filling rate for the aneurysm in which the coil on the acquired X-ray image is being filled. Department and
an image generation unit that generates an image showing a coil filling situation based on the output index;
with
The index related to the coil filling rate is the coil filling rate, the coil unfilling rate, the number of filled coils among the number of coils corresponding to the target coil filling rate, or the number of filled coils subtracted from the number of coils. medical information processing apparatus , which is the remaining number of coils . 2. The medical information processing apparatus according to claim 1, wherein said X-ray images are a first X-ray image and a second X-ray image taken in parallel at two imaging angles different from each other. 2. The medical information processing apparatus according to claim 1, wherein said X-ray images are a series of X-ray images captured by switching imaging angles irregularly. The X-ray image is at least one of a first X-ray image and a second X-ray image captured in parallel at two imaging angles different from each other,
The trained model includes a first trained model when the X-ray images are the first X-ray image and the second X-ray image, and a first trained model when the X-ray image is the first X-ray image or the second X-ray image. a second trained model in some cases, and
The processing unit selects the first trained model when the acquired X-ray images are the first X-ray image and the second X-ray image, and the acquired X-ray image is the first X-ray image. selecting the second trained model if it is a line image or the second X-ray image, based on the selected trained model, the acquired X-ray image, and the acquired three-dimensional medical image; 2. The medical information processing apparatus according to claim 1, wherein the indicator is output by The coil filling factor is a value indicating the volume occupied by the coil with respect to the volume of the aneurysm,
The medical information processing apparatus according to any one of claims 1 to 4, wherein the coil unfilling rate is a value obtained by subtracting the coil filling rate from the volume of the aneurysm. a first acquisition unit that acquires a three-dimensional medical image of an aneurysm of a subject captured by a medical image diagnostic device;
a second acquisition unit that acquires an X-ray image of the aneurysm and the coil captured by an X-ray diagnostic apparatus during coil embolization in which the aneurysm of the subject is filled with the coil;
a display control unit for displaying the acquired X-ray image on a display in real time;
Based on a trained model functioned to output an index related to the coil filling factor for the aneurysm from a three-dimensional medical image of the aneurysm and an X-ray image of the aneurysm and the coil , A process of outputting, from the acquired three-dimensional medical image, the acquired X-ray image, and the trained model, an index related to the coil filling rate for the aneurysm in which the coil on the acquired X-ray image is being filled. Department and
an image generation unit that generates an image showing a coil filling situation based on the output index;
with
The index related to the coil filling rate is the coil filling rate, the coil unfilling rate, the number of filled coils among the number of coils corresponding to the target coil filling rate, or the number of filled coils subtracted from the number of coils. is the remaining number of coils, an X-ray diagnostic apparatus. to the computer,
a first acquisition function for acquiring a three-dimensional medical image of the subject's aneurysm captured by a medical imaging diagnostic device;
a second acquisition function for acquiring an X-ray image of the aneurysm and the coil captured by an X-ray diagnostic apparatus during coil embolization in which the aneurysm of the subject is filled with the coil ;
a display control function for displaying the acquired X-ray image on a display in real time;
Based on a trained model functioned to output an index related to the coil filling factor for the aneurysm from a three-dimensional medical image of the aneurysm and an X-ray image of the aneurysm and the coil , A process of outputting, from the acquired three-dimensional medical image, the acquired X-ray image, and the trained model, an index related to the coil filling rate for the aneurysm in which the coil on the acquired X-ray image is being filled. function,
an image generation function for generating an image showing the coil filling status based on the output index;
to realize
The index related to the coil filling rate is the coil filling rate, the coil unfilling rate, the number of filled coils among the number of coils corresponding to the target coil filling rate, or the number of filled coils subtracted from the number of coils. Program , which is the remaining number of coils .