JP2021185969A - Medical information processing device, x-ray diagnostic device, and program - Google Patents

Abstract

To enhance accuracy of composite display of a two-dimensional x-ray image and three-dimensional data.SOLUTION: A medical information processing device according to embodiment comprises a display control unit, a reception unit and a control unit. The display control unit performs control to display a composite image of an image based on three-dimensional data and a two-dimensional X-ray image. The reception unit receives an adjustment operation to adjust the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image, from a user having referred to the composite image. The control unit causes the three-dimensional data, two-dimensional X-ray image, and a result of the adjustment operation in a storage unit.SELECTED DRAWING: Figure 5

Description

The embodiments disclosed in the present specification and the like relate to a medical information processing device, an X-ray diagnostic device, and a program.

There is known a technique of collecting three-dimensional data on blood vessels and organs of a subject and synthesizing and displaying an image based on the three-dimensional data on an X-ray image collected in two dimensions. Further, when performing such a composite display, the position, orientation, size, etc. of the three-dimensional data with respect to the two-dimensional X-ray image are adjusted.

U.S. Patent Application Publication No. 2012/0130243

One of the problems to be solved by the embodiments disclosed in the present specification and the like is to improve the accuracy of the composite display of the two-dimensional X-ray image and the three-dimensional data. However, the problems solved by the embodiments disclosed in the present specification and the like are not limited to the above problems. The problem corresponding to each effect by each configuration shown in the embodiment described later can be positioned as another problem to be solved by the embodiment disclosed in the present specification and the like.

The medical information processing apparatus of the embodiment includes a display control unit, a reception unit, and a control unit. The display control unit displays a composite image in which an image based on 3D data and a 2D X-ray image are combined. The reception unit receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image from the user who has referred to the composite image. The control unit stores the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation in the storage unit.

FIG. 1 is a block diagram showing an example of the configuration of the medical information processing system according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of the X-ray diagnostic apparatus according to the first embodiment. FIG. 3 is a diagram for explaining acceptance of the adjustment operation according to the first embodiment. FIG. 4 is a flowchart for explaining a series of processes of the medical information processing apparatus according to the first embodiment. FIG. 5 is a diagram showing an example of a trained model generation process according to the first embodiment. FIG. 6 is a flowchart for explaining a series of processes of the medical information processing apparatus according to the first embodiment. FIG. 7 is a block diagram showing an example of the configuration of the X-ray diagnostic apparatus according to the second embodiment.

Hereinafter, embodiments of a medical information processing apparatus, an X-ray diagnostic apparatus, and a program will be described in detail with reference to the accompanying drawings.

(First Embodiment)
In the first embodiment, the medical information processing system 1 shown in FIG. 1 will be described as an example. For example, the medical information processing system 1 includes an X-ray diagnostic apparatus 10 and a medical information processing apparatus 30. Further, the X-ray diagnostic apparatus 10 and the medical information processing apparatus 30 are connected to each other via a network NW. Note that FIG. 1 is a block diagram showing an example of the configuration of the medical information processing system 1 according to the first embodiment.

The X-ray diagnostic apparatus 10 is an apparatus that collects X-ray images from the subject P1. For example, the X-ray diagnostic apparatus 10 collects two-dimensional X-ray images from the subject P1 over time while the endovascular treatment procedure for the subject P1 is being performed, and the collected two-dimensional X-ray images are used for medical purposes. It is sequentially transmitted to the information processing apparatus 30. The configuration of the X-ray diagnostic apparatus 10 will be described later.

The medical information processing apparatus 30 performs various processes described later based on the two-dimensional X-ray image collected by the X-ray diagnostic apparatus 10. For example, the medical information processing apparatus 30 has an input interface 31, a display 32, a memory 33, and a processing circuit 34, as shown in FIG.

The input interface 31 receives various input operations from the user, converts the received input operations into electric signals, and outputs the received input operations to the processing circuit 34. For example, the input interface 31 includes a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch pad for performing an input operation by touching an operation surface, a touch screen in which a display screen and a touch pad are integrated, and an optical sensor. It is realized by the non-contact input circuit, voice input circuit, etc. used. The input interface 31 may be composed of a tablet terminal or the like capable of wireless communication with the processing circuit 34. Further, the input interface 31 may be a circuit that accepts an input operation from the user by motion capture. As an example, the input interface 31 can receive the user's body movement, line of sight, and the like as an input operation by processing the signal acquired through the tracker and the image collected about the user. Further, the input interface 31 is not limited to the one provided with physical operation parts such as a mouse and a keyboard. For example, a processing circuit for an electric signal that receives an electric signal corresponding to an input operation from an external input device provided separately from the main body of the medical information processing device 30 and outputs this electric signal to the main body of the medical information processing device 30. It is included in the example of the input interface 31.

The display 32 displays various information. For example, the display 32 displays various medical images under the control of the processing circuit 34. As an example, the display 32 displays a composite image in which an image based on three-dimensional data and a two-dimensional X-ray image are combined. The display of the composite image on the display 32 will be described later. Further, for example, the display 32 displays a GUI (Graphical User Interface) for receiving various instructions and settings from the user via the input interface 31. For example, the display 32 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 32 may be a desktop type, or may be configured by a tablet terminal or the like capable of wireless communication with the main body of the medical information processing apparatus 30.

Although the medical information processing apparatus 30 will be described as including the display 32 in FIG. 1, the medical information processing apparatus 30 may include a projector in place of or in addition to the display 32. The projector can project onto a screen, a wall, a floor, a body surface of the subject P1 or the like under the control of the processing circuit 34. As an example, the projector can also project onto an arbitrary plane, object, space, etc. by projection mapping. For example, the processing circuit 34 may display a composite image described later on the projector.

The memory 33 stores various data under the control of the processing circuit 34. The storage of data in the memory 33 will be described later. Further, the memory 33 can also store a program for the circuit included in the medical information processing apparatus 30 to realize its function. For example, the memory 33 is realized by a RAM (Random Access Memory), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. Alternatively, the memory 33 may be realized by a server group (cloud) connected to the medical information processing device 30 via a network.

The processing circuit 34 controls the operation of the entire medical information processing apparatus 30 by executing the control function 34a, the display control function 34b, the reception function 34c, and the model generation function 34d. Here, the control function 34a is an example of a control unit. Further, the display control function 34b is an example of the display control unit. The reception function 34c is an example of the reception unit. The model generation function 34d is an example of a model generation unit.

For example, the processing circuit 34 controls the transmission / reception of data via the network NW and the data management in the memory 33 by reading the program corresponding to the control function 34a from the memory 33 and executing the program. Further, the processing circuit 34 controls the display on the display 32 by reading the program corresponding to the display control function 34b from the memory 33 and executing the program. Further, the processing circuit 34 receives an input operation from the user via the input interface 31 by reading the program corresponding to the reception function 34c from the memory 33 and executing the program. Further, the processing circuit 34 executes machine learning by reading a program corresponding to the model generation function 34d from the memory 33 and executing the program to generate the trained model. The details of the processing by the control function 34a, the display control function 34b, the reception function 34c, and the model generation function 34d will be described later.

In the medical information processing apparatus 30 shown in FIG. 1, each processing function is stored in the memory 33 in the form of a program that can be executed by a computer. The processing circuit 34 is a processor that realizes a function corresponding to each program by reading a program from the memory 33 and executing the program. In other words, the processing circuit 34 in the state where the program is read has a function corresponding to the read program.

Although it has been described in FIG. 1 that the control function 34a, the display control function 34b, the reception function 34c, and the model generation function 34d are realized by a single processing circuit 34, processing is performed by combining a plurality of independent processors. The circuit 34 may be configured and the function may be realized by each processor executing a program. Further, each processing function of the processing circuit 34 may be appropriately distributed or integrated into a single or a plurality of processing circuits.

Further, the processing circuit 34 may realize the function by using the processor of the external device connected via the network NW. For example, the processing circuit 34 reads a program corresponding to each function from the memory 33 and executes it, and also uses a server group (cloud) connected to the medical information processing apparatus 30 via a network NW as a computational resource. , Each function shown in FIG. 1 is realized.

Next, the X-ray diagnostic apparatus 10 will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the configuration of the X-ray diagnostic apparatus 10 according to the first embodiment. As shown in FIG. 2, the X-ray diagnostic apparatus 10 includes an X-ray high voltage apparatus 101, an X-ray tube 102, an X-ray filter 103, a top plate 104, a C-arm 105, and an X-ray detector 106. The input interface 107, the display 108, the memory 109, and the processing circuit 110 are provided.

The X-ray high voltage device 101 supplies a high voltage to the X-ray tube 102 under the control of the processing circuit 110. For example, the X-ray high voltage device 101 has an electric circuit such as a transformer and a rectifier, and the high voltage generator that generates a high voltage applied to the X-ray tube 102 and the X-ray tube 102 irradiate the device. It has an X-ray control device that controls an output voltage according to X-rays. The high voltage generator may be a transformer type or an inverter type.

The X-ray tube 102 is a vacuum tube having a cathode (filament) that generates thermions and an anode (target) that receives the collision of thermions and generates X-rays. The X-ray tube 102 generates X-rays by irradiating thermions from the cathode toward the anode using the high voltage supplied from the X-ray high voltage device 101.

The X-ray filter 103 has a collimator that narrows the irradiation range of X-rays generated by the X-ray tube 102, and a filter that adjusts the X-rays exposed from the X-ray tube 102.

The collimator in the X-ray diaphragm 103 has, for example, four sliding diaphragm blades. The collimator slides the diaphragm blades to narrow down the X-rays generated by the X-ray tube 102 and irradiate the subject P1. Here, the diaphragm blade is a plate-shaped member made of lead or the like, and is provided near the X-ray irradiation port of the X-ray tube 102 in order to adjust the X-ray irradiation range.

The filter in the X-ray filter 103 changes the quality of X-rays transmitted depending on the material and thickness for the purpose of reducing the exposure dose to the subject P1 and improving the image quality of the X-ray image, and is absorbed by the subject P1. It reduces the soft line component that tends to occur, and reduces the high energy component that causes a decrease in the contrast of the X-ray image. Further, the filter changes the dose and irradiation range of X-rays depending on the material, thickness, position, etc., and X-rays are emitted so that the X-rays irradiated from the X-ray tube 102 to the subject P1 have a predetermined distribution. Attenuates.

For example, the X-ray diaphragm 103 has a drive mechanism such as a motor and an actuator, and controls X-ray irradiation by operating the drive mechanism under the control of a processing circuit 110 described later. For example, the X-ray diaphragm 103 adjusts the opening degree of the diaphragm blade of the collimator by applying a drive voltage to the drive mechanism according to the control signal received from the processing circuit 110, and irradiates the subject P1. The irradiation range of the X-rays to be generated is controlled. Further, for example, the X-ray diaphragm 103 is irradiated to the subject P1 by adjusting the position of the filter by applying a drive voltage to the drive mechanism according to the control signal received from the processing circuit 110. Control the distribution of X-ray doses.

The top plate 104 is a bed on which the subject P1 is placed, and is arranged on a bed (not shown). The subject P1 is not included in the X-ray diagnostic apparatus 10. For example, the sleeper has a drive mechanism such as a motor and an actuator, and controls the movement / tilt of the top plate 104 by operating the drive mechanism under the control of the processing circuit 110 described later. For example, the sleeper moves or tilts the top plate 104 by applying a drive voltage to the drive mechanism according to the control signal received from the processing circuit 110.

The C-arm 105 holds the X-ray tube 102, the X-ray diaphragm 103, and the X-ray detector 106 so as to face each other with the subject P1 interposed therebetween. For example, the C arm 105 has a drive mechanism such as a motor and an actuator, and rotates or moves by operating the drive mechanism under the control of a processing circuit 110 described later. For example, the C-arm 105 attaches a drive voltage to the drive mechanism in response to a control signal received from the processing circuit 110, so that the X-ray tube 102, the X-ray filter 103, and the X-ray detector 106 are subject to the subject. The X-ray irradiation position and irradiation angle are controlled by rotating and moving with respect to P1. Note that FIG. 2 describes the case where the X-ray diagnostic apparatus 10 is a single plane as an example, but the embodiment is not limited to this, and may be a biplane case.

The X-ray detector 106 is, for example, an X-ray plane detector (Flat Panel Detector: FPD) having detection elements arranged in a matrix. The X-ray detector 106 detects X-rays irradiated from the X-ray tube 102 and transmitted through the subject P1, and outputs a detection signal corresponding to the detected X-ray dose to the processing circuit 110. The X-ray detector 106 may be an indirect conversion type detector having a grid, a scintillator array, and an optical sensor array, or a direct conversion type having a semiconductor element that converts incident X-rays into an electric signal. It may be a detector.

The input interface 107 can be configured in the same manner as the input interface 31 described above. For example, the input interface 107 receives various input operations from the user, converts the received input operations into electric signals, and outputs the received input operations to the processing circuit 110.

The display 108 can be configured in the same manner as the display 32 described above. For example, the display 108 displays a medical image or a GUI under the control of the processing circuit 110. Further, the X-ray diagnostic apparatus 10 may include a projector in place of or in addition to the display 108.

The memory 109 can be configured in the same manner as the memory 33 described above. For example, the memory 109 stores various data under the control of the processing circuit 110, and the circuit included in the X-ray diagnostic apparatus 10 stores a program for realizing the function.

The processing circuit 110 controls the operation of the entire X-ray diagnostic apparatus 10 by executing the control function 110a, the display control function 110b, and the collection function 110c. The control function 110a is an example of a control unit. Further, the display control function 110b is an example of a display control unit. The collection function 110c is an example of a collection unit.

For example, the processing circuit 110 controls the transmission / reception of data via the network NW and the data management in the memory 109 by reading the program corresponding to the control function 110a from the memory 109 and executing the program. Further, for example, the processing circuit 110 controls the display on the display 108 by reading the program corresponding to the display control function 110b from the memory 109 and executing the program. Further, for example, the processing circuit 110 executes image collection from the subject P1 by reading a program corresponding to the collection function 110c from the memory 109 and executing the program. The details of the processing by the control function 110a, the display control function 110b, and the collection function 110c will be described later.

In the X-ray diagnostic apparatus 10 shown in FIG. 2, each processing function is stored in the memory 109 in the form of a program that can be executed by a computer. The processing circuit 110 is a processor that realizes a function corresponding to each program by reading a program from the memory 109 and executing the program. In other words, the processing circuit 110 in the state where the program is read has a function corresponding to the read program.

Although it has been described in FIG. 2 that the control function 110a, the display control function 110b, and the collection function 110c are realized by a single processing circuit 110, the processing circuit 110 is configured by combining a plurality of independent processors. , The function may be realized by each processor executing a program. Further, each processing function of the processing circuit 110 may be appropriately distributed or integrated into a single or a plurality of processing circuits.

Further, the processing circuit 110 may realize the function by using the processor of the external device connected via the network NW. For example, the processing circuit 110 reads a program corresponding to each function from the memory 109 and executes it, and also uses a server group connected to the X-ray diagnostic apparatus 10 via the network NW as a computational resource in FIG. 2. Realize each function shown in.

The configuration example of the medical information processing system 1 has been described above. Under such a configuration, the medical information processing apparatus 30 in the medical information processing system 1 improves the accuracy of the combined display of the two-dimensional X-ray image and the three-dimensional data by the processing by the processing circuit 34.

First, three-dimensional data will be described. The three-dimensional data is, for example, a three-dimensional model showing blood vessels, organs, and the like of the subject P1. For example, 3D data is generated based on a 3D medical image collected from subject P1. Here, the three-dimensional medical image is collected by, for example, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resolution Imaging) device, or the like (not shown).

As an example, the three-dimensional data is generated in the medical information processing apparatus 30. For example, the control function 34a first acquires a three-dimensional medical image via the network NW and stores it in the memory 33. For example, the three-dimensional medical image of the subject P1 collected by an X-ray CT device, an MRI device, or the like is stored in an image storage device (not shown) together with incidental information such as patient information. Examples of such an image storage device include a PACS (Picture Archiving and Communication System) server. In this case, the control function 34a can acquire a three-dimensional medical image from the image storage device via the network NW and store it in the memory 33. The control function 34a may acquire a three-dimensional medical image directly from an X-ray CT device, an MRI device, or the like without going through another device.

Next, the control function 34a performs a three-dimensional data generation process based on the acquired three-dimensional medical image. For example, the control function 34a performs threshold processing on the voxel value in the three-dimensional medical image and extracts the voxel corresponding to the blood vessel to generate a three-dimensional model showing the blood vessel. Further, the control function 34a stores the generated three-dimensional data in the memory 33.

The three-dimensional data may be generated based on the three-dimensional X-ray image collected by the X-ray diagnostic apparatus 10. For example, the collection function 110c controls the operation of the C arm 105 to rotate the X-ray tube 102 around the subject P1 and irradiate the X-ray pulse at a predetermined frame rate to change the X-ray irradiation direction. Collect multiple altered projection data. That is, the collection function 110c executes rotary imaging for the subject P1. Then, the collection function 110c reconstructs a three-dimensional X-ray image based on a plurality of projection data. In this case, the control function 34a can acquire a three-dimensional X-ray image via the network NW and perform a three-dimensional data generation process. A technique for generating 3D data based on a 3D medical image other than a 3D X-ray image, combining it with a 2D X-ray image, and displaying it is also called MMR (Multi Modality Roadmap).

Further, the three-dimensional data may be generated by a device other than the medical information processing device 30. For example, even if a medical image diagnostic device such as an X-ray CT device, an MRI device, or an X-ray diagnostic device 10 collects a three-dimensional medical image and executes a process of generating three-dimensional data based on the three-dimensional medical image. I do not care. In this case, the control function 34a can acquire the three-dimensional data via the network NW and store it in the memory 33.

After the control function 34a acquires the three-dimensional data and stores it in the memory 33, the X-ray diagnostic apparatus 10 executes the collection of the two-dimensional X-ray image from the subject P1. For example, the collection function 110c collects two-dimensional X-ray images over time while the stent graft placement in the abdominal blood vessel of the subject P1 is performed.

Specifically, the collection function 110c controls the X-ray high voltage device 101 and adjusts the voltage supplied to the X-ray tube 102 to control the X-ray dose and on / off of the subject P1. Control. Further, the collection function 110c controls the operation of the X-ray diaphragm 103 and adjusts the opening degree of the diaphragm blades of the collimator to control the irradiation range of the X-rays irradiated to the subject P1. Further, the collection function 110c controls the operation of the X-ray diaphragm 103 and adjusts the position of the filter to control the distribution of the X-ray dose. Further, the collection function 110c controls the operation of the C arm 105 to rotate or move the C arm 105. Further, for example, the collection function 110c moves or tilts the top plate 104 by controlling the operation of the bed. Further, the collection function 110c generates a two-dimensional X-ray image based on the detection signal received from the X-ray detector 106. The collection function 110c may perform various image processing on the generated two-dimensional X-ray image. For example, the collection function 110c can execute noise reduction processing by an image processing filter and scattered ray correction on a two-dimensional X-ray image.

Next, the control function 34a acquires the two-dimensional X-ray image collected by the X-ray diagnostic apparatus 10 via the network NW. For example, first, the control function 110a transmits the two-dimensional X-ray image collected by the collection function 110c to the image storage device via the network NW. In this case, the control function 34a can acquire a two-dimensional X-ray image from the image storage device via the network NW and store it in the memory 33. Alternatively, the control function 34a may acquire a two-dimensional X-ray image directly from the X-ray diagnostic apparatus 10 without going through another device.

Next, the display control function 34b displays a composite image in which an image based on the three-dimensional data and a two-dimensional X-ray image are combined. In other words, the display control function 34b displays an superimposed image of the image based on the three-dimensional data and the two-dimensional X-ray image. For example, the display control function 34b first automatically adjusts the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image. Specifically, the display control function 34b adjusts at least one of the position, orientation, and size of the three-dimensional data with respect to the two-dimensional X-ray image as the arrangement of the three-dimensional data.

The position of the three-dimensional data with respect to the two-dimensional X-ray image is the position in the vertical direction, the horizontal direction, and the depth direction of the two-dimensional X-ray image. For example, the display control function 34b estimates the position of the region of interest in the two-dimensional X-ray image in the three-dimensional space, and three-dimensionally adjusts the arrangement of the three-dimensional data according to the estimated position. Further, in the following, the process of adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image will be simply described as alignment.

Here, the display control function 34b can automatically align the three-dimensional data and the two-dimensional X-ray image by an arbitrary method. For example, the display control function 34b extracts anatomical feature points from each of the three-dimensional data and the two-dimensional X-ray image, and calculates a transformation matrix based on the correspondence of the feature points. Specifically, the display control function 34b calculates a transformation matrix for converting 3D data into a 2D image having the same position, orientation, and size as the 2D X-ray image. Then, the display control function 34b generates a two-dimensional image in which each point on the three-dimensional data is coordinate-converted by a transformation matrix, combines it with the two-dimensional X-ray image, and displays it on the display 32.

Further, when the three-dimensional data is generated based on the three-dimensional X-ray image, the display control function 34b can also calculate the transformation matrix based on the control information of the top plate 104 or the like. That is, when the 3D X-ray image and the 2D X-ray image used for generating the 3D data are collected by the same device, the display control function 34b is 3D based on the control information of the top plate 104 or the like. The positional relationship between the data and the two-dimensional X-ray image can be acquired and the conversion matrix can be calculated. Then, the display control function 34b generates a two-dimensional image in which each point on the three-dimensional data is coordinate-converted by a transformation matrix, combines it with the two-dimensional X-ray image, and displays it on the display 32.

The display control function 34b may perform coordinate transformation of only the surface region of the three-dimensional data by the transformation matrix, or may perform coordinate transformation including the internal region. That is, the display control function 34b may generate a surface-rendered image or a volume-rendered image as an image based on three-dimensional data.

As described above, the display control function 34b can automatically align the 3D data and the 2D X-ray image and display a composite image in which the image based on the 3D data and the 2D X-ray image are combined. can. For example, the collection function 110c collects a two-dimensional X-ray image over time while the procedure is being performed. Further, the display control function 34b sequentially synthesizes an image based on the three-dimensional data with the collected two-dimensional X-ray image, and sequentially displays the generated composite image on the display 32. That is, the display control function 34b can display the composite image in real time.

However, due to various factors, it may not be possible to properly align the 3D data and the 2D X-ray image. For example, since the contrast and brightness of the image, the type of noise generated, and the like are different between the three-dimensional medical image and the two-dimensional X-ray image, it may not be possible to specify the corresponding feature points. In particular, when three-dimensional data is generated based on an X-ray CT image, an MR image, or the like, it is not easy to specify the corresponding feature points because the modality types are different. Further, even when the 3D data is generated based on the 3D X-ray image and the alignment is performed based on the control information of the top plate 104 or the like, the accuracy of the alignment is improved by the body movement of the subject P1 or the like. It may decrease.

In addition, the conditions for creating three-dimensional data may also affect the accuracy of alignment. For example, when three-dimensional data is created by threshold processing, if the threshold is changed, the shape and size of the generated three-dimensional data also change. In addition, when creating 3D data, the shape and size of the 3D data may vary depending on the creator, where the engineer or the like may empirically adjust the creation conditions. Then, the accuracy of alignment with respect to the two-dimensional X-ray image may decrease due to changes in the shape and size of the three-dimensional data.

Further, even in the case where the vertical and horizontal directions of the two-dimensional X-ray image can be properly aligned, an estimation error in the depth direction may occur. In such a case, for example, when the C arm 105 is rotated to change the shooting angle of the two-dimensional X-ray image, the three-dimensional data is displayed shifted in the depth direction.

If the automatic alignment is not done properly, the user can do the alignment manually. For example, the user refers to the composite image displayed on the display 32 and translates, rotates, or enlarges / reduces the three-dimensional data to arrange the three-dimensional data with respect to the two-dimensional X-ray image. adjust. However, performing such a manual operation during the procedure is burdensome for the user. In addition, the accuracy of manual alignment also varies depending on the amount of experience of the user. In particular, the alignment in the depth direction depends largely on experience, and the accuracy tends to vary from user to user.

Here, it is conceivable to perform the alignment of the three-dimensional data with respect to the two-dimensional X-ray image by a machine learning method. However, it is difficult to determine what kind of data is appropriate as training data, and it is not easy to collect the required number of training data to ensure the quality of the trained model.

Therefore, the processing circuit 34 improves the accuracy of the combined display of the two-dimensional X-ray image and the three-dimensional data by the processing described below. Specifically, the processing circuit 34 receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image from the user who has referred to the composite image, and performs the adjustment operation with the three-dimensional data and the two-dimensional X-ray image. Save the results. As a result, the processing circuit 34 makes it possible to perform appropriate learning on the trained model that adjusts the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image, and improves the accuracy of the composite display.

Hereinafter, the processing performed by the processing circuit 34 will be described with reference to FIG. FIG. 3 is a diagram for explaining acceptance of the adjustment operation according to the first embodiment. For example, the display control function 34b first automatically aligns the three-dimensional data D11 with respect to the two-dimensional X-ray image I11. For example, the display control function 34b calculates a transformation matrix A for converting the three-dimensional data D11 into a two-dimensional image having the same position, orientation, and size as the two-dimensional X-ray image I11. Further, the display control function 34b generates an image I21 based on the three-dimensional data D11 by applying the conversion matrix A to the three-dimensional data D11, and the composite image I31 that combines the two-dimensional X-ray image I11 and the image I21. Is displayed on the display 32.

The method of automatic alignment is not particularly limited. For example, the display control function 34b extracts anatomical feature points from each of the three-dimensional data D11 and the two-dimensional X-ray image I11, and calculates the transformation matrix A based on the combination of the corresponding feature points. Further, for example, when the three-dimensional data D11 is generated based on the three-dimensional X-ray image, the display control function 34b can also calculate the transformation matrix A based on the control information of the top plate 104 or the like.

Here, the reception function 34c can receive an adjustment operation for adjusting the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 from the user who has referred to the composite image I31. For example, when the user who refers to the composite image I31 determines that the alignment is not properly performed, the user adjusts the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 via the input interface 31. input.

Specifically, when the user determines that the position of the 3D data D11 is displaced with respect to the 2D X-ray image I11, the user moves the 3D data D11 in parallel and adjusts the position of the 3D data D11. do. Here, the user can translate the three-dimensional data D11 in any direction such as the vertical direction, the horizontal direction, and the depth direction of the two-dimensional X-ray image I11. Further, when the user determines that the orientation of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 is deviated, the user rotates the three-dimensional data D11 to adjust the angle of the three-dimensional data D11. Here, the user can rotate the three-dimensional data D11 three-dimensionally on the rotation axes in arbitrary directions such as the vertical direction, the horizontal direction, and the depth direction of the two-dimensional X-ray image I11. Further, when the user determines that the enlargement ratios do not match between the two-dimensional X-ray image I11 and the three-dimensional data D11, the three-dimensional data D11 is enlarged or reduced to reduce the size of the three-dimensional data D11. adjust.

While the reception function 34c is receiving the adjustment operation from the user, the display control function 34b changes the transformation matrix according to the arrangement of the adjusted three-dimensional data. That is, the display control function 34b changes the transformation matrix so that the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 changes as desired by the user. Further, the display control function 34b applies the transformation matrix to the three-dimensional data D11 every time the transformation matrix is changed, and updates the composite image to be displayed on the display 32.

In FIG. 3, the result of manual alignment will be described as the composite image I41. For example, the display control function 34b applies the transformation matrix A'based on the adjustment operation received from the user to the three-dimensional data D11 to generate the composite image I41 and display it on the display 32. Further, when the control function 34a determines that the adjustment operation by the user is completed while the composite image I41 is displayed, the control function 34a acquires the transformation matrix A'as a result of the adjustment operation and stores it in the memory 33. Further, the control function 34a stores the two-dimensional X-ray image I11 and the three-dimensional data D11 in the memory 33 together with the transformation matrix A'. That is, the control function 34a registers the transformation matrix A', the two-dimensional X-ray image I11, and the three-dimensional data D11 in the database as training data that can be used to generate the trained model M1. The process of generating the trained model M1 will be described later.

Here, an example of a process for determining whether or not the adjustment operation by the user has been completed will be described with reference to FIG. FIG. 4 is a flowchart for explaining a series of processes of the medical information processing apparatus 30 according to the first embodiment.

First, the display control function 34b automatically aligns the three-dimensional data D11 with respect to the two-dimensional X-ray image I11, and displays a composite image obtained by synthesizing the image based on the three-dimensional data D11 and the two-dimensional X-ray image I11. Display (step S101). Next, the display control function 34b determines whether or not to continue the display of the composite image (step S102), and if it continues (step S102 affirmation), the process proceeds to step S103.

In step S103, the reception function 34c determines whether or not the alignment is manually performed. That is, the reception function 34c determines whether or not the user who has referred to the composite image has received the adjustment operation for adjusting the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11. If the alignment is not performed manually (denial in step S103), the reception function 34c shifts to step S102 again.

When the position is manually aligned (affirmation in step S103), the control function 34a determines whether or not there is an operation specified for a certain period of time (step S104). More specifically, the reception function 34c receives an adjustment operation for adjusting the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 from the user who has referred to the composite image, and the display control function 34b responds to the adjustment operation. The converted composite image is applied to the three-dimensional data D11 to update the composite image to be displayed on the display 32, and the control function 34a is an operation specified for a certain period of time while the updated composite image is displayed on the display 32. Determine if there is any.

Here, the fixed time used for the determination in step S104 may be a preset length of time or a user-set length of time. Further, the control function 34a may automatically adjust the fixed time based on the length of time required when the user re-inputs the adjustment operation in the past. The fixed time in step S104 is an example of a predetermined time.

Further, the designated operation used for the determination in step S104 is, for example, an adjustment operation for adjusting the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11. That is, in step S104, the control function 34a determines whether or not the adjustment operation has been re-input. Further, the designated operation used for the determination in step S104 is, for example, an operation of rotating the C arm 105 or an operation of moving the top plate 104.

For example, the user can determine whether or not the alignment in the depth direction is properly performed by rotating the C arm 105. That is, the user rotates the arm 105 to change the shooting angle of the two-dimensional X-ray image I11. Further, the display control function 34b rotates and moves the three-dimensional data D11 according to the change in the shooting angle, and updates the composite image to be displayed on the display 32. Here, where the three-dimensional data D11 is displayed shifted in the depth direction if the alignment in the depth direction is not properly performed, if the shift does not occur even if the C arm 105 is rotated, the user can use the user. It can be determined that the alignment in the depth direction is properly performed. The operation specified in step S104 is an example of a specific operation.

When there is no operation specified for a certain period of time in step S104 (affirmation in step S104), the control function 34a registers the three-dimensional data D11, the two-dimensional X-ray image I11, and the transformation matrix A'in the database (step S105). That is, if there is no operation specified for a certain period of time while the composite image I41 is displayed, the control function 34a determines that the adjustment operation by the user has been completed, and specifies the transformation matrix A'as a result of the adjustment operation. And register it in the database together with the 3D data D11 and the 2D X-ray image I11. For example, the control function 34a attaches incidental information for specifying the three-dimensional data D11 and the two-dimensional X-ray image I11 to the transformation matrix A'and stores them in the memory 33. After step S105, the control function 34a shifts to step S102 again.

On the other hand, if the designated operation is performed within a certain period of time (step S104 negative), the control function 34a determines whether or not to continue the determination in step S104 (step S106), and if it continues (step S106). S106 affirmative), the process proceeds to step S104 again. On the other hand, if the determination in step S104 is not continued (negation in step S106), the control function 34a shifts to step S102 again. For example, when the user determines that it is difficult to align the 3D data D11 and the 2D X-ray image I11 due to an artifact or the like, the control function 34a shifts to step S102 again without registering in the database. do. If it is determined in step S102 that the display is not continued (negation in step S102), the processing circuit 34 ends the processing.

As described above, the control function 34a determines whether or not the adjustment operation by the user is completed, and if it is determined that the adjustment operation is completed, the transformation matrix A'is used as training data that can be used to generate the trained model M1. The two-dimensional X-ray image I11 and the three-dimensional data D11 are registered in the database. As a result, the control function 34a makes it possible to perform appropriate learning on the trained model M1 and can improve the accuracy of the composite display.

The flowchart of FIG. 4 is an example, and various modifications can be made. For example, the control function 34a may determine that the adjustment operation has been completed when the input operation to the effect that the adjustment operation has been completed is performed, and may perform registration in the database.

Next, the generation process of the trained model M1 will be described with reference to FIG. FIG. 5 is a diagram showing an example of the generation process of the trained model M1 according to the first embodiment. In this embodiment, it is assumed that the model generation function 34d in the medical information processing apparatus 30 generates the trained model M1.

Further, in FIG. 5, a neural network will be described as an example of a machine learning method. That is, the model generation function 34d has been trained by training the neural network with the three-dimensional data D11 and the two-dimensional X-ray image I11 as input side data (Input) and the transformation matrix A'as output side data (Output). Generate model M1. In FIG. 5, for convenience of explanation, only one combination of the input side data and the output side data is shown, but the model generation function 34d uses a plurality of combinations of the input side data and the output side data. The neural network can be trained repeatedly.

Specifically, the model generation function 34d inputs the three-dimensional data D11 and the two-dimensional X-ray image I11 into the neural network. Here, in the neural network, information is propagated while being coupled only between adjacent layers in one direction from the input layer side to the output layer side, and the estimation result of the transformation matrix is output from the output layer. For example, the output layer of the neural network outputs the estimation result of the transformation matrix used for appropriately arranging the three-dimensional data D11 with respect to the two-dimensional X-ray image I11. A neural network in which information propagates in one direction from the input layer side to the output layer side is also called a convolutional neural network (CNN).

The model generation function 34d generates the trained model M1 by adjusting the parameters of the neural network so that the neural network can output a preferable result when the input side data is input. For example, the model generation function 34d processes while adjusting the parameters of the neural network until the difference between the transformation matrix estimated based on the input side data and the transformation matrix A'input as the output side data falls below the threshold value. repeat. As a result, the model generation function 34d can generate a trained model M1 functionalized to adjust the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image. For example, in the case shown in FIG. 5, the trained model M1 receives the input of the three-dimensional data and the two-dimensional X-ray image, and appropriately arranges the three-dimensional data with respect to the input two-dimensional X-ray image. It is functionalized to estimate the transformation matrix.

Although the trained model M1 has been described as being composed of a convolutional neural network (CNN), the model generation function 34d has other types such as a fully coupled neural network and a recurrent neural network (RNN). The trained model M1 may be constructed by the neural network of. Further, the model generation function 34d may generate the trained model M1 by a machine learning method other than the neural network. For example, the model generation function 34d uses algorithms such as logistic regression analysis, nonlinear discriminant analysis, support vector machine (SVM), random forest (Random Forest), and naive bayes (Naive Bayes). It may be possible to perform training and generate a trained model M1.

Further, the model generation function 34d may generate a plurality of types of trained models M1. For example, the model generation function 34d may generate a trained model M1 for each site such as a blood vessel or an organ. Further, for example, the model generation function 34d has a trained model M11 using 3D data based on a 3D X-ray image as training data, and a trained model M12 using 3D data based on an X-ray CT image as training data. , A trained model M13 or the like using three-dimensional data based on an MRI image as training data may be generated.

The control function 34a stores the generated trained model M1 in the memory 33. After that, when the two-dimensional X-ray image and the three-dimensional data are combined and displayed for the subject P1 or another subject, the display control function 34b can read the trained model M1 from the memory 33 and use it.

Hereinafter, processing in the case of performing a composite display of a two-dimensional X-ray image and three-dimensional data using the trained model M1 will be described with reference to FIG. FIG. 6 is a flowchart for explaining a series of processes of the medical information processing apparatus 30 according to the first embodiment. In FIG. 6, the subject P2 different from the subject P1 will be described as a composite display of the two-dimensional X-ray image I12 and the three-dimensional data D12. The 3D data D12 is generated, for example, based on a 3D medical image collected from the subject P2. Further, the two-dimensional X-ray image I12 is collected over time by the X-ray diagnostic apparatus 10, for example, in a procedure for the subject P2.

First, the display control function 34b acquires the state of the three-dimensional data D12 (step S201), and reads the trained model M1 from the memory 33. For example, the display control function 34b acquires information such as a portion indicated by the three-dimensional data D12 and the type of the three-dimensional medical image used for generating the three-dimensional data D12 as the state of the three-dimensional data D12. Then, the display control function 34b reads the learned model M1 according to the state of the three-dimensional data D12 from the memory 33.

Next, the display control function 34b inputs the three-dimensional data D12 and the two-dimensional X-ray image I12 to the trained model M1 (step S202). Here, the trained model M1 estimates and outputs a transformation matrix for appropriately arranging the three-dimensional data D12 with respect to the two-dimensional X-ray image I12. Next, the display control function 34b generates an image based on the three-dimensional data D12 by applying the transformation matrix output from the trained model M1 to the three-dimensional data D12 (step S203). Then, the display control function 34b causes the display 32 to display a composite image obtained by synthesizing the image based on the three-dimensional data D12 and the two-dimensional X-ray image I12 (step S204). That is, the display control function 34b inputs the three-dimensional data D12 and the two-dimensional X-ray image I12 to the trained model M1, and receives the output of the adjustment result of the arrangement of the three-dimensional data D12 with respect to the two-dimensional X-ray image I12. Then, a composite image obtained by synthesizing the image based on the three-dimensional data D12 and the two-dimensional X-ray image I12 is displayed.

Next, the display control function 34b determines whether or not to continue displaying the composite image (step S205). Here, when the display is continued (step S205 affirmative), the display control function 34b shifts to step S204 again.

While continuing to display the composite image, the user may instruct the movement / rotation of the top plate 104 and the C arm 105. For example, the user may rotate the arm 105 to change the shooting angle of the two-dimensional X-ray image in order to observe the region of interest in the procedure at a different angle. In such a case, the display control function 34b can rotate and move the three-dimensional data D12 according to the change in the shooting angle to update the composite image to be displayed on the display 32.

For example, the X-ray diagnostic apparatus 10 transmits the information of the coordinate system in the imaging system to the medical information processing apparatus 30 in a state of being attached to the two-dimensional X-ray image I12. Here, the information of the coordinate system is control information of a mechanical system that controls a shooting position and a shooting angle, such as a top plate 104 and a C arm 105. The display control function 34b can update the composite image based on the information of the coordinate system attached to the two-dimensional X-ray image I12. For example, when the control to rotate the arm 105 is performed, the display control function 34b rotates and moves the three-dimensional data D12 according to the amount of rotation of the arm 105 based on the information of the coordinate system. Then, the display control function 34b causes the display 32 to display a composite image obtained by synthesizing the image based on the three-dimensional data D12 after the rotational movement and the two-dimensional X-ray image I12.

That is, it is not necessary to repeatedly use the trained model M1 when the shooting position or shooting angle of the two-dimensional X-ray image I12 changes during the procedure, and the display control function 34b is based on the information in the coordinate system. The arrangement of the three-dimensional data D12 with respect to the line image I12 can be readjusted, and the composite image can be updated quickly.

In addition, it is assumed that the arrangement cannot be readjusted based on the information in the coordinate system, such as when the subject P2 moves. In such a case, the display control function 34b can shift to step S202 again and readjust the arrangement of the three-dimensional data D12 with respect to the two-dimensional X-ray image I12 by using the trained model M1. For example, the medical information processing apparatus 30 has a button for readjustment. Further, the user presses the readjustment button when it is determined that the subject P2 has moved. Then, when the button for readjustment is pressed, the display control function 34b executes readjustment using the trained model M1.

As described above, according to the first embodiment, the display control function 34b displays a composite image obtained by synthesizing an image based on three-dimensional data and a two-dimensional X-ray image. Further, the reception function 34c receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image from the user who has referred to the composite image. Further, the control function 34a stores the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation accepted by the reception function 34c in the memory 33. This makes it possible for the medical information processing apparatus 30 to appropriately learn the trained model M1 that adjusts the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image, and can improve the accuracy of the composite display. ..

That is, it is difficult to accurately align the 3D data and the 2D X-ray image with certain rules, such as when extracting anatomical feature points and performing alignment. For example, when aligning two-dimensional images, it is not necessary to consider the depth direction, and only two directions, vertical and horizontal, need to be adjusted. Therefore, high accuracy can be ensured even when alignment is performed according to certain rules. It is possible. Further, for example, in the alignment of three-dimensional images, since each image has three-dimensional information, it is possible to ensure high accuracy even when the alignment is performed according to a certain rule. On the other hand, in the alignment between the 3D data and the 2D X-ray image, it is necessary to estimate the position of the 3D data in the depth direction with respect to the 2D X-ray image, and it is easy to guarantee the accuracy of the alignment. Not. Further, a user such as a doctor or a technician can appropriately align the two-dimensional X-ray image and the three-dimensional data based on his / her sense and experience. However, it is a burden for the user to perform manual alignment every time the composite display is performed.

On the other hand, according to the medical information processing apparatus 30 according to the first embodiment, the result of manual alignment by the user can be collected as learning data. As a result, the medical information processing apparatus 30 trains the trained model M1 on the user's feelings and experiences, and aligns the 3D data with the 2D X-ray image with high accuracy that cannot be obtained by the alignment processing of a certain rule. Can be realized.

In addition, the medical information processing apparatus 30 can reduce the exposure dose by improving the alignment accuracy. For example, if the 3D data and the 2D X-ray image are not properly aligned, the user manually aligns the 3D data while referring to the 2D X-ray image displayed on the display 32. .. On the other hand, if the alignment is realized with sufficiently high accuracy, it can be said that the opportunity for manual alignment is reduced. Then, it is not necessary to collect two-dimensional X-ray images for manual positioning, and the number of collected two-dimensional X-ray images is reduced, so that the exposure amount can be reduced.

Further, after the trained model M1 is generated, the alignment between the 3D data and the 2D X-ray image is realized by inputting the 3D data and the 2D X-ray image to the trained model M1. To. That is, the medical information processing apparatus 30 can realize highly accurate positioning without performing complicated processing such as position shift detection and drawing processing. Therefore, the medical information processing apparatus 30 can reduce the calculation cost when performing the combined display of the two-dimensional X-ray image and the three-dimensional data.

(Second embodiment)
By the way, although the first embodiment has been described so far, various different embodiments may be implemented in addition to the above-described embodiment.

For example, in the above-described embodiment, the transformation matrix has been described as an example of the result of the adjustment operation. For example, in FIG. 3, it has been described that the control function 34a acquires the transformation matrix A'as a result of the adjustment operation received by the reception function 34c and stores it in the memory 33. However, the embodiments are not limited to this. For example, the control function 34a may store the composite image I41 of FIG. 3 in the memory 33 as a result of the adjustment operation received by the reception function 34c. That is, the control function 34a may store the composite image adjusted by the user and generated according to the arrangement of the three-dimensional data in the memory 33 as a result of the adjustment operation. In this case, the trained model M1 is generated, for example, by training the neural network using the composite image I41 as output side data (Autoput).

Further, in the above-described embodiment, the control function 34a has been described as storing the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation received by the reception function 34c in the memory 33. That is, in the above-described embodiment, the memory 33 has been described as an example of a storage unit that stores three-dimensional data, a two-dimensional X-ray image, and the result of an adjustment operation. However, the embodiments are not limited to this. For example, the control function 34a may store the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation in an external device connected via the network NW. Such an external device is an example of a storage unit.

Further, in the above-described embodiment, the control function 34a has been described as storing the trained model M1 in the memory 33. That is, in the above-described embodiment, the memory 33 has been described as an example of the model storage unit that stores the trained model M1. However, the embodiments are not limited to this. For example, the control function 34a may store the trained model M1 in an external device connected via the network NW. Such an external device is an example of a model storage unit.

Further, although a single memory 33 is shown in FIG. 1, a plurality of memories 33 may be distributed and arranged. For example, the medical information processing apparatus 30 includes a first memory 331, a second memory 332, and a third memory 333 as the memory 33. Here, the first memory 331 stores, for example, a program for the circuit included in the medical information processing apparatus 30 to realize its function. Further, the second memory 332 stores the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation accepted by the reception function 34c under the control of the control function 34a. Further, the third memory 333 stores the trained model M1 under the control of the control function 34a. In such a case, the second memory 332 is an example of the storage unit. Further, the third memory 333 is an example of the model storage unit.

Further, in the above-described embodiment, the generation process of the trained model M1 has been described as being executed by the model generation function 34d of the medical information processing apparatus 30. However, the embodiments are not limited to this. For example, the trained model M1 may be generated in another device (not shown). In this case, the control function 34a can acquire the trained model M1 via the network NW and store it in a model storage unit such as a memory 33.

Further, in the above-described embodiment, the medical information processing apparatus 30 has been described as displaying a composite image. However, the embodiments are not limited to this. For example, the processing circuit 110 of the X-ray diagnostic apparatus 10 may execute the functions corresponding to the control function 34a, the display control function 34b, the reception function 34c, and the model generation function 34d described above. Hereinafter, this point will be described with reference to FIG. 7. FIG. 7 is a block diagram showing an example of the configuration of the X-ray diagnostic apparatus 10 according to the second embodiment. As shown in FIG. 7, the processing circuit 110 executes the control function 110a, the display control function 110b, the collection function 110c, the reception function 110d, and the model generation function 110e. The reception function 110d is an example of the reception unit. Further, the model generation function 110e is an example of a model generation unit.

For example, the display control function 110b first displays a composite image obtained by synthesizing an image based on the three-dimensional data D11 and a two-dimensional X-ray image I11 on the display 108. Next, the reception function 110d receives an adjustment operation for adjusting the arrangement of the three-dimensional data D11 with respect to the two-dimensional X-ray image I11 from the user who has referred to the composite image. Next, the control function 110a stores the three-dimensional data D11, the two-dimensional X-ray image I11, and the result of the adjustment operation received by the reception function 110d in the memory 109. Then, the model generation function 110e generates the trained model M1 by executing machine learning using the three-dimensional data D11, the two-dimensional X-ray image I11, and the result of the adjustment operation as training data.

Further, the control function 110a stores the trained model M1 in, for example, the memory 109. After that, the collection function 110c collects the two-dimensional X-ray image I12. For example, the subject P2 is placed on the top plate 104 in place of the subject P1 shown in FIG. 7, and the collecting function 110c collects the two-dimensional X-ray image I12 from the subject P2. Further, the display control function 110b reads the trained model M1 from the memory 109, and inputs the three-dimensional data D12 and the two-dimensional X-ray image I12 to the trained model M1. Here, the trained model M1 outputs the adjustment result of the arrangement of the three-dimensional data D12 with respect to the two-dimensional X-ray image I12. Further, the display control function 110b receives the output from the trained model M1 to generate an image based on the three-dimensional data D12, synthesizes the image with the two-dimensional X-ray image I12, and displays the combined image on the display 108.

The term "processor" used in the above description refers to, for example, a CPU, a GPU (Graphics Processing Unit), an integrated circuit for a specific application (Application Specific Integrated Circuit: ASIC), a programmable logic device (for example, a simple programmable logic device (Simple Program)). It means a circuit such as a Logic Device (SPLD), a composite programmable logic device (Complex Programmable Logic Device: CPLD), and a field programmable gate array (Field Programmable Gate Array: FPGA). When the processor is, for example, a CPU, the processor realizes a function by reading and executing a program stored in a storage circuit. On the other hand, when the processor is, for example, an ASIC, the function is directly incorporated as a logic circuit in the circuit of the processor instead of storing the program in the storage circuit. It should be noted that each processor of the embodiment is not limited to the case where each processor is configured as a single circuit, and a plurality of independent circuits may be combined to form one processor to realize its function. .. Further, a plurality of components in each figure may be integrated into one processor to realize the function.

Further, in FIG. 1, a single memory 33 has been described as storing a program corresponding to each processing function of the processing circuit 34. Further, in FIGS. 2 and 7, a single memory 109 has been described as storing a program corresponding to each processing function of the processing circuit 110. However, the embodiments are not limited to this. For example, a plurality of memories 33 may be distributed and arranged, and the processing circuit 34 may be configured to read a corresponding program from the individual memories 33. Similarly, a plurality of memories 109 may be distributed and arranged, and the processing circuit 110 may be configured to read a corresponding program from the individual memories 109. Further, instead of storing the program in the memory, the program may be configured to be directly embedded in the circuit of the processor. In this case, the processor realizes the function by reading and executing the program embedded in the circuit.

Each component of each device according to the above-described embodiment is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Further, each processing function performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

Further, the medical information processing method described in the above-described embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program can be distributed via a network such as the Internet. Further, this program is recorded on a non-transient recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, or a DVD that can be read by a computer, and is executed by being read from the recording medium by the computer. You can also do it.

According to at least one embodiment described above, the accuracy of the combined display of the two-dimensional X-ray image and the three-dimensional data can be improved.

Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, changes, and combinations of embodiments can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1 Medical information processing system 10 X-ray diagnostic device 110 Processing circuit 110a Control function 110b Display control function 110c Collection function 110d Reception function 110e Model generation function 30 Medical information processing device 34 Processing circuit 34a Control function 34b Display control function 34c Reception function 34d Model Generation function

Claims (10)

A display control unit that displays a composite image that combines an image based on 3D data and a 2D X-ray image,
A reception unit that receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image from a user who has referred to the composite image.
A medical information processing apparatus including a control unit that stores the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation in a storage unit. The medical information processing apparatus according to claim 1, wherein the control unit identifies the result of the adjustment operation when the specific operation is not performed for a predetermined time, and stores the result in the storage unit. The medical information processing apparatus according to claim 1 or 2, wherein the control unit stores a transformation matrix adjusted by the user according to the arrangement of the three-dimensional data in the storage unit as a result of the adjustment operation. The medical information according to claim 1 or 2, wherein the control unit stores a composite image generated according to the arrangement of the three-dimensional data adjusted by the user in the storage unit as a result of the adjustment operation. Processing equipment. By executing machine learning using the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation as training data, the input two-dimensional X is based on the input three-dimensional data and the input two-dimensional X-ray image. The medical information processing apparatus according to any one of claims 1 to 4, further comprising a model generation unit that generates a trained model functionalized to adjust the arrangement of the input three-dimensional data with respect to the line image. A model storage unit that stores a trained model functionalized to adjust the placement of the input 3D data with respect to the input 2D X-ray image based on the input 3D data and the input 2D X-ray image.
The 3D data and the 2D X-ray image are input to the trained model, the adjustment result of the arrangement of the 3D data with respect to the 2D X-ray image is output, and the image is based on the 3D data. A display control unit that displays a composite image that is a composite of the two-dimensional X-ray image,
A medical information processing device. A collection unit that collects 2D X-ray images,
A display control unit that displays a composite image that combines an image based on 3D data and the 2D X-ray image.
A reception unit that receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image from a user who has referred to the composite image.
An X-ray diagnostic apparatus including a control unit that stores the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation in a storage unit. A model storage unit that stores a trained model functionalized to adjust the placement of the input 3D data with respect to the input 2D X-ray image based on the input 3D data and the input 2D X-ray image.
A collection unit that collects 2D X-ray images,
The 3D data and the 2D X-ray image are input to the trained model, the adjustment result of the arrangement of the 3D data with respect to the 2D X-ray image is output, and the image based on the 3D data is received. And a display control unit that displays a composite image that combines the two-dimensional X-ray image and
X-ray diagnostic device. Display a composite image that combines an image based on 3D data and a 2D X-ray image.
The user who referred to the composite image receives an adjustment operation for adjusting the arrangement of the three-dimensional data with respect to the two-dimensional X-ray image.
A program for causing a computer to execute each process of storing the three-dimensional data, the two-dimensional X-ray image, and the result of the adjustment operation in a storage unit. A trained model functionalized to adjust the arrangement of the input 3D data with respect to the input 2D X-ray image based on the input 3D data and the input 2D X-ray image is read from the model storage unit.
The 3D data and the 2D X-ray image are input to the trained model, the adjustment result of the arrangement of the 3D data with respect to the 2D X-ray image is output, and the image is based on the 3D data. A program that causes a computer to execute each process to display a composite image that is a composite of the two-dimensional X-ray image.

Images