JP7258483B2 - Medical information processing system, medical information processing device and ultrasonic diagnostic device - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to a medical information processing system, a medical information processing apparatus, and an ultrasonic diagnostic apparatus.

Conventionally, a technique is known in which an ultrasonic image is acquired by an ultrasonic diagnostic apparatus and an X-ray image is acquired by an X-ray diagnostic apparatus during a procedure, and the acquired ultrasonic image and the X-ray image are aligned and displayed. It is For example, in the treatment of heart valves, an ultrasound image that clearly visualizes the soft tissue including the heart valve to be treated, and an X-ray image that can visualize a wider range than the ultrasound image. are collected and observed at the same time to improve the accuracy of treatment.

Here, with the above-described technique, it has become possible in recent years to align an ultrasonic image and an X-ray image in real time. As a result, for example, a real-time ultrasonic image acquired during a procedure and a real-time X-ray image are aligned, and a synthesized image obtained by synthesizing the aligned ultrasonic image and the X-ray image is presented to the operator. It is also possible to present.

Japanese Patent Publication No. 2017-507723

The problem to be solved by the present invention is to improve the efficiency of the procedure.

A medical information processing system according to an embodiment includes an acquisition unit and a determination unit. The acquisition unit acquires information on a medical device inserted into the body of a subject and information on a target site for a procedure using the medical device based on ultrasound images acquired from the subject. The determining unit determines, based on a result of registration between the ultrasonic image and an X-ray image collected together with the ultrasonic image, when collecting an X-ray image during a procedure using the medical device on the target site. A direction of X-ray irradiation to the subject is determined.

FIG. 1 is a diagram showing an example of the configuration of a medical information processing system according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an ultrasonic diagnostic apparatus according to the first embodiment; FIG. 3 is a diagram illustrating an example of the configuration of the X-ray diagnostic apparatus according to the first embodiment; FIG. 4 is a diagram showing an example of the configuration of the medical information processing apparatus according to the first embodiment. FIG. 5 is a diagram for explaining an example of a technique according to the first embodiment; FIG. 6A is a diagram showing an example of image display according to the first embodiment. 6B is a diagram illustrating an example of image display according to the first embodiment; FIG. FIG. 7 is a diagram showing an example of display of a synthesized image according to the first embodiment. FIG. 8 is a diagram showing an example of display of a synthesized image according to the first embodiment. FIG. 9 is a diagram for explaining an example of selection of an ultrasound image to be synthesized with a synthesized image according to the first embodiment. FIG. 10 is a diagram illustrating an example of display of plan lines according to the first embodiment. FIG. 11A is a diagram for explaining an angle formed between the direction of the planning line and the irradiation direction of X-rays according to the first embodiment; 11B is a diagram illustrating an example of support information display according to the first embodiment; FIG. FIG. 11C is a diagram showing an example of support information display according to the first embodiment. FIG. 11D is a diagram showing an example of support information display according to the first embodiment. 11E is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. 11F is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. 11G is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. FIG. 12 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 13 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 14 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 15 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 16 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 17 is a flowchart showing the processing procedure of the medical information processing apparatus according to the first embodiment; FIG. 18 is a flowchart showing the processing procedure of the medical information processing apparatus according to the first embodiment; FIG. 19 is a flowchart showing the processing procedure of the medical information processing apparatus according to the first embodiment; FIG. 20 is a flowchart showing a processing procedure of the medical information processing apparatus according to the first embodiment; FIG. 21 is a flowchart showing the processing procedure of the medical information processing apparatus according to the first embodiment; FIG. 22 is a diagram for explaining an example of diagnostic device control processing according to the second embodiment. FIG. 23 is a diagram for explaining an example of diagnostic device control processing according to the second embodiment. FIG. 24 is a flowchart showing a processing procedure of the medical information processing apparatus according to the first embodiment;

Hereinafter, embodiments of a medical information processing system, a medical information processing apparatus, and an ultrasonic diagnostic apparatus will be described in detail with reference to the accompanying drawings. The medical image processing apparatus, X-ray diagnostic apparatus, and medical image processing program according to the present application are not limited to the embodiments described below.

(First embodiment)
First, a medical information processing system according to the first embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of a medical information processing system 1 according to the first embodiment. As shown in FIG. 1, the medical information processing system 1 according to the first embodiment includes an ultrasonic diagnostic apparatus 100, an X-ray diagnostic apparatus 200, an image storage apparatus 300, and a medical information processing apparatus 400. FIG. The ultrasonic diagnostic apparatus 100 , the X-ray diagnostic apparatus 200 , the image storage apparatus 300 and the medical information processing apparatus 400 are interconnected via the network 2 . Note that the configuration shown in FIG. 1 is merely an example, and the network 2 may be connected to other devices.

The ultrasonic diagnostic apparatus 100 operates an ultrasonic probe by an operator, generates an ultrasonic image based on ultrasonic waves transmitted and received by the ultrasonic probe, and displays the generated ultrasonic image on a display. The X-ray diagnostic apparatus 200 generates an X-ray image based on projection data acquired by irradiating a subject with X-rays, and displays the generated X-ray image on a display. Details of the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200 will be described later.

The image storage device 300 is a device that stores medical image data collected by medical image diagnostic devices such as the ultrasonic diagnostic device 100 and the X-ray diagnostic device 200 . For example, the image storage apparatus 300 acquires ultrasonic image data from the ultrasonic diagnostic apparatus 100 via the network 2 and stores the acquired ultrasonic image data in a memory provided inside or outside the apparatus. Also, for example, the image storage apparatus 300 acquires X-ray image data from the X-ray diagnostic apparatus 200 via the network 2 and stores the acquired X-ray image data in a memory provided inside or outside the apparatus. For example, the image storage device 300 is realized by computer equipment such as a server device.

The medical information processing apparatus 400 acquires medical image data via the network 2 and executes various processes using the acquired medical image data. For example, the medical information processing apparatus 400 acquires ultrasound image data and X-ray image data via the network 2, and executes various image processing and display processing. For example, the medical information processing apparatus 400 is realized by computer equipment such as a workstation. Details of the medical information processing apparatus 400 will be described later.

As long as the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200 can be connected via the network 2, the medical information processing apparatus 400 can be installed anywhere. For example, the medical information processing apparatus 400 may be installed at a location different from the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200 . That is, the network 2 may be configured by a closed local network within the hospital, or may be a network via the Internet.

Here, in this embodiment, a case where the medical information processing apparatus 400 executes various processes will be described. Specifically, the medical information processing apparatus 400 receives an ultrasonic image and an X-ray image, aligns the images, and executes various subsequent processes, whereby the ultrasonic diagnostic apparatus 100 and the X-ray image are processed. A case of improving the efficiency of a procedure using the line diagnosis apparatus 200 at the same time will be described. Various types of processing described below may be executed by any device in the medical information processing system 1, or a plurality of processing may be distributed and executed by different devices.

Details of the ultrasonic diagnostic apparatus 100, the X-ray diagnostic apparatus 200, and the medical information processing apparatus 400 will be described below. FIG. 2 is a diagram showing an example of the configuration of the ultrasonic diagnostic apparatus 100 according to the first embodiment. As shown in FIG. 2, the ultrasonic diagnostic apparatus 100 according to the first embodiment has an ultrasonic probe 100a, an input interface 100b, a display 100c, and an apparatus main body 100d. The ultrasonic probe 100a, the input interface 100b, and the display 100c are communicably connected to the device body 100d.

The ultrasonic probe 100a has a plurality of piezoelectric transducers, and these piezoelectric transducers generate ultrasonic waves based on drive signals supplied from the transmission/reception circuit 110 of the apparatus body 100d. Also, the ultrasonic probe 100a receives a reflected wave from the subject P and converts it into an electric signal. That is, the ultrasonic probe 100a scans the subject P with ultrasonic waves and receives reflected waves from the subject P. As shown in FIG. Further, the ultrasonic probe 100a has a matching layer provided on the piezoelectric transducer, a backing material, etc. for preventing the ultrasonic wave from propagating backward from the piezoelectric transducer. The ultrasonic probe 100a is detachably connected to the device main body 100d via a probe connector.

When ultrasonic waves are transmitted from the ultrasonic probe 100a to the subject P, the transmitted ultrasonic waves are successively reflected by discontinuous surfaces of acoustic impedance in the body tissue of the subject P, and are reflected as reflected wave signals from the ultrasonic probe. The signal is received by a plurality of piezoelectric vibrators of 100a. The amplitude of the received reflected wave signal depends on the difference in acoustic impedance at the discontinuity from which the ultrasonic waves are reflected. When the transmitted ultrasonic pulse is reflected by the moving blood flow or the surface of the heart wall, the reflected wave signal depends on the velocity component of the moving object in the ultrasonic transmission direction due to the Doppler effect. subject to frequency shifts.

Here, the ultrasonic probe 100a in the present embodiment is an ultrasonic probe operated on the body surface of the subject or an ultrasonic probe inserted into the body cavity of the subject and operated in the body cavity (intra-cavity probe). is. For example, the ultrasound probe 100a is operated on the body surface of the chest of the subject and can scan the left ventricle, left atrium, aorta, mitral valve, aortic valve, etc. Transthoracic echocardiography (TTE) ) probe. Also, for example, the ultrasonic probe 100a is a transesophageal echocardiography (TEE) probe that can be operated in the esophagus and scan the left ventricle, left atrium, aorta, mitral valve, aortic valve, and the like. . Note that the TEE probe can acquire three-dimensional ultrasound image data in real time.

The input interface 100b includes a trackball for setting a predetermined position (for example, a region of interest, etc.), a switch button, a mouse, a keyboard, a touchpad for performing an input operation by touching an operation surface, a display screen and a touchpad. It is realized by a touch monitor integrated with and a non-contact input circuit using an optical sensor, an audio input circuit, and the like. The input interface 100 b is connected to a processing circuit 150 to be described later, converts an input operation received from an operator into an electric signal, and outputs the electric signal to the processing circuit 150 . It should be noted that the input interface 100b in this specification is not limited to having physical operation parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to the processing circuit 150 is also included in the input interface.

The display 100c displays a GUI (Graphical User Interface) for the operator of the ultrasonic diagnostic apparatus 100 to input various setting requests using the input interface 100b, and displays ultrasonic images generated in the apparatus main body 100d. to display. Further, the display 100c displays various messages and display information in order to notify the operator of the processing status and processing results of the apparatus main body 100d. Moreover, the display 100c has a speaker and can output sound. The display 100c can also display images and the like received from the medical information processing apparatus 400. FIG.

The device main body 100d is a device that generates ultrasonic image data based on reflected wave signals received by the ultrasonic probe 100a. For example, the apparatus main body 100d generates two-dimensional ultrasonic image data based on two-dimensional reflected wave data (echo data) received by the ultrasonic probe 100a. Further, the apparatus main body 100d generates three-dimensional ultrasonic image data (volume data) based on the three-dimensional reflected wave data received by the ultrasonic probe 100a. Then, the apparatus main body 100d generates an ultrasonic image for display from the generated ultrasonic image data.

For example, the apparatus main body 100d has a transmission/reception circuit 110, a B-mode processing circuit 120, a Doppler processing circuit 130, a storage circuit 140, a processing circuit 150, and a communication interface 160, as shown in FIG. The transmission/reception circuit 110, the B-mode processing circuit 120, the Doppler processing circuit 130, the storage circuit 140, the processing circuit 150, and the communication interface 160 are communicably connected to each other. Further, the device main body 100d is connected to the network 2. FIG. Note that the network 2 is an in-hospital LAN or the like, and the ultrasonic diagnostic apparatus 100 is connected for communication with various devices connected to the in-hospital LAN via the network 2 .

The transmission/reception circuit 110 has a pulse generator, a transmission delay section, a pulser, etc., and supplies a drive signal to the ultrasonic probe 100a. A pulse generator repeatedly generates rate pulses at a predetermined rate frequency to form a transmitted ultrasound wave. In addition, the transmission delay unit focuses the ultrasonic waves generated from the ultrasonic probe 100a into a beam, and the pulse generator generates a delay time for each piezoelectric transducer necessary for determining the transmission directivity. given for each rate pulse. Also, the pulsar applies a driving signal (driving pulse) to the ultrasonic probe 100a at a timing based on the rate pulse. That is, the transmission delay unit arbitrarily adjusts the transmission direction of the ultrasonic waves transmitted from the piezoelectric transducer surface by changing the delay time given to each rate pulse.

In addition, the transmission/reception circuit 110 has a preamplifier, an A/D (Analog/Digital) converter, a reception delay unit, an adder, etc., and performs various processing on the reflected wave signal received by the ultrasonic probe 100a. Generate wave data. A preamplifier amplifies the reflected wave signal for each channel. The A/D converter A/D converts the amplified reflected wave signal. The reception delay section provides a delay time necessary for determining reception directivity. The adder adds the reflected wave signals processed by the reception delay unit to generate reflected wave data. The addition processing of the adder emphasizes the reflection component from the direction corresponding to the reception directivity of the reflected wave signal, and the reception directivity and the transmission directivity form a comprehensive beam for ultrasonic wave transmission/reception.

The B-mode processing circuit 120 receives the reflected wave data from the transmitting/receiving circuit 110, performs logarithmic amplification, envelope detection processing, etc., and generates data (B-mode data) in which the signal strength is represented by the brightness of luminance. .

The Doppler processing circuit 130 frequency-analyzes the velocity information from the reflected wave data received from the transmission/reception circuit 110, extracts blood flow, tissue, and contrast agent echo components due to the Doppler effect, and obtains moving body information such as velocity, dispersion, and power. Data extracted from multiple points (Doppler data) is generated. Specifically, the Doppler processing circuit 130 generates Doppler data at each of a plurality of sample points, such as average velocity, average dispersion value, average power value, etc., as motion information of the moving object. Here, the moving body is, for example, blood flow, tissue such as a heart wall, and a contrast agent. The Doppler processing circuit 130 uses information obtained by estimating the average velocity of blood flow, the average variance value of blood flow, the average power value of blood flow, etc. at each of a plurality of sample points as blood flow motion information (blood flow information). Generate.

The B-mode processing circuit 120 and the Doppler processing circuit 130 illustrated in FIG. 2 can process both two-dimensional reflected wave data and three-dimensional reflected wave data. That is, the B-mode processing circuit 120 generates two-dimensional B-mode data from the two-dimensional reflected wave data, and generates three-dimensional B-mode data from the three-dimensional reflected wave data. The Doppler processing circuit 130 also generates two-dimensional Doppler data from the two-dimensional reflected wave data, and generates three-dimensional Doppler data from the three-dimensional reflected wave data.

The storage circuit 140 stores control programs for transmitting/receiving ultrasonic waves, image processing, and display processing, diagnostic information (for example, patient ID, doctor's findings, etc.), various data such as diagnostic protocols and various body marks. . Also, the storage circuit 140 stores various image data. For example, storage circuitry 140 also stores data generated by B-mode processing circuitry 120 and Doppler processing circuitry 130 . The storage circuit 140 can also store reflected wave data. For example, the storage circuit 140 is implemented by a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, or an optical disk.

The processing circuitry 150 controls the entire processing of the ultrasonic diagnostic apparatus 100 . Specifically, the processing circuit 150 controls the transmission/reception circuit 110 and the B-mode processing circuit based on various setting requests input by the operator via the input interface 100b and various control programs and various data read from the storage circuit 140. 120 and controls the processing of the Doppler processing circuitry 130 . Further, the processing circuit 150 controls so that the ultrasonic image for display stored in the storage circuit 140 is displayed on the display 100c, the touch monitor of the input interface 100b, or the like.

Processing circuit 150 performs control function 151 , image generation function 152 , transmission/reception function 153 , and display control function 154 . Here, for example, each processing function executed by the control function 151, the image generation function 152, the transmission/reception function 153, and the display control function 154, which are components of the processing circuit 150, is stored in the storage circuit 140 in the form of a computer-executable program. stored in The processing circuit 150 is a processor that reads each program from the storage circuit 140 and executes it, thereby realizing functions corresponding to each program. In other words, the processing circuit 150 with each program read has each function shown in the processing circuit 150 .

The communication interface 160 is an interface for communicating with various devices connected to the network 2 . For example, the processing circuit 150 can exchange various data with the X-ray diagnostic apparatus 200, the medical information processing apparatus 400, and the like through the communication interface 160. FIG.

The control function 151 controls the entire ultrasonic diagnostic apparatus 100 . For example, the control function 151 controls the transmit/receive circuit 110, the B-mode processing circuit 120, and the Doppler processing circuit 130 to control reflected wave data collection and generation of B-mode and Doppler data. That is, the control function 151 executes two-dimensional ultrasonic scanning and three-dimensional ultrasonic scanning on the subject via the ultrasonic probe 100a. Note that the control function 151 is an example of a collection unit.

The image generation function 152 generates ultrasound image data from data generated by the B-mode processing circuit 120 and the Doppler processing circuit 130 . That is, the image generation function 152 generates two-dimensional B-mode image data representing the intensity of the reflected wave by luminance from the two-dimensional B-mode data generated by the B-mode processing circuit 120 . Further, the image generation function 152 generates two-dimensional Doppler image data representing moving body information from the two-dimensional Doppler data generated by the Doppler processing circuit 130 . The image generation function 152 can also generate M-mode image data from the time-series data of the B-mode data on one scanning line generated by the B-mode processing circuit 120 . The image generation function 152 can also generate a Doppler waveform obtained by plotting blood flow and tissue velocity information in chronological order from the Doppler data generated by the Doppler processing circuit 130 .

Here, the image generation function 152 generally converts (scan-converts) a scanning line signal train of ultrasonic scanning into a scanning line signal train of a video format typified by a television or the like, and converts the ultrasonic wave for display. Generate an image. Specifically, the image generating function 152 generates an ultrasonic image for display by performing coordinate transformation according to the scanning mode of ultrasonic waves by the ultrasonic probe 101 . In addition, the image generation function 152 performs various image processing other than scan conversion, such as image processing (smoothing processing) for regenerating a luminance average value image using a plurality of image frames after scan conversion, Image processing (edge enhancement processing) using a differential filter is performed in the image. In addition, the image generation function 152 synthesizes character information of various parameters, scales, body marks, various markers, etc. with the ultrasound image.

Furthermore, the image generation function 152 generates three-dimensional B-mode image data by performing coordinate transformation on the three-dimensional B-mode data generated by the B-mode processing circuit 120 . The image generation function 152 also generates three-dimensional Doppler image data by performing coordinate transformation on the three-dimensional Doppler data generated by the Doppler processing circuit 130 . That is, the image generating function 152 generates "three-dimensional B-mode image data or three-dimensional Doppler image data" as "three-dimensional ultrasound image data (volume data)". In addition, the image generation function 152 generates MPR images, 3D B-mode image data, and 3D Doppler image data by performing multi-planar transformation on 3D B-mode image data and 3D Doppler image data. A volume rendering image is generated by performing volume rendering processing on image data. In this specification, the ultrasonic image for display, the three-dimensional ultrasonic image data (volume data), each image generated from the volume data, and the like may be collectively referred to as an ultrasonic image.

The transmission/reception function 153 transmits, via the communication interface 160, each device (X-ray (diagnostic device 200, medical information processing device 400, etc.). Also, the transmission/reception function 153 receives images and the like from each device connected to the network 2 .

The display control function 154 controls to display the processing results of various processes such as the measurement process executed by the control function 151 on the display 100c. In addition, the display control function 154 controls to display an ultrasound image or the like generated by the image generation function 152 on the display 100c. The display control function 154 can also control to display an image or the like received from the medical information processing apparatus 400 on the display 100c.

Next, the configuration of the X-ray diagnostic apparatus according to the first embodiment will be described. FIG. 3 is a diagram showing an example of the configuration of the X-ray diagnostic apparatus 200 according to the first embodiment. As shown in FIG. 3, the X-ray diagnostic apparatus 200 according to the first embodiment includes an X-ray high voltage device 201, an X-ray tube 202, an X-ray diaphragm 203, a top plate 204, and a C-arm 205. , an X-ray detector 206, a C-arm rotating/moving mechanism 207, a top plate moving mechanism 208, a C-arm/top plate mechanism control circuit 209, an aperture control circuit 210, a processing circuit 220, and an input interface 230. , a display 240 , a memory circuit 250 and a communication interface 260 .

In the X-ray diagnostic apparatus 200 shown in FIG. 1, each processing function is stored in the storage circuit 250 in the form of a computer-executable program. The C-arm/top plate mechanism control circuit 209, the aperture control circuit 210, and the processing circuit 220 are processors that read and execute programs from the storage circuit 250 to implement functions corresponding to the respective programs. In other words, each circuit with each program read has a function corresponding to the read program.

The X-ray high voltage device 201 is a high voltage power supply that generates a high voltage under the control of the processing circuit 220 and supplies the generated high voltage to the X-ray tube 202 . The X-ray tube 202 uses a high voltage supplied from the X-ray high voltage device 201 to generate X-rays.

The X-ray diaphragm 203 narrows down the X-rays generated by the X-ray tube 202 so that the region of interest of the subject P is selectively irradiated under the control of the diaphragm control circuit 210 . For example, the X-ray diaphragm 203 has four slidable diaphragm blades. The X-ray diaphragm 203 arbitrarily changes the shape, size, and position of the aperture by sliding these diaphragm blades under the control of the diaphragm control circuit 20 . X-ray diaphragm 203 may also include additional filters for adjusting radiation quality.

The top plate 204 is a bed on which the subject P is placed, and is placed on a bed device (not shown). The X-ray detector 206 detects X-rays that have passed through the subject P. FIG. For example, the X-ray detector 206 has detection elements arranged in a matrix. Each detection element converts the X-rays that have passed through the subject P into an electric signal, accumulates the electric signal, and transmits the accumulated electric signal to the processing circuit 220 .

C-arm 205 holds X-ray tube 202 , X-ray diaphragm 203 and X-ray detector 206 . The X-ray tube 202 , the X-ray diaphragm 203 , and the X-ray detector 206 are arranged to face each other with the subject P sandwiched by the C-arm 205 . In FIG. 3, the case where the X-ray diagnostic apparatus 200 is a single plane is described as an example, but the embodiment is not limited to this, and may be a biplane. The C-arms 15 are individually rotated on a plurality of axes by actuators such as motors provided on the supports.

The C-arm rotation/movement mechanism 207 is a mechanism for rotating and moving the C-arm 205 by driving a motor or the like provided on the support. A tabletop moving mechanism 208 is a mechanism for moving the tabletop 204 . For example, the tabletop moving mechanism 208 moves the tabletop 204 using power generated by an actuator.

The C-arm/top plate mechanism control circuit 209 controls the C-arm rotating/moving mechanism 207 and the top plate moving mechanism 208 under the control of the processing circuit 220 , thereby rotating and moving the C-arm 205 and moving the top plate 204 . Adjust movement. Under the control of the processing circuit 220, the aperture control circuit 210 adjusts the opening degree of the aperture blades of the X-ray aperture 203 to change the shape, size, and position of the aperture so that the subject P is irradiated. Controls the irradiation range of X-rays.

The input interface 230 includes a trackball for setting a predetermined area (for example, ROI), a switch button, a mouse, a keyboard, a touch pad for performing an input operation by touching an operation surface, a display screen and a touch pad. is integrated with a touch screen, a non-contact input circuit using an optical sensor, a voice input circuit, and the like, and a foot switch and the like for performing X-ray irradiation and the like.

The input interface 230 is connected to the processing circuit 220 , converts an input operation received from an operator into an electric signal, and outputs the electric signal to the processing circuit 220 . It should be noted that the input interface 230 in this specification is not limited to having physical operation parts such as a mouse and a keyboard. For example, a processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs this electrical signal to the processing circuit 220 is included in the example of the input interface.

The display 240 displays a GUI for accepting operator's instructions and various images generated by the processing circuit 220 . Also, the display 240 displays various messages and display information in order to notify the operator of the processing status and processing results of the processing circuit 220 . The display 240 also has a speaker and can output sound. The display 240 can also display images received from the ultrasonic diagnostic apparatus 100 and the medical information processing apparatus 400 .

The storage circuit 250 receives and stores the image data generated by the processing circuit 220 . The storage circuit 250 also stores X-ray images generated by the processing circuit 220, volume data, images received from the ultrasonic diagnostic apparatus 100 and the medical information processing apparatus 400, and the like. The storage circuit 250 also stores programs corresponding to various functions read and executed by each circuit shown in FIG. For example, the storage circuit 250 is implemented by a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, or an optical disk.

The processing circuit 21 controls the operation of the X-ray diagnostic apparatus 200 as a whole. Specifically, the processing circuit 21 executes a collection function 221 , an image processing function 222 , a transmission/reception function 223 and a control function 224 . The acquisition function 221 generates image data using electrical signals converted from X-rays by the X-ray detector 206 and stores the generated image data in the storage circuit 250 . For example, the acquisition function 221 performs current/voltage conversion, A (Analog)/D (Digital) conversion, and parallel/serial conversion on the electrical signal received from the X-ray detector 206, and converts the electrical signal into , respectively. The acquisition function 221 then stores the generated projection data in the storage circuit 250 . Note that the acquisition function 221 can also reconstruct reconstruction data (volume data) using projection data acquired by rotational imaging, and store the reconstructed volume data in the storage circuit 250 .

The image processing function 222 controls image processing, analysis processing, and the like for projection data. For example, the image processing function 222 generates an X-ray image by performing various image processing on projection data stored in the storage circuit 250 . Alternatively, the image processing function 222 directly acquires projection data from the acquisition function 221 and generates an X-ray image by performing various image processing on the acquired projection data.

The image processing function 222 can also store the processed X-ray image in the storage circuit 250 . For example, the image processing function 222 can perform various processes using image processing filters such as moving average (smoothing) filters, Gaussian filters, median filters, recursive filters, and bandpass filters. Furthermore, the image processing function 222 can also generate a three-dimensional image from volume data. In this specification, the X-ray image, the reconstruction data (volume data), the three-dimensional image generated from the volume data, and the like may be collectively referred to as an X-ray image.

The transmission/reception function 223 transmits the X-ray image to each device (ultrasonic diagnostic device 100, medical information processing device 400, etc.) connected to the network 2 via the communication interface 260. FIG. Also, the transmission/reception function 153 receives images and the like from each device connected to the network 2 .

The control function 224 controls the X-ray high-voltage device 201 according to the operator's instructions transferred from the input interface 230, and adjusts the voltage supplied to the X-ray tube 202 so that the subject P is irradiated. Controls X-ray dosage and ON/OFF. Further, for example, the control function 224 controls the C-arm/top plate mechanism control circuit 209 according to the operator's instructions, and adjusts the rotation and movement of the C-arm 205 and the movement of the top plate 204 .

In addition, the control function 224 controls the aperture control circuit 210 in accordance with an operator's instruction, and adjusts the aperture blades of the X-ray aperture 203 to irradiate the subject P with X-rays. Control range. Further, the control function 224 controls the display 240 to display a GUI for receiving an operator's instruction, an image stored in the storage circuit 250, a processing result by the processing circuit 220, and the like. The control function 224 also controls the display 240 to display images received from the ultrasonic diagnostic apparatus 100 and the medical information processing apparatus 400 .

The communication interface 260 is an interface for communicating with various devices connected to the network 2 . For example, the processing circuit 220 can exchange various data with the ultrasonic diagnostic apparatus 100, the medical information processing apparatus 400, and the like through the communication interface 260. FIG.

Next, the configuration of the medical information processing apparatus 400 according to the first embodiment will be described. FIG. 4 is a diagram showing an example of the configuration of the medical information processing apparatus 400 according to the first embodiment. As shown in FIG. 4, the medical information processing apparatus 400 according to the first embodiment includes a communication interface 410, a memory circuit 420, an input interface 430, a display 440, and a processing circuit 450. Connected.

A communication interface 410 is an interface for communicating with various devices connected to the network 2 . For example, the processing circuit 450 can exchange various data with the ultrasonic diagnostic apparatus 100, the X-ray diagnostic apparatus, or the like through the communication interface 410. FIG.

The storage circuit 420 stores control programs for performing various processes in the medical information processing apparatus 400, various images, and the like. For example, the storage circuit 420 stores ultrasonic images received from the ultrasonic diagnostic apparatus 100 and X-ray images received from the X-ray diagnostic apparatus 200 . Also, for example, the storage circuit 420 stores the processing result by the processing circuit 450 . Data stored in the storage circuit 22 is transmitted to the ultrasonic diagnostic apparatus 100, the X-ray diagnostic apparatus 200, etc. via the network 2. FIG. For example, the storage circuit 420 is implemented by a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, or an optical disk.

The input interface 430 includes a trackball, a switch button, a mouse, a keyboard, a touch pad for performing input operations by touching the operation surface, a touch monitor in which the display screen and the touch pad are integrated, and an optical sensor. It is realized by a non-contact input circuit using , a voice input circuit, and the like. The input interface 430 is connected to the processing circuit 450 , converts an input operation received from an operator into an electric signal, and outputs the electric signal to the processing circuit 450 . It should be noted that the input interface 430 in this specification is not limited to having physical operation parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to the processing circuit 450 is also included in the input interface.

The display 440 displays a GUI for the operator of the medical information processing apparatus 400 to input various requests using the input interface 430, and various messages for notifying the operator of the processing results of the processing circuit 450. and display information. Also, the display 440 displays an ultrasonic image received from the ultrasonic diagnostic apparatus 100, an X-ray image received from the X-ray diagnostic apparatus, and the like. The display 440 also has a speaker and can output sound.

The processing circuit 450 controls the entire processing of the medical information processing apparatus 400 . Specifically, the processing circuit 450 executes a control function 451 , a transmission/reception function 452 , an alignment function 453 , an image processing function 454 and a control information generation function 455 .

Here, for example, each processing function executed by the control function 451, the transmission/reception function 452, the alignment function 453, the image processing function 454, and the control information generation function 455, which are components of the processing circuit 450, can be executed by a computer. stored in the storage circuit 420 in the form of a program. The processing circuit 450 is a processor that reads each program from the storage circuit 420 and executes it, thereby implementing functions corresponding to each program. In other words, the processing circuit 450 with each program read has each function shown in the processing circuit 450 .

The control function 451 controls the entire medical information processing apparatus 400 . For example, the control function 451 controls various processes corresponding to various setting requests input by the operator via the input interface 430 . The control function 451 also displays an ultrasonic image received from the ultrasonic diagnostic apparatus 100, an X-ray image received from the X-ray diagnostic apparatus 200, an image generated by the processing of the processing circuit 450, and the like on the display 440. to control. For example, the control function 451 causes the display 440 to display moving images of real-time ultrasound images and real-time X-ray images. In addition, the control function 451 causes the transmission/reception function 452 to transmit an ultrasonic image, an X-ray image, an image generated by the processing of the processing circuit 450, and the like to the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200. , displays images on the display 100c of the ultrasonic diagnostic apparatus 100 and the display 240 of the X-ray diagnostic apparatus 200, and causes each apparatus to execute processing.

The transmission/reception function 452 receives ultrasound images and X-ray images via the communication interface 410 . Further, the transmission/reception function 452 transmits images generated by the processing of the processing circuit 450, control information, and the like to the ultrasonic diagnostic apparatus 100, the X-ray diagnostic apparatus 200, and the like.

Alignment function 453 performs alignment between the ultrasound and x-ray images received by transmit/receive function 452 . Also, the image processing function 454 performs various image processing on the ultrasonic image and the X-ray image received by the transmission/reception function 452 . For example, the image processing function 454 performs image processing on three-dimensional ultrasound image data (volume data) received from the ultrasound diagnostic apparatus 100 to generate MPR images, volume rendering images, and the like. Note that the control function 451 can display these images in real time. Also, the control information generation function 455 generates control information for controlling the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200 . The details of each of these functions will be described later.

Here, the control function 451 is an example of an acquisition unit, a display control unit, and a control unit. Also, the alignment function 453 is an example of an alignment unit. Also, the image processing function 454 is an example of a generator. Also, the control information generation function 455 is an example of a determination unit.

The configurations of the ultrasonic diagnostic apparatus 100, the X-ray diagnostic apparatus 200, and the medical information processing apparatus 400 have been described above. With this configuration, processing by each function of the processing circuit 450 improves the efficiency of the procedure using the ultrasonic diagnostic apparatus 100 and the X-ray diagnostic apparatus 200 at the same time. Specifically, the processing circuit 450 performs various processes for image display, support information display, and diagnostic apparatus control after positioning the ultrasound image and the X-ray image, thereby improving the efficiency of the procedure. Improve.

Here, in the method of the present application, an ultrasonic image is acquired by the ultrasonic diagnostic apparatus 100 and an X-ray image is acquired by the X-ray diagnostic apparatus 200 at the same time, and various procedures are performed while observing these images. can be applied to For example, the techniques of the present application can be applied to treatments such as heart valves using medical devices. An example of a technique to which the technique of the present application is applied will be described below with reference to FIG. FIG. 5 is a diagram for explaining an example of a technique according to the first embodiment;

For example, the present techniques can be applied to mitral valve treatment using a medical device, as shown in FIG. In such a procedure, an ultrasonic diagnostic apparatus 100 having an ultrasonic probe (TEE probe) that is inserted into the esophagus to transmit and receive ultrasonic waves in the esophagus acquires an ultrasonic image, and at the same time an X-ray image is acquired by the X-ray diagnostic apparatus 200. is collected. Here, the operator who performs the procedure advances the medical device while observing real-time ultrasonic images and real-time X-ray images, penetrates the atrial septum indicated by region R1, and penetrates the mitral valve indicated by region R2. to reach the medical device. In such a procedure, the operator observes an X-ray image that can visualize a wide area around the heart to grasp the positional relationship between the medical device and the target site, and an ultrasonography that more clearly visualizes the target site. Accurate treatment can be performed by observing the sound wave image.

In the method of the present application, in order to improve the efficiency of such a procedure, after aligning the ultrasound image and the X-ray image, various processes are performed for image display, support information display, and diagnostic apparatus control. Details of the various processes are described below. Image display, support information display, and diagnostic apparatus control will be described in order below.

First, alignment according to this embodiment will be described. Any method may be used for the alignment according to the present embodiment as long as it is a method capable of aligning the ultrasound image and the X-ray image. For example, the alignment function 453 can perform alignment using a three-dimensional model showing the structure of the ultrasound probe 100a. In such a case, the positioning function 453 acquires the X-ray image by matching the ultrasonic probe 100a drawn in the received X-ray image with model diagrams obtained by projecting the 3D model from various directions. The position and orientation of the ultrasonic probe 100a in the X-ray coordinate system thus obtained are identified. Then, the alignment function 453 aligns the ultrasound image and the X-ray image by associating the ultrasound coordinate system and the X-ray coordinate system of the ultrasound image acquired by the ultrasound probe 100a. .

The example of alignment described above is merely an example, and various other alignments may be performed. For example, the registration function 453 extracts anatomical feature points from the ultrasound image and the X-ray image, and aligns the ultrasound image and the X-ray image based on the extracted anatomical feature points. Execute. For example, the registration function 453 combines anatomical feature points included in the three-dimensional ultrasound image data (volume data) acquired by the TEE probe and anatomical feature points included in the X-ray image. By matching , the ultrasound coordinate system and the X-ray coordinate system are associated with each other. Alternatively, the ultrasonic probe 100a may be provided with a position sensor capable of detecting the position and orientation within the magnetic field, and alignment may be performed using this position sensor.

Here, when both images are used for registration of an ultrasound image and an X-ray image, the registration function 453 uses a group of ultrasound images acquired over time in real time and a set of ultrasound images acquired over time in real time. Images acquired at approximately the same time are extracted from the X-ray image. That is, the alignment function 453 extracts images acquired at approximately the same time from the ultrasound image group (volume data group) and the X-ray image group sequentially received by the transmission/reception function 452, and extracts the extracted images. Perform alignment using .

For example, when the control function 451 displays moving images of ultrasound images and X-ray images acquired in real time on the display 440, the operator starts alignment via the input interface. , the alignment function 453 extracts images acquired at approximately the same time from the group of ultrasound images (group of volume data) and the group of X-ray images, and uses the extracted images to Perform alignment. Here, the control function 451 causes the display 440 to continuously display the moving images of the ultrasound images and X-ray images that are continuously acquired in real time. That is, the registration function 453 can perform the registration of the ultrasound and x-ray images in the background.

Note that the alignment function 453 can perform real-time alignment. For example, when the ultrasonic probe 100a moves, real-time alignment is performed again. In such a case, for example, the alignment function 453 tracks the position of the ultrasonic probe 100a in the X-ray image acquired in real time, and if the position of the ultrasonic probe 100a moves, the alignment is performed again. I do.

As described above, the alignment function 453 aligns the ultrasound coordinate system with the X-ray coordinate system by performing alignment between the ultrasound image and the X-ray image. The control function 451, the image processing function 454, and the control information generation function 455 execute various processes such as image display, support information display, and diagnosis apparatus control using the alignment result.

(image display)
Image display processing according to the first embodiment will be described below. For example, the control function 451 displays an ultrasonic image and an X-ray image in substantially the same direction side by side based on the registration result so that they can be identified as being in substantially the same direction. If a real-time ultrasonic image acquired by the ultrasonic diagnostic apparatus 100 and a real-time X-ray image acquired by the X-ray diagnostic apparatus 200 are simply displayed, images in different directions are likely to be displayed. Therefore, the control function 451 according to the present embodiment uses the alignment result to display the ultrasonic image and the X-ray image in substantially the same direction side by side.

In such a case, for example, the image processing function 454 sequentially generates ultrasonic images in substantially the same direction as the X-ray images from the three-dimensional ultrasonic image data sequentially acquired over time based on the alignment result. Then, the control function 451 displays the sequentially generated ultrasonic images and the sequentially acquired X-ray images in parallel. Here, the control function 451 can also display the ultrasonic image and the X-ray image on the display 440 of its own apparatus, but the display 240 of the X-ray diagnostic apparatus 200 and the display 100c of the ultrasonic diagnostic apparatus 100 can also be displayed. You can also let

FIG. 6A is a diagram showing an example of image display according to the first embodiment. As shown in FIG. 6A, the control function 451 causes the display 240 to display, for example, an ultrasound image clearly showing the treatment target and an X-ray image in substantially the same direction as the ultrasound image. Here, the control function 451 causes the two images to be identifiably displayed as images in substantially the same direction. For example, as shown in FIG. 6A, the control function 451 causes the ultrasound image and the X-ray image to be displayed side by side to indicate that both images are images in substantially the same direction. Note that the method for indicating that both images are images in substantially the same direction is not limited to the example described above, and may be, for example, a case of displaying a symbol, a mark, or a character.

In addition, the control function 451 displays both images side by side so that the sizes of both images are substantially the same. That is, as shown in FIG. 6A, the control function 451 adjusts and displays the size of the ultrasonic image or the X-ray image so that the sizes of the medical devices are substantially the same. As a result, for example, when the medical device is operated, the medical device is moved in the same manner in both images, and images that are easy to observe can be displayed.

In the above example, a case has been described in which an ultrasonic image in substantially the same direction as the X-ray image is generated from the three-dimensional ultrasonic image data. However, the embodiment is not limited to this, and the X-ray diagnostic apparatus 200 may be controlled to acquire an X-ray image in substantially the same direction as the ultrasonic image. In such a case, for example, the control information generation function 455 generates control information for controlling the C-arm 205 so as to acquire an X-ray image in substantially the same direction as the ultrasonic image based on the alignment result. .

For example, the control information generation function 455 generates information indicating how much and in which direction the C-arm 205 should be moved. The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the X-ray diagnostic apparatus 200 via the transmission/reception function 452 . When the X-ray diagnostic apparatus 200 receives the control information from the medical information processing apparatus 400, the X-ray diagnostic apparatus 200 presents to the operator whether or not to execute control based on the control information. The C-arm 205 is moved based on.

Also, in the above example, the case of displaying a set of an ultrasonic image and an X-ray image has been described. However, if the X-ray diagnostic apparatus 200 is biplane, it may be a case of displaying a set of an ultrasonic image and an X-ray image. 6B is a diagram illustrating an example of image display according to the first embodiment; FIG. For example, as shown in FIG. 6B, the control function 451 causes an ultrasonic image and an X-ray image in substantially the same direction to be displayed side by side. That is, the image processing function 454 generates, from the three-dimensional ultrasound image data, two ultrasound images in substantially the same direction with respect to the two X-ray images acquired by biplane. The control function 451 causes the ultrasound image and the X-ray image in substantially the same direction to be displayed adjacent to each other.

It should be noted that the control function 451 may display not only a set of an ultrasound image and an X-ray image, but also only one of the images. For example, control function 451 may cause an ultrasound image to be displayed in addition to the set of ultrasound and x-ray images shown in FIG. 6A. In one example, the image processing function 454 further generates ultrasound images in a direction orthogonal to the direction of the images shown in FIG. In addition to displaying the generated ultrasound image.

Next, an example of displaying a synthesized image obtained by synthesizing information of an ultrasonic image on an X-ray image will be described. For example, the image processing function 454 generates a plurality of composite images each showing a plurality of pieces of information based on the ultrasound image on the X-ray image based on the registration result. Then, the control function 451 switches and displays a plurality of synthesized images according to a switching operation by the operator via the input interface 430 . FIG. 7 is a diagram showing an example of display of a synthesized image according to the first embodiment.

For example, as shown in the upper diagram of FIG. 7, the image processing function 454 generates a composite image showing an X-ray image of a region R3 indicating the scanning range of the ultrasonic probe 100a. In such a case, the image processing function 454 acquires from the ultrasonic diagnostic apparatus 100 information on the scan range of the ultrasonic probe 100a. Then, the image processing function 454 generates a composite image showing the region R3 in the X-ray image based on the acquired information of the scanning range and the alignment result.

Also, for example, the image processing function 454 generates a composite image by combining the ultrasound image I1 with the corresponding position of the X-ray image based on the alignment result, as shown in the middle diagram of FIG. In such a case, the image processing function 454 generates an ultrasonic image I1 in substantially the same direction as the X-ray image based on the alignment result, and synthesizes the generated ultrasonic image I1 at the corresponding position of the X-ray image. do.

Further, for example, the image processing function 454 composites a mark M1 indicating a treatment target, passing points of the medical device, etc. on the X-ray image based on the alignment result, as shown in the lower diagram of FIG. Generate a composite image. In such a case, first, the operator sets a mark M1 on a treatment target or the like on the ultrasonic image displayed in real time via the input interface 430 . The image processing function 454 identifies the position in the X-ray image of the mark M1 set on the ultrasound image based on the alignment result. Then, the image processing function 454 synthesizes information indicating the mark M1 at the specified position on the X-ray image.

As described above, when the image processing function 454 generates a plurality of composite images, the control function 451 causes the display 440 to display the generated composite images according to the operator's switching operation via the input interface 430 .

Next, an example of displaying a composite image in which the orientation of the ultrasonic image is identifiable will be described. For example, when synthesizing the scan range of an ultrasound image with an X-ray image, or when synthesizing an ultrasound image with an X-ray image, it becomes difficult to identify the orientation of the ultrasound image (front and back of the ultrasound image). Sometimes. Therefore, when synthesizing information based on an ultrasound image with an X-ray image, the control function 451 displays a synthesized image in which the orientation of the ultrasound image is identifiable. FIG. 8 is a diagram showing an example of display of a synthesized image according to the first embodiment.

For example, as shown in FIG. 8, the control function 451 displays a composite image obtained by synthesizing biplane scan sections P1 and P2 of the ultrasonic probe 100a with an X-ray image, and displays an ultrasonic image corresponding to each section. In this case, each ultrasonic image is displayed so that the orientation of the cross section can be identified. For example, as shown in FIG. 8, the control function 451 displays the left and right edges of the scan slice P1 and the left and right edges of the scan slice P2 in different colors. This allows the operator to recognize at a glance in which direction each displayed ultrasound image was scanned. The information for identifying the orientation of the ultrasonic image is not limited to colors, and may be symbols, marks, characters, or the like, for example. Also, in the example of FIG. 8, the case where both ends of the scanned cross section are colored is shown, but either one of them may be colored or marked.

Here, since the positioning function 453 according to the present embodiment can perform positioning in real time, the movement of the ultrasonic probe 100a and the change in the scanning range can be followed, and the scanning range and the ultrasonic wave in the synthesized image can be adjusted. You can change the position of the image. For example, the positioning function 453 tracks the position of the ultrasonic probe 100a in an X-ray image acquired in real time, and when the position of the ultrasonic probe 100a is changed by being operated, or when the ultrasonic wave transmitting/receiving plane changes. If the orientation of is mechanically changed, the alignment is performed again. The image processing function 454 changes the scan range and the position of the ultrasonic image in the synthesized image based on the re-executed alignment result.

In addition, the image processing function 454 receives from the ultrasonic diagnostic apparatus 100 information on a change in the scan range due to a change in the direction of the ultrasonic beam, and based on the received information, determines the scan range in the composite image and the ultrasonic wave. Change the position of the sonar image.

If the alignment process by the alignment function 453 fails, the control function 451 hides the composite image or displays information indicating that the alignment has failed.

Next, an example of selecting an ultrasonic image to be combined with an X-ray image from ultrasonic images of a plurality of scan cross-sections will be described. For example, when ultrasound images are acquired in a plurality of scan planes, the image processing function 454 selects a predetermined plane from the group of ultrasound images acquired in the plurality of planes, and converts the selected plane into an X-ray image. Generate a composite image that is composited above. FIG. 9 is a diagram for explaining an example of selection of an ultrasound image to be synthesized with a synthesized image according to the first embodiment. Note that FIG. 9 shows a case where two cross-sectional ultrasound images are acquired in a biplane.

For example, as shown in FIG. 9, the image processing function 454 selects planes that do not overlap in the depth direction from the scan slices P1 and P2, and generates a composite image by combining the selected planes. For example, as shown in FIG. 9, the image processing function 454 can perform only the scan plane P1, only the scan plane P2, only the area in front of the position where the planes intersect between the scan planes P1 and P2, or Only the region on the far side from the intersection of the cross sections P1 and P2 is selected. Then, the image processing function 454 generates a composite image by combining the ultrasound image corresponding to the selected slice with the corresponding position of the X-ray image. In addition, in FIG. 9, two cross sections are described as an example, but the embodiment is not limited to this, and three or more cross sections may be selected.

As described above, the image processing function 454 can generate a composite image by combining the X-ray image and the ultrasound image based on the alignment results. Here, when generating a composite image, if images collected at different times are combined, a composite image that is difficult to observe will be generated. Therefore, the image processing function 454 selects an ultrasonic image and an X-ray image at approximately the same time from the group of ultrasonic images acquired over time and the group of X-ray images acquired over time to generate a composite image. do.

In such a case, for example, the image processing function 454 refers to the time stamps attached to the ultrasonic image and the X-ray image, selects the ultrasonic image and the X-ray image at approximately the same time, and generates a composite image. do. Also, for example, the image processing function 454 acquires the difference in acquisition time between the received images before the procedure, and selects an ultrasonic image and an X-ray image at approximately the same time based on the acquired difference. Generate a composite image. Alternatively, for example, the image processing function 454 generates electrocardiograms from a group of ultrasound images acquired over time and a group of X-ray images acquired over time, and based on the deviation of the generated electrocardiograms, the collection time is specified, and based on the specified deviation, an ultrasonic image and an X-ray image at approximately the same time are selected to generate a composite image.

Moreover, when the target is the heart, since the movement of the heart stops during diastole and systole, either the ultrasound image or the X-ray image at that time may be used. That is, by using the image when the motion has stopped, the influence of the shift can be reduced. In such a case, the image processing function 454 sequentially acquires diastolic or systolic images for either the ultrasound image or the X-ray image, and combines the acquired images with the other image containing all phases. Generate a synthesized composite image.

In the above example, the case of selecting an ultrasonic image and an X-ray image at approximately the same time to generate a composite image has been described. It is also possible to generate a composite image without performing. Further, when the real-time moving image display is paused to display a still image, an ultrasonic image and an X-ray image at approximately the same time may be selected and displayed.

(Support information display)
Next, support information display processing according to the first embodiment will be described. For example, a planned line (path) may be set when a medical device reaches a treatment target or the like. In this case, there are many cases where the planning line is set on an ultrasonic image in which the treatment target is depicted more clearly. Therefore, the control function 451 acquires information on the planned line of the procedure set on the ultrasound image, and displays the planned line on the X-ray image acquired during the procedure based on the registration result.

FIG. 10 is a diagram illustrating an example of display of plan lines according to the first embodiment. For example, the control function 451 displays an ultrasound image and an X-ray image, respectively, as shown in the upper diagram of FIG. The operator sets the plan line on the ultrasound image via the input interface 430 . Here, the operator can set the blood vessel that is the path of the medical device, points on the path (passing points), the position of the treatment target, etc. on the ultrasound image generated from the three-dimensional ultrasound image data. , to set the 3D feature line.

The control function 451 identifies the position in the X-ray coordinate system of the three-dimensional planning line set in the ultrasound image based on the registration result. Then, the control function 451 displays the plan line on the X-ray image as shown in the lower diagram of FIG. 10 based on the specified position of the plan line.

Here, the control function 451 tracks the positions of the blood vessels set on the ultrasonic image, the points on the route (passing points), and the treatment target using the ultrasonic images acquired in real time by pattern matching or the like. Accordingly, even when the display cross section of the ultrasonic image is changed or the ultrasonic probe 100a is moved, the planning line can be moved in conjunction with it. That is, the control function 451 can also synchronize the planned lines in the X-ray image by detecting the movement of the planned lines in the ultrasonic image.

Next, an example of displaying information indicating the positional relationship between the planned line and the X-ray irradiation direction will be described. That is, the control function 451 displays the positional relationship between the planning line set on the ultrasonic image and the current X-ray irradiation direction. For example, the control function 451 displays information about the angle between the direction of the planning line and the X-ray irradiation direction of the X-ray image acquired during the procedure. Here, the angle formed by the direction of the planning line and the irradiation direction of X-rays will be described. FIG. 11A is a diagram for explaining an angle formed between the direction of the planning line and the irradiation direction of X-rays according to the first embodiment;

As shown in FIG. 11A, the plan line set on the ultrasound image is a three-dimensional plan line. On the other hand, since the X-ray image is two-dimensional, the X-ray irradiation direction (projection direction) should be orthogonal to the planned line in order to more accurately depict the planned line on the X-ray image. desirable. That is, it is desirable that the angle formed by the direction of the planning line and the direction of projection be "90°". Therefore, the control function 451, as information on the angle formed by the direction of the planned line and the irradiation direction of X-rays, provides the angle formed by the planned line set on the ultrasonic image and the current irradiation direction of X-rays as follows: Information indicating how many times the angle is deviated from "90°" is displayed.

11B is a diagram illustrating an example of support information display according to the first embodiment; FIG. For example, the control function 451 calculates, for each position of the planning line, how much the angle formed by the planning line and the X-ray irradiation direction deviates from "90°" based on the alignment result. Then, as shown in FIG. 11B, the control function 451 displays the deviation for each position of the planned line as a numerical value (eg, 5 degrees, 10 degrees, etc.). Here, the control function 451 can calculate and display the deviation for each position of the planned line each time the alignment function 453 performs real-time alignment.

Further, for example, the control function 451 can also hide the planning line in the X-ray image when the angle formed by the planning line and the X-ray irradiation direction deviates from "90°". FIG. 11C is a diagram showing an example of support information display according to the first embodiment. For example, as shown in FIG. 11C, the control function 451 hides the planning line at a position where the angle formed by the planning line and the X-ray irradiation direction deviates from "90°" by a predetermined angle or more.

Also, for example, the control function 451 detects the position of the medical device included in the ultrasound image, and determines how many degrees the angle formed by the planned line at the detected position and the X-ray irradiation direction deviates from "90°". can also be displayed.

Furthermore, the control function 451 acquires information on the orientation of the medical device included in the ultrasound image, and based on the alignment result, determines the orientation of the medical device and the X-ray orientation of the X-ray image acquired during the procedure. It is also possible to display information about the angle with respect to the irradiation direction. FIG. 11D is a diagram showing an example of support information display according to the first embodiment. For example, the control function 451 detects the position of the medical device from the ultrasonic image by pattern matching or the like, and identifies the longitudinal direction of the medical device. The control function 451 then displays how many degrees the angle between the longitudinal direction of the medical device and the X-ray irradiation direction deviates from “90°”. For example, the control function 451 causes the displacement of the tip of the medical device to be displayed numerically (1 degree), as shown in FIG. 11D.

As described above, the control function 451 numerically displays information about the angle between the direction of the planning line and the direction of X-ray irradiation, and information about the angle between the direction of the medical device and the direction of X-ray irradiation. can be done. It should be noted that the control function 451 can express the information about the angle not only numerically but also graphically or in color. For example, the control function 451 indicates the color of the plan line in blue when the deviation is within 5 degrees, and indicates the color of the plan line in red when the deviation is 10 degrees or more. Also, for example, the control function 451 indicates a predetermined figure at a position where the deviation is 10 degrees or more.

(Diagnostic device control)
Next, diagnostic device control processing according to the first embodiment will be described. As described above, the control function 451 determines by how many degrees the angle formed by the planning line and the X-ray irradiation direction deviates from "90°", and the angle formed by the longitudinal direction of the medical device and the X-ray irradiation direction. It is possible to calculate and display how many degrees it deviates from "90°". Therefore, the control function 451 controls the X-ray diagnostic apparatus 200 using this information.

In such a case, for example, the control information generation function 455 generates control information for controlling the C-arm 205 so that the deviation calculated by the control function 451 is "0°". That is, the control information generation function 455 determines the X-ray irradiation direction for the object to be a direction orthogonal to the direction of the planned line. The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the X-ray diagnostic apparatus 200 via the transmission/reception function 452 . When the X-ray diagnostic apparatus 200 receives the control information from the medical information processing apparatus 400, the X-ray diagnostic apparatus 200 presents to the operator whether or not to execute control based on the control information. The C-arm 205 is moved based on. Note that the control function 451 can also control the X-ray diagnostic apparatus 200 to automatically move the C-arm 205 in response to receiving control information.

11E is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. For example, when the C-arm 205 is controlled by the control information for setting the angle between the planned line and the X-ray irradiation direction to "90°", the planned line displayed in the X-ray image after control is as shown in the figure. The longest appears, as shown in 11E.

As in the above example, the control function 451 can also control the angle between the longitudinal direction of the medical device and the X-ray irradiation direction to be "90°". Here, when controlling the angle formed by the longitudinal direction of the medical device and the X-ray irradiation direction to be "90°", the control function 451 interlocks with the movement of the medical device to change the X-ray irradiation direction. It is also possible to control to change In such a case, the control function 451 calculates by how many degrees the angle between the direction of the tip region of the medical device and the X-ray irradiation direction deviates from "90°" each time the medical device is moved. The control information generation function 455 generates control information for setting the deviation to “0°” every time the control function 451 calculates the deviation. Each time control information is generated by the control information generation function 455 , the generated control information is transmitted to the X-ray diagnostic apparatus 200 via the transmission/reception function 452 . Here, the control function 451 controls the X-ray diagnostic apparatus 200 to automatically move the C-arm 205 in response to the reception of the control information, thereby automatically moving the medical device so as to eliminate the displacement caused by the movement. , the C-arm 205 can be moved. Note that the distal end region of the medical device is, for example, a region extending from the distal end of the medical device to a position separated by a predetermined distance.

Next, an example of controlling the diagnostic device so that the planned line faces a predetermined direction will be described. For example, the control information generation function 455 determines the X-ray irradiation direction with respect to the subject so that the planned line is shown horizontally or vertically on the X-ray image. 11F is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. For example, the control information generation function 455 generates control information for controlling the C-arm 205 so that the plan line is horizontal on the X-ray image as shown in FIG. 11F. The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the X-ray diagnostic apparatus 200 via the transmission/reception function 452 .

Next, an example of controlling the position where the region of interest is displayed will be described. For example, the control function 451 controls acquisition of the X-ray image group so that the region of interest is continuously displayed at a predetermined position in the X-ray image in the X-ray image group acquired over time during the procedure. 11G is a diagram for explaining an example of diagnostic device control according to the first embodiment; FIG. For example, as shown in FIG. 11G, the control function 451 controls the angle of the C-arm 205, the detection position of the X-ray detector 206, or the X-ray detection position so that the region of interest is always positioned in the center of the screen during the procedure. Controls image rotation, enlargement/reduction, display position, etc.

Next, an example of determining the irradiation direction of X-rays based on information about the medical device and information about the target site of the procedure will be described. In such a case, the control function 451 provides information on the medical device to be inserted into the body of the subject and information on the target site of the procedure using the medical device based on the ultrasound image collected from the subject. get. For example, the control function 451 acquires information on a planned line for operating the medical device set in the ultrasound image by the operator via the input interface 430 and a treatment site. Then, the control information generation function 455 acquires an X-ray image during the procedure using the medical device for the target site based on the alignment result of the ultrasound image and the X-ray image acquired together with the ultrasound image. The X-ray irradiation direction for the actual subject is determined. Furthermore, the control information generation function 455 generates control information for irradiating X-rays from the determined irradiation direction.

In one example, the control function 451 obtains path information of the medical device to the heart valve and shape information of the heart valve based on the ultrasound image. For example, when the operator sets the path of the medical device to the heart valve and the shape of the heart valve on the ultrasound image, the control function 451 acquires the set path and the position and shape of the heart valve. The control information generation function 455 determines the irradiation direction of the X-ray so that the heart valve is horizontally oriented and the opening and closing of the valve is depicted on the X-ray image.

FIG. 12 is a diagram for explaining an example of diagnostic device control according to the first embodiment. For example, as shown in FIG. 12, the control information generation function 455 generates control information for irradiating X-rays in a direction in which the heart valve is horizontal and the opening and closing of the valve is visualized. The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the X-ray diagnostic apparatus 200 via the transmission/reception function 452 . When the X-ray diagnostic apparatus 200 receives the control information from the medical information processing apparatus 400, the X-ray diagnostic apparatus 200 presents to the operator whether or not to execute control based on the control information. The C-arm 205 is moved based on. Note that the control function 451 can also control the X-ray diagnostic apparatus 200 to automatically move the C-arm 205 in response to receiving control information.

In addition, for example, the control function 451 acquires information on the suturing plan line of the heart valve by the medical device based on the ultrasound image. For example, the control function 451 acquires suturing plan line information for suturing a heart valve set in an ultrasound image. The control information generation function 455 determines the irradiation direction of X-rays so that the overlap of suture planning lines on the X-ray image is minimized. That is, the control information generation function 455 generates control information for irradiating X-rays from a direction that minimizes the overlap of suture planning lines based on the alignment result.

Also, for example, the control function 451 acquires information on the path of the medical device to the chordae tendineae of the heart valve and information on the shape of the chordae tendineae, based on the ultrasound image. The control information generation function 455 determines the X-ray irradiation direction so that the overlap of the chordae tendineae on the X-ray image is minimized. That is, the control information generation function 455 generates control information for irradiating X-rays from a direction in which the tendinae to be treated do not overlap other tendinea tendineae, based on the alignment result.

Also, for example, the control function 451 acquires information on the shape of the medical device and information on the shape of the target site. The control information generation function 455 determines the X-ray irradiation direction for the target site based on the placement state of the medical device for the target site. FIG. 13 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 13 shows a procedure for placing a medical device in the left atrial appendage (LAA).

For example, the control function 451 identifies, based on the ultrasound image, that the shape of the device and the shape of the ostium of the left atrial appendage are circular, as shown in FIG. When a circular medical device is placed in a circular target site (left atrial appendage), the control information generation function 455 generates X-rays from a direction orthogonal to a plane in which the circularity of the left atrial appendage is lowest. Determine the X-ray irradiation direction for the left atrial appendage as it is irradiated.

For example, the control information generation function 455 calculates the circularity of the entrance of the left atrial appendage in each direction indicated by arrows a1 to a9 in FIG. ). Then, the control information generation function 455 generates control information for irradiating X-rays from a direction (arrow a5) perpendicular to the plane including the specified direction. By irradiating X-rays from the direction of arrow a5 in this way, the gap between the entrance of the left atrial appendage and the medical device can be clearly observed in the collected X-ray image, as shown in FIG. becomes possible.

The determination of the X-ray irradiation direction using the circularity described above can be used not only for procedures on the left atrial appendage, but also for placement of a medical device in the interatrial septum, for example. FIG. 14 is a diagram for explaining an example of diagnostic device control according to the first embodiment. FIG. 14 shows a procedure for placing a medical device in a defective part of the interatrial septum. Similarly, in the procedure shown in FIG. 14, the control information generation function 455 calculates the degree of circularity of the defective portion of the interatrial septum from various directions, and selects the direction with the largest deviation from the circle among the calculated degrees of circularity. Identify. Then, the control information generation function 455 generates control information for irradiating X-rays from a direction orthogonal to a plane including the specified direction.

Next, an example using blood flow information in addition to information about the medical device will be described. In such a case, the control function 451 acquires position information of the medical device and blood flow information for each position in the target site based on the ultrasound image. For example, control function 451 identifies the location of an indwelling medical device and identifies backflow of blood around it. In one example, control function 451 identifies the location and shape of regurgitation based on Doppler images collected about the periphery of the medical device.

Here, if there is backflow, the control information generation function 455 identifies the direction in which the backflow is most visible, and generates control information for X-ray irradiation from the identified direction. Further, for example, when there are multiple backflows, the control information generation function 455 identifies the direction in which the plurality of backflows are separated from the ultrasonic image, and generates control information for irradiating X-rays from the identified direction. do. As a result, the operator can efficiently find the backflow of blood and take treatment.

The control information generation function 455 can also determine the X-ray irradiation direction so that the position of the medical device and the position of the blood flow information are separated on the X-ray image. FIG. 15 is a diagram for explaining an example of diagnostic device control according to the first embodiment. For example, as shown in FIG. 15, the control information generation function 455 generates control information for irradiating X-rays from directions in which the position of the medical device and the position of the backflow are separated on the X-ray image. As a result, even when another medical device is placed in the backflow site, the procedure can be performed without the first placed medical device becoming an obstacle.

Next, a case in which the X-ray irradiation direction is determined according to the shape of the medical device will be described. In such a case, the control function 451 acquires information on the shape of the medical device and information on the position of the target site. The control information generation function 455 determines the X-ray irradiation direction so that the shape of the medical device can be identified on the X-ray image in the procedure of placing the medical device on the target site. FIG. 16 is a diagram for explaining an example of diagnostic device control according to the first embodiment.

For example, the control function 451 identifies the location of the target region and the location and shape of the medical device based on the ultrasound images. Here, if the medical device is, for example, a circular device with a gap in part as shown in FIG. The control information is generated so that X-rays are emitted from a direction in which is clearly drawn. That is, the control information generation function 455 generates control information for irradiating X-rays from the direction shown in the right diagram of FIG.

As described above, the control information generation function 455 generates control information for controlling the irradiation direction of X-rays in the X-ray diagnostic apparatus 200 in various situations. The control function 451 controls the X-ray diagnostic apparatus 200 by controlling to transmit the control information generated by the control information generation function 455 to the X-ray diagnostic apparatus 200 .

The identification of the shape and position of the medical device in the ultrasonic image and the identification of the shape and position of the backflow described in the above example can be automatically performed by the control function 451 performing processing on the ultrasonic image. , or may be manually specified by the operator on the ultrasound image.

Next, processing of the medical information processing apparatus 400 according to the first embodiment will be described with reference to FIGS. 17 to 21. FIG. 17 to 21 are flowcharts showing processing procedures of the medical information processing apparatus 400 according to the first embodiment. Here, FIGS. 17 to 19 show processing related to image display. Also, FIG. 20 shows processing related to support information display and diagnosis device control. Also, FIG. 21 shows processing related to diagnostic device control. 17 to 21, the same processing is given the same reference numerals.

Steps S101 and S104 shown in FIG. 17 are steps in which the processing circuit 450 reads out a program corresponding to the control function 451 from the storage circuit 420 and executes it. Step S102 is a step in which the processing circuit 450 reads out a program corresponding to the alignment function 453 from the storage circuit 420 and executes it. Step S103 is a step in which the processing circuit 450 reads a program corresponding to the image processing function 454 from the storage circuit 420 and executes it.

For example, as shown in FIG. 17, processing circuitry 450 acquires an ultrasound image and an X-ray image (step S101) and performs registration (step S102). Then, the processing circuit 450 generates an ultrasonic image and an X-ray image in substantially the same direction (step S103), and parallelizes the ultrasonic image and the X-ray image together with information indicating that they are in substantially the same direction. display (step S104).

Step S201 shown in FIG. 18 is a step in which the processing circuit 450 reads out a program corresponding to the image processing function 454 from the storage circuit 420 and executes it. Steps S202 to S205 are steps in which the processing circuit 450 reads out a program corresponding to the control function 451 from the storage circuit 420 and executes it.

For example, as shown in FIG. 18, processing circuitry 450 acquires an ultrasound image and an X-ray image (step S101) and performs registration (step S102). Then, the processing circuit 450 generates various X-ray images (composite images) to which information based on the ultrasound image is added (step S201), and displays one of the various X-ray images (composite images). (step S202).

Then, the processing circuit 450 determines whether or not a switching operation has been received (step S203). Here, when the switching operation is accepted (Yes at step S203), the processing circuitry 450 displays a composite image corresponding to the switching (step S204). After that, when the end operation is received (Yes at step S205), the processing circuit 450 ends the process. On the other hand, if the end operation has not been received (No at step S205), the processing circuit 450 returns to step S203 and executes the determination process.

Step S301 shown in FIG. 19 is a step in which the processing circuit 450 reads out a program corresponding to the image processing function 454 from the storage circuit 420 and executes it. Step S302 is a step in which the processing circuit 450 reads out a program corresponding to the control function 451 from the storage circuit 420 and executes it.

For example, as shown in FIG. 19, processing circuitry 450 acquires an ultrasound image and an X-ray image (step S101) and performs registration (step S102). Then, the processing circuit 450 generates a composite image by combining the ultrasonic image and the X-ray image (step S301), and displays the composite image in which the orientation of the ultrasonic image can be identified (step S302).

Steps S401 to S405 shown in FIG. 20 are steps in which the processing circuit 450 reads a program corresponding to the control function 451 from the storage circuit 420 and executes it.

For example, as shown in FIG. 20, processing circuitry 450 acquires ultrasound and X-ray images (step S101) and performs registration (step S102). Then, the processing circuit 450 receives the planned lines on the ultrasonic image (step S401), and displays the planned lines received on the ultrasonic image on the X-ray image based on the registration result (step S402). Further, the processing circuitry 450 calculates and displays the deviation between the planned line and the X-ray projection direction (step S403).

After that, the processing circuit 450 controls the X-ray diagnostic apparatus 200 so that the projection direction of the X-rays with respect to the planning line becomes the processing angle (for example, 90°) (step S404). Then, the processing circuit 450 causes the planning line to be displayed on the acquired X-ray image after control (step S405).

Steps S501, S503, and S504 shown in FIG. 21 are steps in which the processing circuit 450 reads out a program corresponding to the control function 451 from the storage circuit 420 and executes it. Step S502 is a step in which the processing circuit 450 reads out a program corresponding to the control information generation function 455 from the storage circuit 420 and executes it.

For example, as shown in FIG. 21, processing circuitry 450 acquires an ultrasound image and an X-ray image (step S101) and performs registration (step S102). Then, the processing circuit 450 acquires information about the medical device and the target site based on the ultrasound image (step S501). Then, the processing circuitry 450 determines the X-ray irradiation direction based on the positional relationship between the medical device and the target site (step S502).

Then, the processing circuit 450 controls the X-ray diagnostic apparatus 200 so that X-rays are emitted from the determined irradiation direction (step S503). Processing circuitry 450 then causes the acquired X-ray image to be displayed after control (step S504).

As described above, according to the first embodiment, the control function 451 receives information about the medical device inserted into the body of the subject and information about the target site of the procedure using the medical device from the subject. Acquisition based on the acquired ultrasound images. The control information generation function 455 is based on the alignment result of the ultrasound image and the X-ray image acquired together with the ultrasound image, and determines the timing of acquiring the X-ray image during the procedure using the medical device for the target site. The irradiation direction of X-rays to the subject is determined. Therefore, the medical information processing apparatus 400 according to the first embodiment can display an easy-to-observe X-ray image, making it possible to improve the efficiency of the procedure.

Further, according to the first embodiment, the registration function 453 performs the following from the group of ultrasonic images acquired over time from the subject and the group of X-ray images acquired over time together with the group of ultrasonic images: An ultrasonic image and an X-ray image acquired at substantially the same time are extracted, and registration is performed using the extracted ultrasonic image and X-ray image. Therefore, the medical information processing apparatus 400 according to the first embodiment can improve the processing efficiency of alignment, and can further improve the efficiency of the procedure.

Further, according to the first embodiment, the control function 451 displays the ultrasonic image and the X-ray image in substantially the same direction side by side based on the alignment result so that they can be identified as being in the substantially same direction. Therefore, the medical information processing apparatus 400 according to the first embodiment can display an ultrasound image and an X-ray image that can be easily compared, and can improve the efficiency of the procedure.

Further, according to the first embodiment, the image processing function 454 generates a plurality of composite images each showing a plurality of pieces of information based on the ultrasound image on the X-ray image, based on the alignment result. The control function 451 switches and displays the plurality of synthesized images according to a switching operation by the operator. Therefore, the medical information processing apparatus 400 according to the first embodiment can easily switch and display various synthesized images, thereby improving the efficiency of the procedure.

Further, according to the first embodiment, the image processing function 454 generates a synthesized image by synthesizing the information based on the ultrasound image on the X-ray image based on the registration result. The control function 451 displays a composite image in which the orientation of the ultrasonic image is identifiable. Therefore, the medical information processing apparatus 400 according to the first embodiment can accurately grasp the orientation of the ultrasonic image in the composite image, and improve the efficiency of the procedure.

Further, according to the first embodiment, the image processing function 454 selects a predetermined plane from a group of ultrasound images acquired in a plurality of cross sections, and combines the selected predetermined plane with an X-ray image. Generate an image. Therefore, the medical information processing apparatus 400 according to the first embodiment can improve the visibility of the synthesized image and improve the efficiency of the procedure.

Further, according to the first embodiment, the image processing function 454 extracts ultrasonic images and X-ray images obtained at approximately the same time from the group of ultrasonic images acquired over time and the group of X-ray images acquired over time. Select images to generate a composite image. Therefore, the medical information processing apparatus 400 according to the first embodiment can display a composite image using an ultrasound image and an X-ray image that are acquired at the same time. to improve the efficiency of

Further, according to the first embodiment, the control function 451 acquires information on the plan line of the procedure set on the ultrasound image. Control function 451 causes the planning lines to be displayed on the X-ray images acquired during the procedure based on the registration results. Therefore, the medical information processing apparatus 400 according to the first embodiment can display the plan line on the ultrasound image and the X-ray image, thereby improving the efficiency of the procedure.

Further, according to the first embodiment, the control function 451 displays information about the angle between the direction of the planning line and the X-ray irradiation direction of the X-ray image acquired during the procedure. Therefore, the medical information processing apparatus 400 according to the first embodiment can determine the deviation of the angle between the X-ray image and the planning line, making it possible to improve the efficiency of the procedure.

Further, according to the first embodiment, the control information generation function 455 determines the X-ray irradiation direction for the object to be the direction orthogonal to the direction of the planned line. Therefore, the medical information processing apparatus 400 according to the first embodiment can observe an accurate plan line on an X-ray image, and can improve the efficiency of a procedure.

Further, according to the first embodiment, the control information generation function 455 determines the X-ray irradiation direction with respect to the subject so that the planned line is shown horizontally or vertically on the X-ray image. Therefore, the medical information processing apparatus 400 according to the first embodiment can improve the visibility of the plan line on the X-ray image, and improve the efficiency of the procedure.

Also according to the first embodiment, the control function 451 obtains the orientation information of the medical device contained in the ultrasound image. Based on the alignment result, the control function 451 displays information about the angle between the direction of the medical device and the X-ray irradiation direction of the X-ray image acquired during the procedure. Therefore, the medical information processing apparatus 400 according to the first embodiment can determine the angle deviation between the X-ray image and the medical device, thereby improving the efficiency of the procedure.

Further, according to the first embodiment, the control function 451 controls the X-ray image group so that the region of interest continues to be displayed at a predetermined position in the X-ray image in the X-ray image group acquired over time during the procedure. Controls acquisition of line images. Therefore, the medical information processing apparatus 400 according to the first embodiment can always grasp the position of the region of interest in the real-time moving image, making it possible to improve the efficiency of the procedure.

Further, according to the first embodiment, the control function 451 acquires information on the path of the medical device to the target site and information on the shape of the target site. The control information generation function 455 determines the X-ray irradiation direction for the target site based on the information on the path and the information on the shape of the target site. Therefore, the medical information processing apparatus 400 according to the first embodiment can acquire X-ray images from appropriate angles according to the procedure, making it possible to improve the efficiency of the procedure.

Also according to the first embodiment, the control function 451 obtains path information of the medical device to the heart valve and shape information of the heart valve based on the ultrasound image. The control information generation function 455 determines the irradiation direction of the X-ray so that the heart valve is horizontally oriented and the opening and closing of the valve is depicted on the X-ray image. Therefore, the medical information processing apparatus 400 according to the first embodiment makes it possible to improve the efficiency of procedures for heart valves.

Further, according to the first embodiment, the control function 451 acquires the information of the suturing plan line of the heart valve by the medical device based on the ultrasound image. The control information generation function 455 determines the irradiation direction of X-rays so that the overlap of suture planning lines on the X-ray image is minimized. Therefore, the medical information processing apparatus 400 according to the first embodiment makes it possible to improve the efficiency of heart valve suturing procedures.

Further, according to the first embodiment, the control function 451 acquires information on the path of the medical device to the chordae tendineae of the heart valve and information on the shape of the chordae tendineae based on the ultrasound image. The control information generation function 455 determines the X-ray irradiation direction so that the overlap of the chordae tendineae on the X-ray image is minimized. Therefore, the medical information processing apparatus 400 according to the first embodiment makes it possible to improve the efficiency of the procedure for the chordae tendineae.

Further, according to the first embodiment, the control function 451 acquires information on the shape of the medical device and information on the shape of the target site. The control information generation function 455 determines the X-ray irradiation direction for the target site based on the placement state of the medical device for the target site. Therefore, the medical information processing apparatus 400 according to the first embodiment can acquire an X-ray image from a direction that takes into account the shape of the medical device and the shape of the target site, thereby improving the efficiency of the procedure. to

Further, according to the first embodiment, when a circular medical device is placed in a circular target site, the control information generation function 455 performs The irradiation direction of the X-rays with respect to the target part is determined so that the X-rays are irradiated from the orthogonal direction. Therefore, the medical information processing apparatus 400 according to the first embodiment makes it possible to improve the efficiency of a procedure when placing a circular medical device in a circular target site.

Further, according to the first embodiment, the control function 451 acquires position information of the medical device and blood flow information for each position in the target region based on the ultrasound image. The control information generation function 455 determines the X-ray irradiation direction so that the position of the medical device and the position of the blood flow information are separated on the X-ray image. Therefore, the medical information processing apparatus 400 according to the first embodiment can display an easy-to-observe X-ray image of backflow after placement of a medical device, thereby improving the efficiency of the procedure.

Further, according to the first embodiment, the control function 451 acquires information on the shape of the medical device and information on the position of the target site. The control information generation function 455 determines the X-ray irradiation direction so that the shape of the medical device can be identified on the X-ray image in the procedure of placing the medical device on the target site. Therefore, the medical information processing apparatus 400 according to the first embodiment can acquire X-ray images from a direction that takes into account the shape of the medical device, making it possible to improve the efficiency of the procedure.

(Second embodiment)
In the first embodiment described above, the case where the X-ray diagnostic apparatus 200 is controlled based on the ultrasonic image has been described. In the second embodiment, a case of controlling the ultrasonic diagnostic apparatus 100 based on X-ray images will be described. In addition, hereinafter, the same reference numerals may be given to the same configurations as in the first embodiment, and the description thereof may be omitted.

A control information generation function 455 according to the second embodiment determines acquisition conditions for ultrasound images based on X-ray images acquired during a procedure. 22 and 23 are diagrams for explaining an example of diagnostic device control processing according to the second embodiment. Note that FIGS. 22 and 23 show the processing after the alignment of the ultrasound image and the X-ray image is completed.

For example, as shown in FIG. 22, when the operator sets a straight line L1 on the X-ray image via the input interface 430, the control function 451 identifies the position of the straight line L1 in the X-ray coordinate system. Then, as shown in FIG. 22, the control function 451 identifies the position of a plane P3 connecting the X-ray focal point and the straight line L1. Furthermore, the control function 451 identifies the position of the plane P3 in the ultrasonic coordinate system based on the alignment result.

The control information generation function 455 generates control information that sets scan conditions for scanning the plane P3 in the ultrasound coordinate system specified by the control function 451 . The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the ultrasonic diagnostic apparatus 100 via the transmission/reception function 452 . When the ultrasonic diagnostic apparatus 100 receives the control information from the medical information processing apparatus 400, the ultrasonic diagnostic apparatus 100 presents to the operator whether or not to execute control based on the control information. Execute the scan with the scan conditions based on the Note that the control function 451 can also control the ultrasonic diagnostic apparatus 100 to automatically scan in response to reception of control information.

Further, for example, as shown in FIG. 23, when X-ray images in two directions are acquired, the control function 451 estimates a treatment target from the information of the same site included in the X-ray images in two directions, and estimates The position of the treated object in the X-ray coordinate system is specified. Then, the control function 451 identifies the position of the treatment target in the ultrasonic coordinate system based on the alignment result. The control information generation function 455 generates control information that sets scan conditions for scanning the treatment target in the ultrasonic coordinate system specified by the control function 451 . For example, the control information generation function 455 determines the scan direction, beam direction, and scan cross-section position for scanning the treatment target. The control function 451 causes the control information generated by the control information generation function 455 to be transmitted to the ultrasonic diagnostic apparatus 100 via the transmission/reception function 452 .

Next, processing of the medical information processing apparatus 400 according to the second embodiment will be described using FIG. FIG. 24 is a flow chart showing the processing procedure of the medical information processing apparatus 400 according to the second embodiment. In FIG. 24, the same reference numerals are given to the same processes as the processes in the first embodiment (the processes shown in FIG. 21).

Step S601 shown in FIG. 24 is a step in which the processing circuit 450 reads a program corresponding to the control information generation function 455 from the storage circuit 420 and executes it. Steps S602 and S603 are steps in which the processing circuit 450 reads a program corresponding to the control function 451 from the storage circuit 420 and executes it.

For example, as shown in FIG. 24, processing circuitry 450 acquires an ultrasound image and an X-ray image (step S101) and performs registration (step S102). Then, the processing circuitry 450 determines ultrasound scanning conditions based on the X-ray image (step S602). Then, the processing circuitry 450 controls the ultrasonic diagnostic apparatus 100 so as to perform scanning under the determined scanning conditions (step S602). Thereafter, the processing circuit 450 causes the acquired ultrasound image to be displayed after control (step S603).

As described above, according to the second embodiment, the control information generation function 455 determines acquisition conditions for ultrasound images based on X-ray images acquired during a procedure. Therefore, the medical information processing apparatus 400 according to the second embodiment can acquire appropriate ultrasound images, and can improve the efficiency of the procedure.

(Third embodiment)
Now, although the first and second embodiments have been described so far, various different modes other than the above-described first and second embodiments may be implemented.

As described above, in the method of the present application, the position of the medical device and the position of the target site can be identified in the three-dimensional ultrasonic image data, and by aligning the ultrasonic image and the X-ray image, the X-ray image can be obtained. can also be specified. Therefore, according to the method of the present application, for example, it is also possible to identify the position of the catheter and the position of the target site and present the operation direction of the catheter.

In such a case, for example, the control function 451 identifies the position of the catheter in the three-dimensional ultrasound image data and the position of the target site, and three-dimensionally determines the movement direction from the current position of the catheter to the target site. Vectorize in the ultrasonic coordinate system. Then, the control function 451 identifies the movement direction of the catheter in the X-ray coordinate system based on the alignment result, and presents it on the X-ray image. The control function 451 can also present the direction of movement of the catheter on the displayed two-dimensional ultrasound image. Note that the moving direction of the catheter may be presented by voice or the like.

Furthermore, when the operator sets the movement direction of the catheter on the ultrasonic image, the control function 451 calculates the vector direction of the set movement direction in the three-dimensional ultrasonic coordinate system, and based on the alignment result. can be used to identify the movement direction of the catheter in the X-ray coordinate system and present it on the X-ray image.

Further, in the above-described embodiment, the case where the medical information processing apparatus 400 executes each process has been described. However, the embodiment is not limited to this, and for example, the ultrasonic diagnostic apparatus 100 may perform the above-described processing. In such a case, the processing circuit 150 executes the same processing as the control function 451, alignment function 453, image processing function 454, and control information generation function 455 described above.

Further, the control function 451, the alignment function 453, the image processing function 454, and the control information generation function 455 described above may be distributed and executed by devices included in the medical information processing system 1. FIG.

In the above-described embodiment, an example in which each processing function is realized by a single processing circuit (processing circuit 450) has been described, but the embodiment is not limited to this. For example, the processing circuit 450 may be configured by combining a plurality of independent processors, and each processor may implement each processing function by executing each program. Further, each processing function of the processing circuit 450 may be appropriately distributed or integrated in a single or a plurality of processing circuits and implemented.

In addition, the term "processor" used in the above description includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an application specific integrated circuit (ASIC), a programmable logic device ( For example, it means circuits such as Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA). The processor implements its functions by reading and executing programs stored in the storage circuit 420 . Note that instead of storing the program in the memory circuit 420, the program may be configured to be directly installed in the circuit of the processor. In this case, the processor implements its functions by reading and executing the program embedded in the circuit. Note that each processor of the present embodiment is not limited to being configured as a single circuit for each processor, and may be configured as one processor by combining a plurality of independent circuits to realize its function. good.

Here, the medical information processing program executed by the processor is preinstalled in a ROM (Read Only Memory), storage unit, or the like and provided. This medical information processing program is a file in a format that can be installed in these devices or in a format that can be executed. disk) or other computer-readable storage medium. Also, this medical information processing program may be stored on a computer connected to a network such as the Internet, and may be provided or distributed by being downloaded via the network. For example, this medical information processing program is composed of modules including each functional part described later. As actual hardware, the CPU reads out a program from a storage medium such as a ROM and executes it, so that each module is loaded onto the main storage device and generated on the main storage device.

Also, each component of each device illustrated in the above-described embodiment is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Furthermore, all or any part of each processing function performed by each device can be implemented by a CPU and a program analyzed and executed by the CPU, or implemented as hardware based on wired logic.

As described above, according to at least one embodiment, it is possible to improve the efficiency of the procedure.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

400 medical information processing apparatus 450 processing circuit 451 control function 453 alignment function 454 image processing function 455 control information generation function

Claims (33)

an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a generation unit that generates a plurality of composite images in which a plurality of pieces of information based on the ultrasonic image are respectively indicated on the X-ray image based on the alignment result;
a display control unit that switches and displays the plurality of synthesized images in accordance with a switching operation by an operator;
with
The medical information processing system, wherein the determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image. Ultrasound images and X-ray images acquired at approximately the same time from a group of ultrasound images acquired over time from the subject and a group of X-ray images acquired over time together with the group of ultrasound images. 2. The medical information processing system of claim 1, further comprising an alignment unit that extracts and performs alignment using the extracted ultrasound and X-ray images. 3. The display control unit according to claim 1 or 2, wherein, based on the alignment result, the ultrasonic image and the X-ray image in substantially the same direction are displayed side by side so as to be identifiable as being in substantially the same direction. Medical information processing system. The generation unit generates a composite image by combining information based on an ultrasound image with an X-ray image based on the alignment result,
3. The medical information processing system according to claim 1, wherein said display control unit displays said composite image in which the orientation of said ultrasonic image is identifiable. 5. The generator according to claim 4 , wherein a predetermined plane is selected from a group of ultrasonic images acquired in a plurality of cross-sections, and a composite image is generated by synthesizing the selected plane on the X-ray image. Medical information processing system. The generating unit selects an ultrasonic image and an X-ray image at approximately the same time from a group of ultrasonic images acquired over time and a group of X-ray images acquired over time to generate the combined image. The medical information processing system according to claim 4 or 5 . The acquisition unit acquires information on the plan line of the procedure set on the ultrasound image,
3. The medical information processing system according to claim 1 , wherein said display control unit displays said plan line on an X-ray image acquired during said procedure based on said alignment result. 8. The medical information processing system according to claim 7 , wherein said display control unit displays information about an angle between the direction of said plan line and the irradiation direction of X-rays in an X-ray image acquired during said procedure. 9. The medical information processing system according to claim 7 , wherein said determining unit further determines an X-ray irradiation direction for said subject to be a direction perpendicular to the direction of said plan line. 9. The medical information processing system according to claim 7 , wherein said determining unit further determines an X-ray irradiation direction for said subject so that said plan line is shown horizontally or vertically on said X-ray image. . The acquisition unit acquires direction information of the medical device included in the ultrasonic image,
2. The display control unit displays information about an angle between a direction of the medical device and an X-ray irradiation direction of an X-ray image acquired during the procedure, based on the alignment result. Or the medical information processing system according to 2. Further comprising a control unit for controlling acquisition of the X-ray image group so that the region of interest is continuously displayed at a predetermined position in the X-ray image in the X-ray image group acquired over time during the procedure, The medical information processing system according to claim 1 or 2. The acquisition unit acquires information on a route of the medical device to the target site and information on the shape of the target site,
3. The medical information processing system according to claim 1, wherein said determining unit further determines an X-ray irradiation direction for said target site based on information on said path and information on a shape of said target site. The acquisition unit acquires information on a path of the medical device to the heart valve and information on the shape of the heart valve based on the ultrasound image,
14. The medical information processing system according to claim 13 , wherein said determination unit determines the irradiation direction of said X-rays so that said heart valve is horizontally oriented and opening and closing of said valve is depicted on an X-ray image. The acquisition unit acquires information on a suturing plan line of the heart valve by the medical device based on the ultrasonic image,
14. The medical information processing system according to claim 13 , wherein said determining unit determines the irradiation direction of said X-rays so as to minimize overlap of said suturing plan lines on an X-ray image. The acquisition unit acquires information on the path of the chordae tendineae of the heart valve of the medical device and information on the shape of the chordae tendineae based on the ultrasound image,
14. The medical information processing system according to claim 13 , wherein said determining unit determines the irradiation direction of said X-rays so as to minimize overlap of said chordae tendineae on an X-ray image. The acquisition unit acquires shape information of the medical device and shape information of the target site,
3. The medical information processing system according to claim 1, wherein said determining unit further determines an X-ray irradiation direction with respect to said target site based on the arrangement state of said medical device with respect to said target site. When a circular medical device is to be placed in a circular target site, the determining unit is configured to irradiate the X-rays from a direction orthogonal to a plane in which the degree of circularity of the target site is lowest. 18. The medical information processing system according to claim 17 , further comprising: determining an X-ray irradiation direction for said target region. The acquisition unit acquires position information of the medical device and blood flow information for each position in the target site based on the ultrasonic image,
3. The medical device according to claim 1, wherein the determination unit further determines the irradiation direction of the X-ray so that the position of the medical device and the position of the blood flow information are separated on an X-ray image. Information processing system. The acquisition unit acquires information on the shape of the medical device and information on the position of the target site,
wherein the determination unit further determines the irradiation direction of the X-rays so that the shape of the medical device can be identified on an X-ray image in a procedure of placing the medical device on the target site. Item 3. The medical information processing system according to Item 1 or 2. 3. The medical information processing system according to claim 1, wherein said determining unit determines acquisition conditions for ultrasonic images based on X-ray images acquired during said procedure. The acquisition unit acquires information on the position of the catheter and information on the position of the target part included in the ultrasonic image,
The determination unit determines an operation direction of the catheter with respect to the target site,
3. The medical information processing system according to claim 1, further comprising a presentation unit that presents an operation direction of said catheter to an operator. an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a generation unit that generates a plurality of composite images in which a plurality of pieces of information based on the ultrasonic image are respectively indicated on the X-ray image based on the alignment result;
a display control unit that switches and displays the plurality of synthesized images in accordance with a switching operation by an operator;
with
The medical information processing apparatus, wherein the determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image. a collection unit that collects ultrasonic images by transmitting and receiving ultrasonic waves to and from a subject;
an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device, based on the ultrasonic image;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a generation unit that generates a plurality of composite images in which a plurality of pieces of information based on the ultrasonic image are respectively indicated on the X-ray image based on the alignment result;
a display control unit that switches and displays the plurality of synthesized images in accordance with a switching operation by an operator;
with
The ultrasonic diagnostic apparatus, wherein the determining unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image. an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires information on the plan line of the procedure set on the ultrasound image,
A medical information processing system, further comprising a display control unit that displays the plan line on an X-ray image acquired during the procedure, based on the alignment result. an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires direction information of the medical device included in the ultrasonic image,
A medical information processing system, further comprising a display control unit configured to display information about an angle formed between a direction of the medical device and an X-ray irradiation direction of an X-ray image acquired during the procedure, based on the alignment result. . an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a control unit that controls acquisition of the group of X-ray images so that a region of interest is continuously displayed at a predetermined position in the X-ray image in the group of X-ray images acquired over time during the procedure;
with
The medical information processing system, wherein the determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image. an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires information on the plan line of the procedure set on the ultrasound image,
A medical information processing apparatus, further comprising a display control unit that displays the plan line on an X-ray image acquired during the procedure based on the alignment result. an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires direction information of the medical device included in the ultrasonic image,
A medical information processing apparatus, further comprising a display control unit configured to display information about an angle formed between a direction of the medical device and an X-ray irradiation direction of an X-ray image acquired during the procedure, based on the alignment result. . an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device based on an ultrasonic image acquired from the subject;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a control unit that controls acquisition of the group of X-ray images so that a region of interest is continuously displayed at a predetermined position in the X-ray image in the group of X-ray images acquired over time during the procedure;
with
The medical information processing apparatus, wherein the determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image. a collection unit that collects ultrasonic images by transmitting and receiving ultrasonic waves to and from a subject;
an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device, based on the ultrasonic image;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires information on the plan line of the procedure set on the ultrasound image,
The ultrasonic diagnostic apparatus further comprising a display control unit that displays the planned line on the X-ray image acquired during the procedure based on the alignment result. a collection unit that collects ultrasonic images by transmitting and receiving ultrasonic waves to and from a subject;
an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device, based on the ultrasonic image;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
with
The determination unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image,
The acquisition unit acquires direction information of the medical device included in the ultrasonic image,
An ultrasonic diagnostic apparatus, further comprising a display control unit configured to display information about an angle between a direction of the medical device and an X-ray irradiation direction of an X-ray image acquired during the procedure, based on the alignment result. . a collection unit that collects ultrasonic images by transmitting and receiving ultrasonic waves to and from a subject;
an acquisition unit that acquires information on a medical device to be inserted into the body of a subject and information on a target site for a procedure using the medical device, based on the ultrasonic image;
For the subject when acquiring an X-ray image during a procedure using the medical device for the target site based on the alignment result of the ultrasonic image and the X-ray image acquired together with the ultrasonic image a determination unit that determines the irradiation direction of X-rays;
a control unit that controls acquisition of the group of X-ray images so that a region of interest is continuously displayed at a predetermined position in the X-ray image in the group of X-ray images acquired over time during the procedure;
with
The ultrasonic diagnostic apparatus, wherein the determining unit determines the irradiation direction of the X-rays with respect to the subject so as to acquire an X-ray image in substantially the same direction as the ultrasonic image.

Images