JP2022055170A - Computer program, image processing method and image processing device - Google Patents

Abstract

To provide a computer program which displays a three-dimensional image of a blood vessel so as to be superimposed on an angiography image and allows an operator to intuitively grasp the state of the blood vessel easily.SOLUTION: A computer program causes a computer to execute processing of acquiring a plurality of first images obtained by imaging a lateral cross section of a hollow organ at plural spots by using a catheter, generating a three-dimensional image of the hollow organ on the basis of the acquired first images, acquiring a plurality of second images obtained by imaging the hollow organ and the catheter from different directions outside an organism, specifying a physical relation between the first image and the second image on the basis of the acquired second images, and mapping and displaying the three-dimensional image so as to be superimposed on the second image on the basis of the specified physical relation.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to computer programs, image processing methods and image processing devices.

Percutaneous Coronary Intervention (PCI) is one of the treatments for chronic total occupation (CTO) in which the coronary artery is chronically completely occluded and blood does not flow. There is a method called. PCI is a minimally invasive treatment in which an occluded lesion is expanded with a balloon catheter and a stent is placed to reconstruct a blood vessel.

In PCI, the surgeon can confirm the obstruction of a blood vessel by intravascular ultrasound (IVUS: Intra Vascular Ultra Sound) examination using a catheter. In addition, it is possible to confirm the running of the blood vessel to the obstruction by an angiographic examination in which the blood vessel is imaged using X-rays while injecting a contrast medium into the blood vessel.

Japanese Unexamined Patent Publication No. 2017-153621

By the way, in PCI, the antegrade first guide wire may enter a false cavity such as a blood vessel wall. In this case, the IVUS catheter is inserted into the stray first guide wire, the true cavity is confirmed by the image obtained by IVUS (hereinafter referred to as IVUS image), and the second guide wire is passed through the obstruction. Is being done.

However, the IVUS image is a cross-sectional image of the blood vessel, and it is not easy for the operator to construct a three-dimensional image of the blood vessel in the head and grasp the positional relationship between the IVUS catheter and the true lumen.
On the other hand, since the image obtained by the angiography examination (hereinafter referred to as an angio image) is a fluoroscopic image taken from a predetermined direction outside the living body, it is relatively easy to grasp the positional relationship between the blood vessel and the IVUS catheter. However, it is not possible to obtain an image of the blood vessel on the obstruction side through which the contrast medium does not pass, and the position of the true lumen is unknown.
It is also conceivable to refer to the IVUS image and grasp the position of the true cavity in the angio image. Generally, the relationship between the direction in which the angio image is taken and the vertical and horizontal directions of the IVUS image is unknown, but the IVUS image is based on the position of the marker provided at the tip of the IVUS catheter that does not transmit X-rays. It is also conceivable to confirm the positional relationship with the angio image. However, it is still not easy for the surgeon to reconstruct the three-dimensional image of the blood vessel in his head using both images. In addition, it is not easy to operate the device for confirming the positional relationship between the IVUS image and the angio image.

On one aspect, it is intended to provide a computer program, an image processing method, and an image processing device capable of superimposing a three-dimensional image of a blood vessel on an angio image and making it easier for an operator to intuitively grasp the state of the blood vessel. The purpose.

A computer program relating to one aspect acquires a plurality of first images obtained by capturing a cross section of a lumen organ at a plurality of points using a catheter, and based on the acquired plurality of first images, the cavity organ. A three-dimensional image is generated, a plurality of second images obtained by imaging the lumen organ and the catheter from a plurality of different directions outside the living body are acquired, and the first image and the first image are obtained based on the acquired second images. 2 The positional relationship of the images is specified, and based on the specified positional relationship, the computer is made to execute a process of mapping the three-dimensional image onto the second image and displaying the superimposed image.

In the image processing method relating to one aspect, a plurality of first images obtained by capturing a cross section of a lumen organ at a plurality of points using a catheter are acquired, and the cavity organ is based on the acquired plurality of first images. Three-dimensional images of The computer executes a process of specifying the positional relationship of the second image, mapping the three-dimensional image onto the second image, and displaying the superimposed image based on the specified positional relationship.

The image processing apparatus according to one side surface is based on a first acquisition unit that acquires a plurality of first images obtained by capturing a cross section of a lumen organ at a plurality of locations using a catheter, and a plurality of acquired first images. , A generation unit that generates a three-dimensional image of the lumen organ, and a second acquisition unit that acquires a plurality of second images of the lumen organ and the catheter from a plurality of different directions outside the living body. Based on the second image of the above, the specific unit that specifies the positional relationship between the first image and the second image, and the three-dimensional image is mapped to the second image and displayed in an superimposed manner based on the specified positional relationship. It is equipped with a superposed display unit.

According to the present disclosure, a three-dimensional image of a blood vessel can be superimposed and displayed on an angio image to make it easier for an operator to intuitively grasp the state of the blood vessel.

It is explanatory drawing which shows the structural example of the diagnostic imaging apparatus. It is a side sectional view which shows the structural example of a catheter. It is a block diagram which shows the configuration example of an image processing apparatus. It is a schematic diagram which shows the side cross section of a blood vessel. It is explanatory drawing which shows the difference between an IVUS image and an angio image. It is a functional block diagram of an image processing apparatus. It is a composite image in which a three-dimensional image is mapped and superimposed on an angio image. It is explanatory drawing which shows the image recognition method using a learning model. It is explanatory drawing which shows the coordinate conversion method. It is explanatory drawing which shows the method of making a coordinate transformation matrix. It is a flowchart which shows the creation processing procedure of a coordinate transformation matrix. It is a flowchart which shows the image processing procedure.

Specific examples of the computer program, the image processing method, and the image processing apparatus according to the embodiment of the present disclosure will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims. Further, at least a part of the embodiments and modifications described below may be arbitrarily combined.
In the present embodiment, cardiac catheterization, which is an endovascular treatment, will be described as an example, but the luminal organs targeted for catheterization are not limited to blood vessels, and other luminal organs such as bile duct, pancreatic duct, bronchus, and intestine, for example. May be.

<Overall configuration of diagnostic imaging device 100>
FIG. 1 is an explanatory diagram showing a configuration example of the diagnostic imaging apparatus 100. The diagnostic imaging device 100 according to the present embodiment includes an intravascular ultrasonic examination device 101, an angiography device 102, an image processing device 3, a display device 4, and an input device 5. The image processing device 3 according to the present embodiment superimposes and displays a three-dimensional image of a blood vessel (luminal organ) on an angio image, and enables the operator to intuitively grasp the state of the blood vessel. be.

The intravascular ultrasound examination device 101 generates an IVUS image (first image) including an ultrasonic tomographic image which is a cross section of a blood vessel (luminal organ) by an intravascular ultrasound (IVUS: Intra Vascular Ultra Sound) method. , A device for performing ultrasonography and diagnosis in blood vessels. The intravascular ultrasonic examination device 101 includes a catheter 1 and an MDU (Motor Drive Unit) 2.

FIG. 2 is a side sectional view showing a configuration example of the catheter 1. The catheter 1 according to the present embodiment is a catheter for diagnostic imaging for obtaining an ultrasonic tomographic image of a blood vessel by the IVUS method. The catheter 1 has an ultrasonic probe 10 at the tip for obtaining an ultrasonic tomographic image of a blood vessel. The ultrasonic probe 10 has a tube portion, and the shaft 11 is inserted into the tube portion. The shaft 11 can move forward and backward along the pipe portion, and can rotate in the circumferential direction. The tip of the shaft 11 is provided with an ultrasonic transmission / reception unit 10a that emits ultrasonic waves in the blood vessel and receives reflected waves (ultrasonic echo) reflected by a biological tissue of the blood vessel or a medical device. The ultrasonic probe 10 is configured to be able to move forward and backward in the longitudinal direction of the blood vessel while rotating in the circumferential direction of the blood vessel.
The catheter 1 has a guide wire lumen through which the first guide wire GW1 is inserted at the tip thereof. The center line of the tube portion in the guide wire lumen and the center line of the tube portion of the ultrasonic probe 10 are separated by a predetermined length.

Further, the catheter 1 does not transmit X-rays in order to determine the positional relationship between the IVUS image obtained by the intravascular ultrasonic examination device 101 and the angio image obtained by the angiography device 102, so that the first to third markers 11a do not transmit X-rays. , 11b, 11c. The first to third markers 11a, 11b, and 11c are provided in a non-linear manner so as not to be arranged in a straight line. For example, the first marker 11a and the second marker 11b are provided at the tip of the shaft 11 with the ultrasonic wave transmission / reception unit 10a interposed therebetween. The third marker 11c is provided on the guide wire lumen. When the catheter 1 configured in this way is imaged with X-rays, for example, as shown in the lower figure of FIG. 2, an angio image which is an X-ray fluoroscopic image including images of the first to third markers 11a, 11b, 11c can be obtained.
The positions where the first to third markers 11a, 11b, and 11c are provided are examples. The first marker 11a and the second marker 11b may be provided not on the shaft 11 but on the main body of the catheter 1 having a tube portion through which the shaft 11 is inserted. That is, the positions of the first to third markers 11a, 11b, and 11c may be constant regardless of the position of the shaft 11.

The MDU 2 is a drive device to which the catheter 1 is detachably attached, and controls the operation of the catheter 1 inserted into the blood vessel by driving the built-in motor according to the operation of a medical worker. The MDU2 rotates the ultrasonic transmission / reception portion 10a of the catheter 1 in the circumferential direction while moving it from the distal end (distal) side to the proximal end (proximal) side (see FIG. 8). The ultrasonic probe 10 continuously scans the inside of a blood vessel at predetermined time intervals, and outputs an IVUS image based on the detected reflected ultrasonic data to the image processing device 3.

The image processing device 3 generates a plurality of IVUS images in chronological order including an ultrasonic tomographic image of a cross-sectional layer of a blood vessel based on the reflected wave data output from the ultrasonic probe 10 of the catheter 1 (FIG. 3). 8). Since the ultrasonic probe 10 scans the inside of the blood vessel while moving in the blood vessel from the distal end (distal) side to the proximal end (proximal) side, a plurality of IVUS images in chronological order extend from the distal side to the proximal side. It is a tomographic image of blood vessels observed at multiple locations.

The angiography device 102 is an imaging device for injecting a contrast medium into a patient's blood vessel and imaging the blood vessel from outside the patient's body using X-rays to obtain an angio image which is a fluoroscopic image of the blood vessel. The angiography apparatus 102 includes an X-ray source and an X-ray sensor, and the X-ray sensor receives the X-rays emitted from the X-ray source to image a fluoroscopic image of the patient. The angiography apparatus 102 outputs the angio image obtained by imaging to the image processing apparatus 3.

When the blood vessel into which the catheter 1 is inserted is imaged by using the angiography apparatus 102, an angio image including an image of the blood vessel, the catheter 1, and the first guide wire GW1 is obtained. When the second guide wire GW2 described later is inserted into the blood vessel, an angio image including an image of the blood vessel, the catheter 1, the first guide wire GW1, and the second guide wire GW2 is obtained. Further, as described above, the catheter 1 is provided with the first to third markers 11a, 11b, 11c that do not transmit X-rays, and an angio image including the first marker image, the second marker image, and the third marker image. Is obtained.

In this embodiment, the angiography device 102 that mainly captures a two-dimensional angio image will be described as an example, but any device that images the patient's luminal organ and catheter 1 from a plurality of directions outside the living body. , Is not particularly limited. For example, it may be a three-dimensional CT angiography, a magnetic resonance imaging (MRI) image, or the like.

The display device 4 is a liquid crystal display panel, an organic EL display panel, or the like, and displays medical images such as IVUS images and angio images generated by the image processing device 3.

The input device 5 is an input interface such as a keyboard and a mouse that accepts input of various setting values at the time of inspection, operation of the image processing device 3, and the like. The input device 5 may be a touch panel, soft keys, hard keys, or the like provided on the display device 4.

<Hardware configuration of image processing device 3>
FIG. 3 is a block diagram showing a configuration example of the image processing device 3. The image processing device 3 is a computer and includes a control unit 31, a main storage unit 32, an input / output I / F 33, and an auxiliary storage unit 34.

The control unit 31 includes one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), GPGPUs (General-purpose computing on graphics processing units), and TPUs (Tensor Processing Units).
It is configured by using an arithmetic processing unit such as.

The main storage unit 32 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 31 to execute arithmetic processing. Remember.

The input / output I / F 33 is an interface to which the intravascular ultrasonic examination device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The control unit 31 acquires an IVUS image or an angio image via the input / output I / F 33. Further, the control unit 31 displays the medical image on the display device 4 by outputting the medical image signal of the IVUS image or the angio image to the display device 4 via the input / output I / F 33. Further, the control unit 31 receives the information input to the input device 5 via the input / output I / F 33.

The auxiliary storage unit 34 is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), and a flash memory. The auxiliary storage unit 34 stores various data necessary for processing of the computer program P executed by the control unit 31 and the control unit 31. Further, the auxiliary storage unit 34 stores the learning model 35. The details of the learning model 35 will be described later.

The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P may be written in the auxiliary storage unit 34 at the manufacturing stage of the image processing device 3, or the image processing device 3 acquires what is distributed by the remote server device by communication and writes it in the auxiliary storage unit 34. You may memorize it. The computer program P may be readable and recorded on a recording medium 30 such as a magnetic disk, an optical disk, or a semiconductor memory.

The control unit 31 acquires an IVUS image generated by the diagnostic imaging apparatus 100 by reading and executing the computer program P stored in the auxiliary storage unit 34, and obtains a three-dimensional image of the blood vessel having the CTO lesion site C. The process of generating and superimposing the generated three-dimensional image on the angio image is executed.

The image processing device 3 may be a multi-computer including a plurality of computers. Further, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, it is assumed that the image processing device 3 is one computer.

<Relationship between CTO lesion site and angio image and IVUS image>
Before explaining the function of the image processing apparatus 3, the relationship between the chronic complete occlusion lesion and the angio image and the IVUS image will be described.

FIG. 4 is a schematic view showing a side cross section of a blood vessel. In the blood vessel shown in FIG. 4, a CTO (Chronic Total Occlusion) lesion site C in which the coronary artery is chronically completely occluded and blood does not flow can be seen. In FIG. 4, the blood vessel occlusion site on the right side of the broken line is the CTO lesion site C.
In the PCI treatment of the CTO lesion site C, it is necessary to grasp the position of the true lumen A in the occluded blood vessel and insert the guide wire into the true lumen A. By the way, in PCI, the antegrade first guide wire GW1 may enter the false cavity B. FIG. 4 shows a state in which the first guide wire GW1 has entered the false cavity B of the blood vessel wall V. In this case, the positions of the blood vessel wall V, the true cavity A, and the false cavity B can be confirmed by inserting the catheter 1 of the intravascular ultrasonic examination device 101 into the stray first guide wire GW1 and obtaining an IVUS image. Will be done. The surgeon grasps the position of the true lumen A and inserts the second guide wire GW2 into the CTO lesion site C and the true lumen A as shown in FIG.
However, it is not easy to grasp the positions of the true cavity A and the false cavity B from the IVUS image and the angio image.

FIG. 5 is an explanatory diagram showing the difference between the IVUS image and the angio image. FIG. 5A is a three-dimensional image of a blood vessel. FIG. 5B is a schematic diagram of an angio image, and FIG. 5C is a schematic diagram of an IVUS image.
As shown in FIGS. 5A and 5B, the angio image is a fluoroscopic image obtained by imaging a blood vessel from the lateral direction (outside in the radial direction), and is suitable for grasping the running of the blood vessel. However, since the contrast medium does not pass through the CTO lesion site C, it is not possible to obtain images of the blood vessel wall V, the true cavity A, and the false cavity B in the portion shown by the broken line in FIG. 5B. Only the images of the catheter 1 and the first guide wire GW1 can be seen.
On the other hand, as shown in FIGS. 5A and 5C, the IVUS image is a cross-sectional image of a blood vessel (a cross-sectional image cut along a plane perpendicular to the center line of the blood vessel), and is a CTO lesion site C, true cavity A, and pseudo-cavity B. , The position of the catheter 1 and the first guide wire GW1 can be confirmed. Further, the position of the second guide wire GW2 can be confirmed by taking an image of a portion where the second guide wire GW2 (not shown) is present. However, the IVUS image is a cross-sectional image of the blood vessel, and it is not easy for the operator to construct a three-dimensional image of the blood vessel in the head and grasp the positional relationship between the catheter 1 and the true lumen A.

The IVUS image is an image obtained by emitting ultrasonic waves and detecting reflected waves while rotating the ultrasonic probe 10 of the catheter 1 in the circumferential direction, and the vertical and horizontal directions are undefined. Therefore, as shown by the white arrow in FIG. 5C, the angio image does not immediately show from which direction the blood vessel in the IVUS image was imaged. Angio images taken from multiple different directions are acquired, and the operator estimates the positional relationship between the IVUS image and the angio image from the images of the catheter 1, the first guide wire GW1, and the first to third markers 11a, 11b, and 11c. However, it is not easy because it is necessary to reconstruct the three-dimensional image of the true lumen A at the CTO lesion site C in the head.

Therefore, the image processing apparatus 3 according to the present embodiment generates a three-dimensional image of the blood vessel of the CTO lesion site C based on the IVUS image, and maps and superimposes the three-dimensional image on the angio image for display. To execute.

<Functional block diagram of image processing device 3>
FIG. 6 is a functional block diagram of the image processing apparatus 3, and FIG. 7 is a composite image in which a three-dimensional image is mapped and superimposed on an angio image. The control unit 31 of the image processing device 3 functions as a recognition unit 3a, a three-dimensional image generation unit 3b, a coordinate conversion unit 3c, and an image composition unit 3d by reading and executing the computer program P stored in the auxiliary storage unit 34. do.

The recognition unit 3a executes a process of recognizing a predetermined object included in the IVUS image by using the learning model 35. For example, the recognition unit 3a recognizes the image of the blood vessel wall V, the image of the true cavity A, the image of the false cavity B, the image of the catheter 1, and the images of the first guide wire GW1 and the second guide wire GW2 included in the IVUS image. do. The image recognized and processed by the recognition unit 3a is an image having pixel values according to the type of the object, that is, an image labeled with the pixel values according to the type of the object (hereinafter, referred to as a labeled IVUS image). be. The recognition unit 3a executes recognition processing for each of the plurality of IVUS images, and outputs the plurality of labeled IVUS images obtained by the recognition processing to the three-dimensional image generation unit 3b.

The three-dimensional image generation unit 3b generates a three-dimensional image representing a blood vessel of the CTO lesion site C based on a plurality of labeled IVUS images, and outputs the generated three-dimensional image to the coordinate conversion unit 3c. The three-dimensional image can be generated by, for example, the voxel method. The three-dimensional image is represented by volume data represented by a voxel coordinate value in a predetermined coordinate system and a voxel value indicating an object type. The data format of the three-dimensional image is not particularly limited, and may be polygon data or point cloud data.

The coordinate conversion unit 3c converts the coordinates of the three-dimensional image into the two-dimensional coordinate system of the angio image, and outputs the coordinate-converted two-dimensional image to the image composition unit 3d. The details of the coordinate conversion will be described later.

The image composition unit 3d executes a composition process for superimposing a two-dimensional image of the blood vessel whose coordinates have been converted on the angio image as shown in FIG. 7. The image processing unit may make the two-dimensional image of the blood vessel translucent and superimpose it on the angio image. That is, by the processing of the coordinate conversion unit 3c and the image composition unit 3d, the three-dimensional image of the blood vessel is mapped to the angio image and displayed in an superimposed manner.

<Learning model 35>
The learning model 35 is a model that recognizes a predetermined object included in the IVUS image. In the learning model 35, for example, by using an image recognition technique using semantic segmentation, objects can be classified on a pixel-by-pixel basis, and the CTO lesion site C included in the IVUS image, specifically, the CTO lesion site C. Can recognize the blood vessel wall V, the true cavity A, the false cavity B, the catheter 1, the first guide wire GW1, and the second guide wire GW2.

FIG. 8 is an explanatory diagram showing an image recognition method using the learning model 35. The learning model 35 is trained so that the images of the blood vessel wall V, the true cavity A, the false cavity B, the catheter 1, the first guide wire GW1, and the second guide wire GW2 in the IVUS image can be recognized on a pixel-by-pixel basis.

The learning model 35 is, for example, a convolutional neural network (CNN) that has been trained by deep learning. The learning model 35 recognizes an object on a pixel-by-pixel basis by an image recognition technique using so-called semantic segmentation.
The learning model 35 outputs an input layer 35a into which an IVUS image is input, an intermediate layer 35b that extracts and restores the feature amount of the image, and a labeled IVUS image indicating the objects included in the IVUS image in pixel units. It has a layer 35c. The learning model 35 is, for example, U-Net.

The input layer 35a of the learning model 35 has a plurality of neurons that receive the input of the pixel value of each pixel included in the IVUS image, and passes the input pixel value to the intermediate layer 35b. The intermediate layer 35b has a convolutional layer (CONV layer) and a deconvolutional layer (DECONV layer). The convolution layer is a layer that dimensionally compresses image data. Dimensional compression extracts the features of the object. The deconvolution layer undergoes deconvolution processing and is restored to its original dimension. The restoration process in the deconvolution layer generates a labeled IVUS image in which whether or not each pixel in the image is a predetermined object is represented by a pixel value according to the type of the object. The output layer 35c has a plurality of neurons that output a labeled IVUS image. In the labeled IVUS image, for example, the pixel corresponding to the blood vessel wall V corresponds to the class "1", the pixel class corresponding to the true cavity A "2", the pixel class corresponding to the false cavity B "3", and the catheter 1. Pixel class "4", pixel class "5" corresponding to the first and second guide wires GW1 and GW2, and pixels corresponding to other images are images of class "0".

The training model 35 prepares training data having an IVUS image including a blood vessel wall V, a catheter 1, a guide wire, a true cavity A and a false cavity B, and a labeled IVUS image showing each object image in the IVUS image. It can be generated by machine learning an unlearned neural network using the training data. Specifically, the control unit 31 inputs a plurality of IVUS images included in the training data to the input layer 35a of the neural network model before learning, undergoes arithmetic processing in the intermediate layer 35b, and is output from the output layer 35c. Get the image. Then, the control unit 31 compares the image output from the output layer 35c with the labeled IVUS image included in the training data, and intermediates the image output from the output layer 35c so as to approach the labeled IVUS image. The parameters used for the arithmetic processing on the layer 35b are optimized. The parameter is, for example, a weight (coupling coefficient) between neurons. The method of optimizing the parameters is not particularly limited, but for example, the control unit 31 optimizes various parameters by using a steepest descent method, an error back propagation method, or the like.

According to the learning model 35 thus trained, by inputting the IVUS image into the learning model 35, the labeled IVUS indicating the blood vessel wall V, the catheter 1, the guide wire, the true cavity A and the false cavity B in pixel units. An image is obtained.

<Coordinate conversion and superimposition display of 3D images>
FIG. 9 is an explanatory diagram showing a coordinate conversion method. Hereinafter, in order to facilitate understanding, a world coordinate system, which is a fixed predetermined three-dimensional Cartesian coordinate system, will be introduced, and a three-dimensional image of a blood vessel will be mapped to an angio image via the world coordinate system. In FIG. 9, black circles indicate images of the first to third markers 11a, 11b, and 11c. For the sake of simplicity, the images of the first to third markers 11a, 11b, and 11c are assumed to be images of each marker before scanning the blood vessel by the catheter 1, that is, before moving the shaft 11.

In FIG. 9, the angio image shown in the upper right is represented by a two-dimensional angio coordinate system. The Xa-axis and Ya-axis of the angio coordinate system are, for example, horizontal lines (horizontal axis) and vertical lines (vertical axis) of an image displayed on the display device 4. In FIG. 9, (xa, ya) is the coordinate value of the pixels constituting the angio image in the angio coordinate system.

The positions of the first to third markers 11a, 11b, 11c images in the angio image change depending on the imaging direction. Therefore, consider the absolute positions of the first to third markers 11a, 11b, and 11c in the world coordinate system. The world coordinate system is any three-dimensional Cartesian coordinate system. The method for setting the world coordinate system is arbitrary, but for example, the imaging direction by the angiography apparatus 102 may be set so as to be the origin and the imaging direction to change around the Xw axis.

The positions of the first to third markers 11a, 11b, and 11c provided on the catheter 1 before scanning the blood vessel are fixed. At least, the positional relationship between the first and second markers 11a and 11b is constant, and the positional relationship between the straight line passing through the first and second markers 11a and 11b and the third marker 11c is constant.
Therefore, from the coordinate positions of the first to third marker images in the angio image obtained by imaging from a plurality of different imaging directions θ, the coordinate positions of the first to third markers 11a, 11b, 11c in the world coordinate system (xw, yw, zw) can be obtained. Further, it is possible to obtain a coordinate transformation matrix fwa that converts the coordinates (xw, yw, zw) of the world coordinate system into the coordinates (xa, ya, za) of the angio coordinate system. The coordinate transformation matrix fwa can be represented by a product of a rotation matrix, a translation matrix, a normal projection matrix, an enlargement / reduction matrix, and the like. The coordinate transformation matrix fwa can be represented by a function of the imaging direction θ by the angiography apparatus 102.

On the other hand, the 3D image generated based on the IVUS image is mapped to the IVUS coordinate system. The IVUS coordinate system is, for example, a three-dimensional Cartesian coordinate system. The method of taking the IVUS coordinate system is arbitrary. For example, a three-dimensional coordinate system in which the first marker and the second marker pass through the Zi axis and the third marker passes through the Xi axis can be considered. Specifically, the center line of the catheter 1 in the three-dimensional image, in other words, the ultrasonic waves are set to zero in the X coordinate value and the Y coordinate value of the plurality of boxels at the output points, and the first guide wire GW1 in the three-dimensional image is configured. Of the boxels, the three-dimensional coordinates may be mapped to the IVUS coordinate system so that the Y coordinate value of the boxel whose Z-axis coordinate value is zero is zero. In FIG. 9, (xi, yi, zi) are coordinate values with voxels constituting a three-dimensional image in the IVUS coordinate system.
When the ultrasonic probe 10 moves, the positions of the first marker and the second marker change, but since the first marker and the second marker are always on the Zi axis, there is no problem in one IVUS coordinate system in three dimensions. Images can be mapped.
By aligning the positions of the first to third markers 11a, 11b, and 11c with the IVUS coordinate system in this way, the three-dimensional coordinates are mapped to the IVUS coordinate system as if the catheter 1 were aligned with a predetermined direction. Can be done.

As described above, the positions of the first to third markers 11a, 11b, 11c in the IVUS coordinate system are known, and the coordinate positions of the first to third markers 11a, 11b, 11c in the world coordinate system are also obtained. The coordinate conversion matrix fiw that converts the coordinates (xi, yi, zi) of the IVUS coordinate system to the coordinates (xw, yw, zw) of the world coordinate system can be obtained. This coordinate transformation matrix fiw can be represented by the product of a rotation matrix, a translation matrix, and an enlargement / reduction matrix.

After all, the coordinate transformation matrix fia that maps a three-dimensional image in the IVUS coordinate system to the angio coordinate system can be represented by the product of the coordinate transformation matrix fiw and the coordinate transformation matrix fwa.

<Derivation process of coordinate transformation matrix>
FIG. 10 is an explanatory diagram showing a method of creating the coordinate transformation matrix fia, and FIG. 11 is a flowchart showing a procedure for creating the coordinate transformation matrix. The control unit 31 acquires an angio image and an imaging direction θ obtained by imaging from a plurality of different imaging directions θ by the angiography apparatus 102 (step S11). Next, the control unit 31 determines whether or not an angio image has been captured from a predetermined number of directions (step S12). The predetermined number is 2 or more, but the more the imaging direction is, the higher the accuracy of the coordinate transformation matrix, that is, the accuracy of mapping the three-dimensional image of the blood vessel to the angio image.

When angio images captured from a predetermined number of directions have not been obtained (step S12: NO), the control unit 31 outputs a signal instructing the angiography apparatus 102 to change the imaging direction to change the imaging direction θ (step S12: NO). Step S13), the process is returned to step S11. When the angiography apparatus 102 automatically continuously changes the imaging direction θ, the process of step S13 is unnecessary.
The control unit 31 that executes the processes of steps S11 to S13 acquires a plurality of angio images (second images) obtained by imaging the blood vessels (luminal organs) and the catheter 1 from a plurality of different directions outside the living body. 2 Functions as an acquisition unit.

When a plurality of angio images captured from a predetermined number of directions are obtained (step S12: YES), the control unit 31 is a first to third marker in the plurality of angio images obtained by imaging from a plurality of different imaging directions θ. The coordinate transformation matrix fia is calculated using the coordinate values of 11a, 11b, 11c and the image pickup direction θ (step S14), and the process is completed.
The control unit 31 that executes the process of step S14 specifies the positional relationship between the IVUS image (first image) and the angio image (second image) based on the acquired plurality of angio images (second images). Functions as a specific part.

Considering the decomposition, if the coordinate values of the first to third markers 11a, 11b, 11c in the world coordinate system are obtained, the coordinate conversion matrix fiw from the IVUS coordinate system to the world coordinate system can be obtained. Further, if the coordinate values of the first to third markers 11a, 11b, 11c in the world coordinate system are obtained, the control unit 31 obtains them by the coordinate conversion matrix fwa (function of θ) from the world coordinate system to the angio coordinate system. The coordinate values of the first to third marker images to be obtained match the coordinate values of the first to third marker images in the angio image obtained by imaging from the imaging direction θ, for example, by the most probable estimation method. The coordinate conversion matrix fwa can be obtained. Then, by obtaining the product of the coordinate transformation matrix fiw and the coordinate transformation matrix fwa, it is possible to obtain the coordinate transformation matrix fia that maps the three-dimensional image in the IVUS coordinate system to the angio coordinate system.

Since the image of the blood vessel mapped to the angio coordinate system is a two-dimensional image that reflects all the positions, sizes, directions and inclinations of the blood vessels in the three-dimensional space, the two-dimensional image is superimposed on the angio image. This allows the surgeon to easily understand the three-dimensional structure of the blood vessels in the angio image.

<3D image generation and superimposed display processing>
FIG. 12 is a flowchart showing an image processing procedure. The image processing device 3 will be described assuming that the trained learning model 35 and the coordinate transformation matrix have been created by the processing of FIG.

The control unit 31 acquires a plurality of IVUS images output from the intravascular ultrasonic examination device 101 (step S31). The control unit 31 that executes the process of step S31 serves as a first acquisition unit that acquires a plurality of IVUS images (first images) obtained by capturing a cross section of a blood vessel (luminal organ) at a plurality of locations using a catheter. Function.

Next, the control unit 31 recognizes the object included in the IVUS image by inputting the acquired IVUS image into the learning model 35 (step S32). Then, the control unit 31 generates a three-dimensional image of the blood vessel based on the labeled IVUS image obtained by the object recognition process in step S32 (step S33). The three-dimensional image is, for example, a three-dimensional image of the blood vessel wall V, the true cavity A, the false cavity B, the catheter 1, the first guide wire GW1 and the second guide wire GW2.
The control unit 31 that executes the processes of steps S31 and S33 functions as a generation unit that generates a three-dimensional image of a blood vessel (luminal organ) based on the acquired plurality of IVUS images (first image). ..

Then, the control unit 31 acquires an angio image and an imaging direction θ from the angiography apparatus 102 (step S34). The control unit 31 obtains a coordinate conversion matrix based on the imaging direction θ, converts a three-dimensional image in the IVUS coordinate system into a two-dimensional image in the angio coordinate system (step S35), and uses the coordinate conversion matrix to convert the three-dimensional image into an angio coordinate system. The two-dimensional image of the blood vessel mapped to the above is superimposed on the angio image (step S36), and the composite image obtained by mapping the three-dimensional image of the blood vessel to the angio image is displayed in the display process (step S37).
The control unit 31 that executes the processes of steps S35 and S36 functions as a superimposed display unit that maps a three-dimensional image onto an angio image (second image) and displays them in an superimposed manner based on the specified positional relationship. ..

Next, the control unit 31 determines whether or not to end the process (step S38). When it is determined to end (step S38: YES), the control unit 31 ends the superimposing process of the three-dimensional image of the blood vessel. If it is determined that the process is not completed (step S38: NO), the control unit 31 returns the process to step S34.

As described above, according to the image processing device 3 or the like according to the present embodiment, a three-dimensional image of a blood vessel can be mapped to an angio image and displayed in an superimposed manner. It is possible to make it easier for the operator to intuitively grasp the state of blood vessels.

Further, by obtaining the coordinate transformation matrix based on the coordinate values of the first to third markers 11a, 11b, and 11c provided on the catheter 1, the three-dimensional image of the blood vessel is accurately mapped to the angio image and displayed in an superimposed manner. Can be made to.

Further, the blood vessel wall V, the true cavity A, the false cavity B, the catheter 1, the first guide wire GW1 and the second guide wire GW2 included in the IVUS image are recognized to generate a three-dimensional image, and the image is mapped to the angio image. It can be superimposed and displayed. Therefore, the operator can easily recognize the positions of the blood vessel wall V, the true cavity A, the false cavity B, and the catheter 1 in the angio image.
Further, since the image processing device 3 has a configuration in which each object included in the three-dimensional image is superimposed and displayed on the angio image as a transparent image, the image of the angio image, the true cavity A of the blood vessel based on the IVUS image, and the true cavity A of the blood vessel are displayed. It is possible to more accurately grasp the relationship between objects such as the false cavity B.

Furthermore, by repeatedly executing the processes of steps S34 to S37, an angio image and an imaging direction θ are acquired in chronological order, a coordinate conversion image corresponding to the imaging direction θ is obtained each time, and a three-dimensional image is obtained. It can be superimposed and displayed on the angio image acquired in chronological order. Even if the position of the blood vessel in the angio image changes due to the change in the imaging direction θ, the three-dimensional image can be mapped and superimposed on the corresponding position so as to follow the change in the position of the blood vessel.

The image processing device 3 needs to superimpose all the objects such as the blood vessel wall V, the true cavity A, the false cavity B, the catheter 1, the first guide wire GW1 and the second guide wire GW2 included in the IVUS image on the angio image. It may be configured to accept the object to be displayed from the user. The image processing device 3 generates a three-dimensional image including the received object to be displayed. Specifically, the control unit 31 obtains a labeled IVUS image including only the display target object by setting the pixel value with the class of the non-display target object to zero from the labeled IVUS image. be able to. The control unit 31 can obtain a three-dimensional image representing the display target based on the labeled IVUS image including only the object to be displayed. For example, the control unit 31 can generate a three-dimensional image showing only the blood vessel wall V and the true cavity A, map it to an angio image, and superimpose it.

Further, characters, symbols, etc. indicating the contents of the object of the image to be superimposed on the angio image may be configured to be superimposed and displayed on the angio image. Further, a transparent image of a different color may be superimposed on the angio image for each type of object.

(Modification 1)
Further, in the present embodiment, an example in which the coordinate transformation matrix is obtained by the first to third markers 11a, 11b, 11c provided on the catheter 1 has been described, but the present invention is not limited thereto. For example, the control unit 31 converts coordinates based on the marker images of the first marker 11a and the second marker 11b provided on the catheter 1 and the image of the tip of the first guide wire GW1 or the second guide wire GW2. You may find the matrix. Only the criteria for specifying the relationship of each coordinate are different, and the coordinate transformation matrix can be calculated by the method described in the embodiment.
In addition, the marker setting method is not particularly limited as long as it is a site having three or more points that can be imaged by the angiography device 102 and can specify the position and posture in the three-dimensional space.

(Modification 2)
Further, in the embodiment, an example of acquiring information on the imaging direction θ together with the angio image from the angiography apparatus 102 has been described, but the imaging direction θ is not essential information. The control unit 31 may obtain the coordinate transformation matrix without using the imaging direction θ. The control unit 31 can obtain a relative change in the imaging direction θ if the positions of the first to third marker images included in the plurality of angio images obtained from the plurality of different imaging directions θ are known. The control unit 31 can use the imaging direction in which one angio image is captured as a reference direction, and can represent the imaging direction in which another angio image is obtained by a relative angle from the reference direction. This relative angle can be obtained from two angio images, and if the coordinate transformation matrix is expressed by this relative angle, a three-dimensional image of the blood vessel can be obtained from the angiography device 102 without obtaining information on the imaging direction θ. It can be mapped and superimposed on an image.

1 Catheter 3 Image processing device 3a Recognition unit 3b Three-dimensional image generation unit 3c Coordinate conversion unit 3d Image synthesis unit 4 Display device 5 Input device 10 Ultrasonic probe 11 Shaft 10a Ultrasonic transmission / reception unit 31 Control unit 35 Learning model P Computer program A True lumen B False lumen C CTO lesion site V Blood vessel wall GW1 1st guide wire GW2 2nd guide wire 100 Diagnostic imaging device 101 Intravascular ultrasonography device 102 Angiography device

Claims (10)

A plurality of first images obtained by photographing the cross section of the luminal organ at multiple points using a catheter were acquired.
Based on the obtained plurality of first images, a three-dimensional image of the luminal organ is generated.
A plurality of second images obtained by imaging the luminal organ and the catheter from a plurality of different directions outside the living body were acquired.
Based on the acquired plurality of second images, the positional relationship between the first image and the second image is specified.
A computer program that causes a computer to execute a process of mapping and superimposing the three-dimensional image on the second image based on the specified positional relationship. The second image includes a marker image obtained by imaging a marker provided on the catheter.
The computer program according to claim 1, wherein the computer executes a process of specifying the positional relationship between the first image and the second image based on the marker images in the acquired plurality of second images. The second image includes a plurality of marker images obtained by imaging three or more markers located on a non-linear line provided on the catheter.
The computer according to claim 1 or 2, wherein the computer is made to execute a process for specifying the positional relationship between the first image and the second image based on a plurality of marker images included in each of the acquired second images. program. The second image includes a plurality of marker images obtained by imaging a plurality of markers provided on the catheter, and an image of a guide wire.
1. The computer program according to claim 2. Acquire the second image in chronological order,
The positional relationship between the first image and the acquired second image is sequentially specified, and
Any of claims 1 to 4, which causes the computer to execute a process of sequentially mapping and superimposing and displaying the three-dimensional images on each of the second images acquired in time series based on the specified positional relationship. The computer program according to item 1. The luminal organ is a blood vessel, and recognizes an image of a true lumen or a false lumen contained in a plurality of acquired first images.
Any of claims 1 to 5, which causes the computer to execute a process of generating a three-dimensional image representing at least the true cavity or the false lumen in the tubular organ based on the recognized image of the true cavity or the false lumen. Or the computer program described in paragraph 1. The luminal organ is a blood vessel, and by inputting the acquired first image into a learning model that recognizes the image of the true cavity or the false lumen included in the image, the true cavity or the false lumen included in the first image can be obtained. The computer program according to claim 6, wherein the computer performs an image recognition process. The image of the guide wire included in the acquired plurality of first images is recognized, and the image is recognized.
The computer program according to any one of claims 1 to 7, wherein the computer executes a process of generating a three-dimensional image expressing the guide wire based on the recognized image of the guide wire. A plurality of first images obtained by photographing the cross section of the luminal organ at multiple points using a catheter were acquired.
Based on the obtained plurality of first images, a three-dimensional image of the luminal organ is generated.
A plurality of second images obtained by imaging the luminal organ and the catheter from a plurality of different directions outside the living body were acquired.
Based on the acquired plurality of second images, the positional relationship between the first image and the second image is specified.
An image processing method in which a computer executes a process of mapping the three-dimensional image onto the second image and superimposing the display based on the specified positional relationship. A first acquisition unit that acquires a plurality of first images obtained by capturing a cross section of a luminal organ at multiple locations using a catheter, and a first acquisition unit.
A generator that generates a three-dimensional image of the luminal organ based on the acquired plurality of first images,
Based on the second acquisition unit that acquires a plurality of second images obtained by imaging the luminal organ and the catheter from a plurality of different directions outside the living body, and the plurality of acquired second images, the first image and the second image A specific part that specifies the positional relationship, and
An image processing device including a superimposed display unit that maps the three-dimensional image onto the second image and superimposes and displays the three-dimensional image based on the specified positional relationship.

Images