JP7489882B2 - Computer program, image processing method and image processing device - Google Patents

Description

The present disclosure relates to a computer program, an image processing method, and an image processing device.

One of the treatments for chronic total occlusion (CTO), a condition in which the coronary artery becomes chronically and completely blocked, preventing blood flow, is a method called percutaneous coronary intervention (PCI). PCI is a minimally invasive treatment that expands the blocked area with a balloon catheter and places a stent to reconstruct the blood vessel.

In PCI, the surgeon can identify the blocked area of the blood vessels using an intravascular ultrasound (IVUS) test with a catheter. In addition, the course of the blood vessels leading to the blocked area can be confirmed by an angiography test, in which X-rays are used to photograph the blood vessels while a contrast agent is injected into the blood vessels.

JP 2017-153621 A

In PCI, the antegrade first guidewire may become lost in a false lumen in the vascular wall or the like. In this case, an IVUS catheter is inserted into the lost first guidewire, and the true lumen is confirmed using an image obtained by IVUS (hereinafter referred to as an IVUS image), and a second guidewire is passed through the occluded area.

However, IVUS images are cross-sectional images of blood vessels, and it is not easy for the surgeon to mentally construct a three-dimensional image of the blood vessels and grasp the positional relationship between the IVUS catheter and the true lumen.
On the other hand, images obtained by angiography (hereinafter referred to as an angiogram) are fluoroscopic images taken from a specific direction outside the living body, so it is relatively easy to grasp the positional relationship between the blood vessels and the IVUS catheter. However, it is not possible to obtain an image of the blood vessels on the occluded side, where the contrast medium does not pass, and the position of the true lumen cannot be determined.
It is also possible to refer to the IVUS image and grasp the position of the true lumen in the angio image. Generally, the relationship between the direction in which the angio image is captured and the up, down, left, and right directions of the IVUS image is unknown, but it is also possible to confirm the positional relationship between the IVUS image and the angio image by using the position of a marker that does not transmit X-rays provided at the tip of the IVUS catheter. However, it is still not easy for the surgeon to reconstruct a three-dimensional image of the blood vessel in his mind using both images. In addition, it is not easy to operate the equipment to confirm the positional relationship between the IVUS image and the angio image.

In one aspect, the objective is to provide a computer program, an image processing method, and an image processing device that can superimpose a three-dimensional image of blood vessels on an angio image, making it easier for the surgeon to intuitively grasp the state of the blood vessels.

A computer program according to one aspect causes a computer to execute a process of acquiring a plurality of first images obtained by imaging a cross section of a tubular organ at a plurality of locations using a catheter, generating a three-dimensional image of the tubular organ based on the acquired plurality of first images, acquiring a plurality of second images obtained by imaging the tubular organ and the catheter from a plurality of different directions outside the living body, determining the positional relationship between the first and second images based on the acquired plurality of second images, and mapping the three-dimensional image onto the second image and displaying them superimposed on each other based on the determined positional relationship.

In one aspect of the image processing method, a computer executes a process of acquiring a plurality of first images obtained by imaging a cross section of a tubular organ at a plurality of locations using a catheter, generating a three-dimensional image of the tubular organ based on the acquired plurality of first images, acquiring a plurality of second images obtained by imaging the tubular organ and the catheter from a plurality of different directions outside the living body, determining the positional relationship between the first image and the second image based on the acquired plurality of second images, and mapping the three-dimensional image onto the second image and displaying them in a superimposed manner based on the determined positional relationship.

An image processing device according to one aspect includes a first acquisition unit that acquires a plurality of first images obtained by capturing images of a cross section of a tubular organ at a plurality of locations using a catheter, a generation unit that generates a three-dimensional image of the tubular organ based on the acquired plurality of first images, a second acquisition unit that acquires a plurality of second images obtained by capturing images of the tubular organ and the catheter from a plurality of different directions outside the living body, an identification unit that identifies the positional relationship between the first image and the second image based on the acquired plurality of second images, and an overlay display unit that maps the three-dimensional image onto the second image and displays it in a superimposed manner based on the identified positional relationship.

According to the present disclosure, a three-dimensional image of blood vessels can be superimposed on an angio image, making it easier for the surgeon to intuitively grasp the condition of the blood vessels.

FIG. 1 is an explanatory diagram showing an example of the configuration of an image diagnostic apparatus. FIG. 2 is a side cross-sectional view showing an example of the configuration of a catheter. FIG. 1 is a block diagram showing an example of the configuration of an image processing device. FIG. 1 is a schematic diagram showing a cross-sectional side view of a blood vessel. FIG. 1 is an explanatory diagram showing the difference between an IVUS image and an angio image. FIG. 2 is a functional block diagram of the image processing device. This is a composite image in which a three-dimensional image is projected and superimposed onto an angio image. FIG. 1 is an explanatory diagram showing an image recognition method using a learning model. FIG. 11 is an explanatory diagram showing a coordinate conversion method. FIG. 11 is an explanatory diagram showing a method for creating a coordinate transformation matrix. 13 is a flowchart showing a procedure for creating a coordinate transformation matrix. 13 is a flowchart showing an image processing procedure.

Specific examples of a computer program, an image processing method, and an image processing device according to an embodiment of the present disclosure will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is intended to include all modifications within the scope of the claims and meaning equivalent thereto. In addition, at least some of the embodiments and modifications described below may be combined in any combination.
In this embodiment, cardiac catheterization, which is an intravascular treatment, will be described as an example. However, the tubular organs that are the subject of catheterization are not limited to blood vessels, and may be other tubular organs such as the bile duct, pancreatic duct, bronchi, and intestines.

<Overall configuration of image diagnostic apparatus 100>
1 is an explanatory diagram showing an example of the configuration of an image diagnostic apparatus 100. The image diagnostic apparatus 100 according to this embodiment includes an intravascular ultrasound examination apparatus 101, an angiography apparatus 102, an image processing apparatus 3, a display apparatus 4, and an input apparatus 5. The image processing apparatus 3 according to this embodiment displays a three-dimensional image of a blood vessel (hollow organ) superimposed on an angio image, making it easier for an operator to intuitively grasp the state of the blood vessel.

The intravascular ultrasound examination device 101 is a device for generating an IVUS image (first image) including an ultrasonic cross-sectional image of a blood vessel (hollow organ) by the intravascular ultrasound (IVUS) method, and for performing intravascular ultrasound examination and diagnosis. The intravascular ultrasound examination device 101 includes a catheter 1 and an MDU (Motor Drive Unit) 2.

FIG. 2 is a side cross-sectional view showing an example of the configuration of the catheter 1. The catheter 1 according to this embodiment is an imaging diagnostic catheter for obtaining an ultrasonic tomographic image of a blood vessel by the IVUS method. The catheter 1 has an ultrasonic probe 10 at its tip for obtaining an ultrasonic tomographic image of a blood vessel. The ultrasonic probe 10 has a tube portion, and a shaft 11 is inserted into the tube portion. The shaft 11 can advance and retreat along the tube portion, and can also rotate in the circumferential direction. The tip portion of the shaft 11 is provided with an ultrasonic transmission/reception unit 10a that emits ultrasonic waves in the blood vessel and receives reflected waves (ultrasonic echoes) reflected by the biological tissue of the blood vessel or a medical device. The ultrasonic probe 10 is configured to be able to advance and retreat in the longitudinal direction of the blood vessel while rotating in the circumferential direction of the blood vessel.
The catheter 1 has a guidewire lumen at its tip, through which the first guidewire GW1 is inserted. The center line of the tubular portion of the guidewire lumen and the center line of the tubular portion of the ultrasound probe 10 are spaced apart by a predetermined distance.

The catheter 1 also has first to third markers 11a, 11b, and 11c that do not transmit X-rays in order to determine the positional relationship between the IVUS image obtained by the intravascular ultrasound examination device 101 and the angio image obtained by the angiography device 102. The first to third markers 11a, 11b, and 11c are arranged in a non-linear manner so as not to be arranged in a straight line. For example, the first marker 11a and the second marker 11b are arranged at the tip of the shaft 11, sandwiching the ultrasound transmission/reception unit 10a therebetween. The third marker 11c is arranged in the guidewire lumen. When the catheter 1 configured in this way is imaged with X-rays, an angio image is obtained, which is an X-ray fluoroscopic image including images of the first to third markers 11a, 11b, and 11c, as shown in the lower diagram of FIG. 2, for example.
The positions where the first to third markers 11a, 11b, and 11c are provided are merely examples. The first marker 11a and the second marker 11b may be provided on the catheter 1 body having a tubular portion through which the shaft 11 is inserted, instead of on the shaft 11. In other words, the first to third markers 11a, 11b, and 11c may be provided at constant positions regardless of the position of the shaft 11.

The MDU 2 is a drive unit to which the catheter 1 is detachably attached, and controls the operation of the catheter 1 inserted into the blood vessel by driving a built-in motor in response to the operation of a medical professional. The MDU 2 rotates the ultrasound transmission/reception unit 10a of the catheter 1 in the circumferential direction while moving it from the tip (distal) side to the base (proximal) side (see Figure 8). The ultrasound probe 10 continuously scans the inside of the blood vessel at a predetermined time interval, and outputs an IVUS image based on the reflected wave data of the detected ultrasound to the image processing device 3.

Based on the reflected wave data output from the ultrasound probe 10 of the catheter 1, the image processing device 3 generates multiple chronologically ordered IVUS images including ultrasound cross-sectional images of transverse layers of the blood vessel (see FIG. 8). The ultrasound probe 10 scans the blood vessel while moving from the tip (distal) side to the base (proximal) side, so the multiple chronologically ordered IVUS images are cross-sectional images of the blood vessel observed at multiple locations from the distal to proximal ends.

The angiography device 102 is an imaging device that uses X-rays to image the blood vessels of a patient from outside the patient's body while injecting a contrast agent into the blood vessels, and obtains an angio image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and images an X-ray fluoroscopic image of the patient by the X-ray sensor receiving X-rays irradiated from the X-ray source. The angiography device 102 outputs the angio image obtained by imaging to the image processing device 3.

When the angiography device 102 is used to image a blood vessel into which the catheter 1 is inserted, an angio image including images of the blood vessel, the catheter 1, and the first guidewire GW1 is obtained. When the second guidewire GW2 described below is inserted into the blood vessel, an angio image including images of the blood vessel, the catheter 1, the first guidewire GW1, and the second guidewire GW2 is obtained. In addition, as described above, the catheter 1 is provided with the first to third markers 11a, 11b, and 11c that are opaque to X-rays, and an angio image including the first marker image, the second marker image, and the third marker image is obtained.

In this embodiment, the angiography device 102 that mainly captures two-dimensional angiographic images will be described as an example, but there is no particular limitation as long as it is a device that captures images of the patient's tubular organs and catheter 1 from multiple directions outside the body. For example, it may be a three-dimensional CT angiography, a magnetic resonance imaging (MRI) image, etc.

The display device 4 is a liquid crystal display panel, an organic EL display panel, or the like, and displays medical images such as IVUS images and angio images generated by the image processing device 3.

The input device 5 is an input interface such as a keyboard or mouse that accepts input of various setting values when performing an inspection and operations of the image processing device 3. The input device 5 may be a touch panel, soft keys, hard keys, etc. provided on the display device 4.

<Hardware configuration of image processing device 3>
3 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a control unit 31, a main storage unit 32, an input/output I/F 33, and an auxiliary storage unit .

The control unit 31 includes one or more central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), and tensor processing units (TPUs).
The present invention is configured using a processor such as the above.

The main memory unit 32 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 31 to execute arithmetic processing.

The input/output I/F 33 is an interface to which the intravascular ultrasound device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images or angio images via the input/output I/F 33. The control unit 31 also displays medical images on the display device 4 by outputting medical image signals of the IVUS images or angio images to the display device 4 via the input/output I/F 33. Furthermore, the control unit 31 accepts information input to the input device 5 via the input/output I/F 33.

The auxiliary memory unit 34 is a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The auxiliary memory unit 34 stores the computer program P executed by the control unit 31 and various data required for the processing of the control unit 31. The auxiliary memory unit 34 also stores the learning model 35. The learning model 35 will be described in detail later.

The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit 34 during the manufacturing stage of the image processing device 3, or the image processing device 3 may obtain the computer program P via communication from a remote server device and store it in the auxiliary storage unit 34. The computer program P may be recorded in a readable manner on the recording medium 30, such as a magnetic disk, optical disk, or semiconductor memory.

The control unit 31 reads and executes the computer program P stored in the auxiliary memory unit 34 to acquire the IVUS image generated by the imaging diagnostic device 100, generate a three-dimensional image of the blood vessel having the CTO lesion site C, and perform processing to map the generated three-dimensional image onto the angio image and display it superimposed.

The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed using software. In the following explanation, the image processing device 3 is described as being a single computer.

<Relationship between CTO lesion site and angio and IVUS images>
Before describing the functions of the image processing device 3, the relationship between chronic total occlusion lesions and angio images and IVUS images will be described.

Fig. 4 is a schematic diagram showing a side cross section of a blood vessel. In the blood vessel shown in Fig. 4, a CTO (Chronic Total Occlusion) lesion site C is seen, in which the coronary artery is chronically and completely blocked, preventing blood flow. In Fig. 4, the vascular occlusion site to the right of the dashed line is the CTO lesion site C.
In PCI treatment of a CTO lesion site C, it is necessary to grasp the position of the true lumen A in an occluded blood vessel and insert a guidewire into the true lumen A. Meanwhile, in PCI, the antegrade first guidewire GW1 may stray into the false lumen B. FIG. 4 shows a state in which the first guidewire GW1 has strayed into the false lumen B of a blood vessel wall V. In this case, the catheter 1 of the intravascular ultrasound inspection device 101 is inserted into the stray first guidewire GW1, and an IVUS image is obtained to confirm the positions of the blood vessel wall V, the true lumen A, and the false lumen B. The surgeon grasps the position of the true lumen A, and inserts the second guidewire GW2 into the CTO lesion site C and the true lumen A as shown in FIG. 4.
However, it is not easy to determine the positions of the true lumen A and false lumen B from IVUS images and angio images.

5A and 5B are explanatory diagrams showing the difference between an IVUS image and an angio image. Fig. 5A is a three-dimensional image of a blood vessel. Fig. 5B is a schematic diagram of an angio image, and Fig. 5C is a schematic diagram of an IVUS image.
As shown in Figures 5A and 5B, an angio image is a fluoroscopic image obtained by imaging a blood vessel from the lateral direction (diametrically outward), and is suitable for understanding the course of the blood vessel. However, since the contrast agent does not pass through the CTO lesion site C, it is not possible to obtain images of the blood vessel wall V, true lumen A, and false lumen B in the portion indicated by the dashed line in Figure 5B. Only images of the catheter 1 and the first guidewire GW1 can be seen.
On the other hand, as shown in Figures 5A and 5C, an IVUS image is a cross-sectional image of a blood vessel (a cross-sectional image cut along a plane perpendicular to the center line of the blood vessel), and the positions of the CTO lesion site C, true lumen A, false lumen B, catheter 1, and first guidewire GW1 can be confirmed. In addition, if an image of a site where a second guidewire GW2 (not shown) is present is taken, the position of the second guidewire GW2 can also be confirmed. However, since an IVUS image is a cross-sectional image of a blood vessel, it is not easy for the surgeon to mentally construct a three-dimensional image of the blood vessel and grasp the positional relationship between the catheter 1 and the true lumen A.

In addition, an IVUS image is an image obtained by emitting ultrasound and detecting reflected waves while rotating the ultrasound probe 10 of the catheter 1 in the circumferential direction, and the up, down, left and right directions are not fixed. For this reason, as shown by the white arrow in Figure 5C, it is not immediately clear from which direction the blood vessels in the IVUS image were imaged. The surgeon must obtain angio images taken from multiple different directions, estimate the positional relationship between the IVUS image and the angio images from the images of the catheter 1, the first guidewire GW1, and the first to third markers 11a, 11b, 11c, and mentally reconstruct a three-dimensional image of the true lumen A at the CTO lesion site C, which is not an easy task.

Therefore, the image processing device 3 according to this embodiment generates a three-dimensional image of the blood vessels at the CTO lesion site C based on the IVUS image, and performs processing to map and superimpose the three-dimensional image onto the angio image for display.

<Functional block diagram of image processing device 3>
Fig. 6 is a functional block diagram of the image processing device 3, and Fig. 7 shows a composite image in which a three-dimensional image is mapped and superimposed on an angio image. The control unit 31 of the image processing device 3 reads out and executes a computer program P stored in the auxiliary storage unit 34, thereby functioning as a recognition unit 3a, a three-dimensional image generation unit 3b, a coordinate conversion unit 3c, and an image synthesis unit 3d.

The recognition unit 3a executes a process of recognizing a predetermined object included in the IVUS image using the learning model 35. For example, the recognition unit 3a recognizes an image of the vascular wall V, an image of the true lumen A, an image of the false lumen B, an image of the catheter 1, and an image of the first guidewire GW1 and the second guidewire GW2 included in the IVUS image. The image recognized by the recognition unit 3a is an image having pixel values according to the type of object, that is, an image labeled by pixel values according to the type of object (hereinafter referred to as a labeled IVUS image). The recognition unit 3a executes a recognition process for each of the multiple IVUS images, and outputs the multiple labeled IVUS images obtained by the recognition process to the three-dimensional image generation unit 3b.

The three-dimensional image generating unit 3b generates a three-dimensional image representing the blood vessels of the CTO lesion site C based on multiple labeled IVUS images, and outputs the generated three-dimensional image to the coordinate conversion unit 3c. The three-dimensional image can be generated, for example, by the voxel method. The three-dimensional image is represented by volume data represented by the coordinate values of voxels in a predetermined coordinate system and voxel values indicating the type of object. The data format of the three-dimensional image is not particularly limited, and may be polygon data or point cloud data.

The coordinate conversion unit 3c converts the three-dimensional image into a two-dimensional coordinate system for the angio image, and outputs the two-dimensional image after the coordinate conversion to the image synthesis unit 3d. Details of the coordinate conversion will be described later.

The image synthesis unit 3d executes synthesis processing to superimpose the coordinate-converted two-dimensional image of the blood vessels on the angio image as shown in FIG. 7. The image processing unit may make the two-dimensional image of the blood vessels semi-transparent and superimpose it on the angio image. In other words, the three-dimensional image of the blood vessels is mapped onto the angio image and superimposed by the processing of the coordinate conversion unit 3c and the image synthesis unit 3d.

<Learning Model 35>
The learning model 35 is a model that recognizes a predetermined object included in an IVUS image. The learning model 35 can classify objects on a pixel-by-pixel basis by utilizing an image recognition technique using, for example, semantic segmentation, and can recognize a CTO lesion site C included in an IVUS image, specifically, a blood vessel wall V, a true lumen A, a false lumen B, a catheter 1, a first guidewire GW1, and a second guidewire GW2.

Figure 8 is an explanatory diagram showing an image recognition method using the learning model 35. The learning model 35 is trained to be able to recognize images of the vascular wall V, true lumen A, false lumen B, catheter 1, first guidewire GW1, and second guidewire GW2 in an IVUS image on a pixel-by-pixel basis.

The learning model 35 is, for example, a convolutional neural network (CNN) that has been trained by deep learning. The learning model 35 recognizes an object on a pixel-by-pixel basis by an image recognition technique using so-called semantic segmentation.
The learning model 35 has an input layer 35a to which an IVUS image is input, an intermediate layer 35b to extract and restore image features, and an output layer 35c to output a labeled IVUS image indicating an object included in the IVUS image in pixel units. The learning model 35 is, for example, a U-Net.

The input layer 35a of the learning model 35 has a plurality of neurons that accept input of pixel values of each pixel included in the IVUS image, and passes the input pixel values to the intermediate layer 35b. The intermediate layer 35b has a convolution layer (CONV layer) and a deconvolution layer (DECONV layer). The convolution layer is a layer that compresses the dimensions of image data. The feature amount of the object is extracted by the dimensional compression. The deconvolution layer performs deconvolution processing and restores the original dimensions. The restoration processing in the deconvolution layer generates a labeled IVUS image in which each pixel in the image is a specified object or not, expressed by a pixel value according to the type of object. The output layer 35c has a plurality of neurons that output the labeled IVUS image. The labeled IVUS image is, for example, an image in which pixels corresponding to the vascular wall V are class "1," pixels corresponding to the true lumen A are class "2," pixels corresponding to the false lumen B are class "3," pixels corresponding to the catheter 1 are class "4," pixels corresponding to the first and second guidewires GW1 and GW2 are class "5," and pixels corresponding to other images are class "0."

The learning model 35 can be generated by preparing training data including an IVUS image including the vascular wall V, the catheter 1, the guidewire, the true lumen A, and the false lumen B, and a labeled IVUS image showing each object image in the IVUS image, and by using the training data to perform machine learning on an untrained neural network. Specifically, the control unit 31 inputs a plurality of IVUS images included in the training data to the input layer 35a of the neural network model before learning, and obtains an image output from the output layer 35c after arithmetic processing in the intermediate layer 35b. The control unit 31 then compares the image output from the output layer 35c with the labeled IVUS image included in the training data, and optimizes parameters used in the arithmetic processing in the intermediate layer 35b so that the image output from the output layer 35c approaches the labeled IVUS image. The parameters are, for example, weights (coupling coefficients) between neurons. The method of optimizing the parameters is not particularly limited, but the control unit 31 optimizes various parameters using, for example, the steepest descent method, the error backpropagation method, or the like.

According to the learning model 35 trained in this manner, by inputting an IVUS image into the learning model 35, a labeled IVUS image showing the vascular wall V, catheter 1, guidewire, true lumen A, and false lumen B in pixel units is obtained.

<Coordinate conversion and superimposed display of 3D images>
Fig. 9 is an explanatory diagram showing the coordinate conversion method. In the following, for ease of understanding, a world coordinate system, which is a fixed, predetermined three-dimensional orthogonal coordinate system, is introduced, and a three-dimensional image of a blood vessel is mapped onto an angio image via the world coordinate system. In Fig. 9, black circles indicate images of the first to third markers 11a, 11b, and 11c. For ease of explanation, the images of the first to third markers 11a, 11b, and 11c are assumed to be images of each marker before blood vessel scanning by the catheter 1, that is, before the shaft 11 is moved.

In FIG. 9, the angio image shown in the upper right is represented in a two-dimensional angio coordinate system. The Xa and Ya axes of the angio coordinate system are, for example, the horizontal line (horizontal axis) and vertical line (vertical axis) of the image displayed on the display device 4. In FIG. 9, (xa, ya) are the coordinate values of the pixels that make up the angio image in the angio coordinate system.

The positions of the first to third markers 11a, 11b, and 11c in the angio image change depending on the imaging direction. Therefore, we consider the absolute positions of the first to third markers 11a, 11b, and 11c in the world coordinate system. The world coordinate system is an arbitrary three-dimensional orthogonal coordinate system. The method for setting the world coordinate system is arbitrary, but for example, it can be set so that the imaging direction by the angiography device 102 is the origin and the imaging direction changes around the Xw axis.

Before vascular scanning, the positions of the first to third markers 11a, 11b, and 11c provided on the catheter 1 are fixed. At least, the positional relationship between the first and second markers 11a and 11b is constant, and the positional relationship between a straight line passing through the first and second markers 11a and 11b and the third marker 11c is constant.
Therefore, the coordinate positions (xw, yw, zw) of the first to third markers 11a, 11b, and 11c in the world coordinate system can be obtained from the coordinate positions of the first to third marker images in an angio image obtained by imaging from a plurality of different imaging directions θ. Also, a coordinate transformation matrix fwa that transforms the coordinates (xw, yw, zw) in the world coordinate system into the coordinates (xa, ya, za) in the angio coordinate system can be obtained. The coordinate transformation matrix fwa can be expressed as a product of a rotation matrix, a translation matrix, an orthogonal projection matrix, a scaling matrix, and the like. The coordinate transformation matrix fwa can be expressed as a function of the imaging direction θ by the angiography device 102.

On the other hand, the three-dimensional image generated based on the IVUS image is mapped to the IVUS coordinate system. The IVUS coordinate system is, for example, a three-dimensional orthogonal coordinate system. The IVUS coordinate system can be arbitrarily taken. For example, a three-dimensional coordinate system in which the first marker and the second marker pass through the Zi axis and the third marker passes through the Xi axis can be considered. Specifically, the three-dimensional coordinates may be mapped to the IVUS coordinate system so that the X-coordinate value and the Y-coordinate value of the center line of the catheter 1 in the three-dimensional image, in other words, the output point of the ultrasound, of a plurality of voxels are set to zero, and the Y-coordinate value of the voxels with the Z-axis coordinate value of zero among the voxels constituting the first guide wire GW1 in the three-dimensional image is zero. In FIG. 9, (xi, yi, zi) are coordinate values of the voxels constituting the three-dimensional image in the IVUS coordinate system.
When the ultrasound probe 10 moves, the positions of the first marker and the second marker change, but since the first marker and the second marker are always on the Zi axis, a three-dimensional image can be mapped onto a single IVUS coordinate system without any problems.
By aligning the positions of the first to third markers 11a, 11b, and 11c with the IVUS coordinate system in this manner, it is possible to map three-dimensional coordinates onto the IVUS coordinate system, as if the catheter 1 were aligned with a predetermined orientation.

As described above, the positions of the first to third markers 11a, 11b, and 11c in the IVUS coordinate system are known, and the coordinate positions of the first to third markers 11a, 11b, and 11c in the world coordinate system have also been determined, so it is possible to determine the coordinate transformation matrix fiw that converts the coordinates (xi, yi, zi) in the IVUS coordinate system into the coordinates (xw, yw, zw) in the world coordinate system. This coordinate transformation matrix fiw can be expressed as the product of a rotation matrix, a translation matrix, and a scaling matrix.

In the end, the coordinate transformation matrix fia that maps a three-dimensional image in the IVUS coordinate system to the angio coordinate system can be expressed as the product of the coordinate transformation matrix fiw and the coordinate transformation matrix fwa.

<Process of deriving coordinate transformation matrix>
Fig. 10 is an explanatory diagram showing a method for creating the coordinate transformation matrix fia, and Fig. 11 is a flowchart showing a process for creating the coordinate transformation matrix. The control unit 31 acquires angio images obtained by imaging from a plurality of different imaging directions θ using the angiography device 102 and the imaging directions θ (step S11). Next, the control unit 31 judges whether angio images have been captured from a predetermined number of directions (step S12). The predetermined number is 2 or more, and the more imaging directions there are, the higher the accuracy of the coordinate transformation matrix, i.e., the accuracy of mapping a three-dimensional image of blood vessels onto an angio image.

If angiographic images captured from the predetermined number of directions have not been obtained (step S12: NO), the control unit 31 outputs a signal to the angiography device 102 to instruct the angiography device 102 to change the imaging direction θ (step S13), and returns the process to step S11. Note that if the angiography device 102 automatically changes the imaging direction θ continuously, the process of step S13 is not necessary.
In addition, the control unit 31 that executes the processing of steps S11 to S13 functions as a second acquisition unit that acquires multiple angio-images (second images) of blood vessels (hollow organs) and catheter 1 from multiple different directions outside the living body.

When multiple angio images captured from a predetermined number of directions have been obtained (step S12: YES), the control unit 31 calculates the coordinate transformation matrix fia using the coordinate values of the first to third markers 11a, 11b, 11c and the imaging direction θ in the multiple angio images captured from multiple different imaging directions θ (step S14), and terminates the processing.
In addition, the control unit 31 that executes the processing of step S14 functions as an identification unit that identifies the positional relationship between the IVUS image (first image) and the angio image (second image) based on the multiple acquired angio images (second images).

Breaking it down, once the coordinate values of the first to third markers 11a, 11b, and 11c in the world coordinate system are obtained, the coordinate transformation matrix fiw from the IVUS coordinate system to the world coordinate system can be obtained. Also, once the coordinate values of the first to third markers 11a, 11b, and 11c in the world coordinate system are obtained, the control unit 31 can obtain the coordinate transformation matrix fwa, for example, by the maximum likelihood estimation method, so that the coordinate values of the first to third marker images obtained by the coordinate transformation matrix fwa (a function of θ) from the world coordinate system to the angio coordinate system match the coordinate values of the first to third marker images in the angio image obtained by imaging from the imaging direction θ. Then, by obtaining the product of the coordinate transformation matrix fiw and the coordinate transformation matrix fwa, the coordinate transformation matrix fia that maps the three-dimensional image in the IVUS coordinate system to the angio coordinate system can be obtained.

In addition, the image of the blood vessels mapped onto the angio coordinate system is a two-dimensional image that reflects the position, size, orientation and inclination of the blood vessels in three-dimensional space. By superimposing this two-dimensional image on the angio image, the surgeon can easily understand the three-dimensional structure of the blood vessels in the angio image.

<3D image generation and superimposed display processing>
12 is a flowchart showing the image processing procedure. The image processing device 3 will be described on the assumption that a learned learning model 35 has been learned and a coordinate transformation matrix has been created in the process of FIG.

The control unit 31 acquires multiple IVUS images output from the intravascular ultrasound device 101 (step S31). The control unit 31 that executes the process of step S31 functions as a first acquisition unit that acquires multiple IVUS images (first images) of a cross section of a blood vessel (hollow organ) captured at multiple locations using a catheter.

Next, the control unit 31 recognizes objects included in the IVUS images by inputting the acquired IVUS images into the learning model 35 (step S32). The control unit 31 then generates a three-dimensional image of the blood vessel based on the labeled IVUS images obtained by the object recognition process of step S32 (step S33). The three-dimensional image is, for example, an image that three-dimensionally represents the vascular wall V, the true lumen A, the false lumen B, the catheter 1, the first guidewire GW1, and the second guidewire GW2.
The control unit 31 that executes the processes of steps S31 and S33 functions as a generating unit that generates a three-dimensional image of a blood vessel (hollow organ) based on a plurality of acquired IVUS images (first images).

The control unit 31 then acquires the angio image and the imaging direction θ from the angiography device 102 (step S34). The control unit 31 obtains a coordinate transformation matrix based on the imaging direction θ, and converts the three-dimensional image in the IVUS coordinate system into a two-dimensional image in the angio coordinate system using the coordinate transformation matrix (step S35), superimposes the two-dimensional image of the blood vessels mapped onto the angio coordinate system on the angio image (step S36), and displays a composite image obtained by mapping the three-dimensional image of the blood vessels onto the angio image in a display process (step S37).
The control unit 31 that executes the processes of steps S35 and S36 functions as a superimposition display unit that maps the three-dimensional image onto the angio image (second image) and displays the image in a superimposed manner based on the identified positional relationship.

Next, the control unit 31 determines whether or not to end the process (step S38). If it is determined that the process should be ended (step S38: YES), the control unit 31 ends the superimposition process of the 3D blood vessel image. If it is determined that the process should not be ended (step S38: NO), the control unit 31 returns the process to step S34.

As described above, the image processing device 3 according to this embodiment can project a 3D image of blood vessels onto an angio image and display it in a superimposed manner. This makes it easier for the surgeon to intuitively grasp the state of the blood vessels.

In addition, by calculating a coordinate transformation matrix based on the coordinate values of the first to third markers 11a, 11b, and 11c provided on the catheter 1, a three-dimensional image of the blood vessels can be accurately mapped onto the angio image and displayed in a superimposed manner.

Furthermore, the vascular wall V, true lumen A, false lumen B, catheter 1, first guidewire GW1, and second guidewire GW2 contained in the IVUS image can be recognized to generate a three-dimensional image, which can be mapped onto the angio image and displayed superimposed thereon. Therefore, the surgeon can easily recognize the positions of the vascular wall V, true lumen A, false lumen B, and catheter 1 in the angio image.
Furthermore, since the image processing device 3 is configured to map each object contained in the three-dimensional image onto the angio image as a transparent image and display it superimposed, it is possible to more accurately grasp the relationship between the image of the angio image and objects such as the true lumen A and false lumen B of the blood vessels based on the IVUS image.

Furthermore, by repeatedly executing the processes of steps S34 to S37, it is possible to acquire angio images and imaging directions θ in chronological order, obtain coordinate transformed images according to the imaging direction θ each time, and superimpose a three-dimensional image on the angio images acquired in chronological order. Even if the position of the blood vessel in the angio image changes due to transformation of the imaging direction θ, the three-dimensional image can be mapped and superimposed at the corresponding position so as to follow the transformation of the blood vessel position.

The image processing device 3 does not need to superimpose all objects such as the vascular wall V, true lumen A, false lumen B, catheter 1, first guidewire GW1, and second guidewire GW2 included in the IVUS image on the angio image, and may be configured to receive objects to be displayed from the user. The image processing device 3 generates a three-dimensional image including the received objects to be displayed. Specifically, the control unit 31 can obtain a labeled IVUS image including only the objects to be displayed by setting pixel values assigned to classes of objects not to be displayed to zero from the labeled IVUS image. The control unit 31 can obtain a three-dimensional image representing the objects to be displayed based on the labeled IVUS image including only the objects to be displayed. For example, the control unit 31 can generate a three-dimensional image representing only the vascular wall V and true lumen A, map it onto the angio image, and superimpose it.

Also, the system may be configured to superimpose characters, symbols, etc., indicating the contents of the object of the image to be superimposed on the angio image on the angio image. Furthermore, the system may be configured to superimpose transparent images of different colors on the angio image for each type of object.

(Variation 1)
In addition, in the present embodiment, an example has been described in which a coordinate transformation matrix is calculated using the first to third markers 11a, 11b, and 11c provided on the catheter 1, but the present invention is not limited to this. For example, the control unit 31 may calculate the coordinate transformation matrix based on marker images of the first marker 11a and the second marker 11b provided on the catheter 1 and an image of the tip of the first guidewire GW1 or the second guidewire GW2. The only difference is the criteria for identifying the relationship between the coordinates, and the coordinate transformation matrix can be calculated by the method described in the embodiment.
In addition, the method of setting the markers is not particularly limited as long as the markers are three or more points that can be imaged by the angiography device 102 and whose positions and orientations in three-dimensional space can be identified.

(Variation 2)
Furthermore, in the embodiment, an example has been described in which information on the imaging direction θ is obtained from the angiography device 102 together with the angio image, but the imaging direction θ is not essential information. The control unit 31 may obtain the coordinate transformation matrix without using the imaging direction θ. If the positions of the first to third marker images included in the multiple angio images obtained from multiple different imaging directions θ are known, the control unit 31 can obtain a relative change in the imaging direction θ. The control unit 31 can represent the imaging direction in which one angio image is captured as a reference direction, and the imaging direction in which another angio image is obtained as a relative angle from the reference direction. This relative angle can be obtained from two angio images, and if the coordinate transformation matrix is expressed by this relative angle, a three-dimensional image of blood vessels can be mapped and superimposed on the angio image without obtaining information on the imaging direction θ from the angiography device 102.

REFERENCE SIGNS LIST 1 Catheter 3 Image processing device 3a Recognition unit 3b Three-dimensional image generation unit 3c Coordinate conversion unit 3d Image synthesis unit 4 Display device 5 Input device 10 Ultrasonic probe 11 Shaft 10a Ultrasonic transmission/reception unit 31 Control unit 35 Learning model P Computer program A True lumen B False lumen C CTO lesion site V Vascular wall GW1 First guide wire GW2 Second guide wire 100 Imaging diagnostic device 101 Intravascular ultrasound examination device 102 Angiography device

Claims (9)

A plurality of first images are obtained by capturing images of a cross section of a blood vessel at a plurality of locations using a catheter;
Recognizing an image of the true lumen or false lumen included in the acquired plurality of first images;
generating a three-dimensional image representing at least the true lumen or false lumen in the blood vessel based on the image of the recognized true lumen or false lumen;
acquiring a plurality of second images of the blood vessel and the catheter taken from a plurality of different directions outside the living body;
Identifying a positional relationship between the first image and the second image based on the acquired plurality of second images;
A computer program that causes a computer to execute a process of mapping the three-dimensional image onto the second image and displaying the image in a superimposed manner based on the specified positional relationship. the second image includes a marker image obtained by capturing an image of a marker provided on the catheter,
The computer program product according to claim 1 , further comprising: determining a positional relationship between the first image and the second image based on the marker images in the acquired second images. the second image includes a plurality of marker images obtained by imaging three or more markers that are provided on the catheter and positioned on a non-linear line,
The computer program product according to claim 1 or 2, which causes the computer to execute a process of identifying a positional relationship between the first image and the second image based on a plurality of marker images included in each of the plurality of acquired second images. The second image includes a plurality of marker images obtained by capturing images of a plurality of markers provided on the catheter, and an image of a guidewire,
The computer program according to claim 1 or 2, which causes the computer to execute a process of identifying a positional relationship between the first image and the second image based on the plurality of marker images and the image of the guide wire included in each of the plurality of acquired second images. acquiring second images in a time series;
sequentially identifying a positional relationship between the first image and the acquired second image;
The computer program according to any one of claims 1 to 4, which causes the computer to execute a process of sequentially mapping and superimposing the three-dimensional image on each of the second images acquired in a time series based on the specified positional relationship. The computer program according to any one of claims 1 to 5, which causes the computer to execute a process of recognizing an image of a true lumen or a false lumen contained in a first image by inputting the acquired first image into a learning model that recognizes an image of a true lumen or a false lumen contained in the image. Recognizing an image of the guidewire included in the acquired plurality of first images;
The computer program according to any one of claims 1 to 6, which causes the computer to execute a process of generating a three-dimensional image representing the guidewire based on the recognized image of the guidewire. A plurality of first images are obtained by capturing images of a cross section of a blood vessel at a plurality of locations using a catheter;
Recognizing an image of the true lumen or false lumen included in the acquired plurality of first images;
generating a three-dimensional image representing at least the true lumen or false lumen in the blood vessel based on the image of the recognized true lumen or false lumen;
acquiring a plurality of second images of the blood vessel and the catheter taken from a plurality of different directions outside the living body;
Identifying a positional relationship between the first image and the second image based on the acquired plurality of second images;
based on the specified positional relationship, the three-dimensional image is mapped onto the second image and displayed in a superimposed manner. a first acquisition unit that acquires a plurality of first images by capturing images of a cross section of a blood vessel at a plurality of locations using a catheter;
a recognition unit that recognizes an image of a true lumen or a false lumen included in the acquired plurality of first images;
a generating unit that generates a three-dimensional image representing at least the true lumen or false lumen in the blood vessel based on the image of the recognized true lumen or false lumen;
a second acquisition unit that acquires a plurality of second images obtained by capturing images of the blood vessel and the catheter from a plurality of different directions outside the living body; and a determination unit that determines a positional relationship between the first image and the second image based on the acquired plurality of second images.
and a superimposition display unit that maps the three-dimensional image onto the second image and displays the three-dimensional image superimposed on the second image based on the specified positional relationship.

Images