EP4690092A1

EP4690092A1 - X-ray fadeout in angiographic image overlay

Info

Publication number: EP4690092A1
Application number: EP25725024.1A
Authority: EP
Inventors: Mona KUCHENBROD; Ferdinand STORCH
Original assignee: Brainlab SE
Current assignee: Brainlab SE
Priority date: 2024-06-07
Filing date: 2025-05-16
Publication date: 2026-02-11
Also published as: WO2025252437A1; WO2025252315A1

Abstract

An image pair comprising angiographic image data and bone image data showing the same area of a patient and taken from the same viewing direction onto the patient is acquired and overlaid over a live fluoroscopic image. In order to increase the perceivability of information in the angiographic image and/or the live fluoroscopic image in the overlay, a part of the bone image data is altered, for example attenuated, when the overlay is calculated.

Description

Brainlab AG

X-RAY FADEOUT IN ANGIOGRAPHIC IMAGE OVERLAY

FIELD OF THE INVENTION

The present invention relates to a computer-implemented method of overlaying angiographic information and fluoroscopic information over a fluoroscopic image of a patient, a corresponding computer program, a computer-readable storage medium storing such a program and a computer executing the program, as well as a medical system comprising an electronic data storage device and the aforementioned computer.

TECHNICAL BACKGROUND

Angiographic images representing at least a part of the vascular system of a patient are helpful tools in medical applications, for example for monitoring a medical intervention. But capturing angiographic images involves the administration of a contrast agent, which can be toxic to the patient, such that capturing a sequence of live angiographic images is disadvantageous. This document relates to an alternative in which angiographic imaging is performed only once and the vascular system comprised in the angiographic images is overlayed over live fluoroscopic images of the patient. In this context, the vascular system consists of multiple vascular structures, like vessels or bifurcations.

The present invention can be used for PAD (Peripheral Artery Disease) procedures e.g. in connection with a medical navigation system of Brainlab AG.

Aspects of the present invention, examples and exemplary steps and their embodiments are disclosed in the following. Different exemplary features of the invention can be combined in accordance with the invention wherever technically expedient and feasible. EXEMPLARY SHORT DESCRIPTION OF THE INVENTION

In the following, a short description of the specific features of the present invention is given which shall not be understood to limit the invention only to the features or a combination of the features described in this section.

According to the present invention, a pair comprising angiographic image data and bone image data showing the same area of the patient and taken from the same viewing direction onto the patient is acquired and overlaid over a live fluoroscopic image. In order to increase the perceivability of information in the angiographic image and/or the live fluoroscopic image in the overlay, a part of the bone image data is altered, for example attenuated, when the overlay is calculated.

GENERAL DESCRIPTION OF THE INVENTION

In this section, a description of the general features of the present invention is given for example by referring to possible embodiments of the invention.

In general, the invention reaches the aforementioned object by providing, in a first aspect, a computer-implemented medical method of overlaying angiographic information over a fluoroscopic image. The method comprises executing, on at least one processor of at least one computer, the following exemplary steps which are executed by the at least one processor.

In a (for example first) exemplary step, fluoroscopic image data representing the fluoroscopic image are acquired. The fluoroscopic image is from now on referred to as live fluoroscopic image or live image to distinguish it from other fluoroscopic images. The fluoroscopic image data are thus referred to as live fluoroscopic image data or live image data in analogy.

In a (for example second) exemplary step, angiographic image data representing an angiographic image are acquired. The angiographic image shows parts of the vascular system of the patient. In a (for example third) exemplary step, bone image data representing a bone image are acquired, wherein the bone image and the angiographic image show the same area of the patient and are taken from the same viewing position relative to the patient and wherein the fluoroscopic image shows at least a part of the area of the patient which is shown by the bone image and the angiographic image.

The bone image represents parts of the bone structure of the patient. The bone image can for example be a fluoroscopic image or a generic bone image, like a bone outline or a digitally reconstructed radiograph (DRR). The bone outline represents the contour of one or more bones or parts thereof. The bone outline may be derived from a fluoroscopic image, for example by image processing. The bone image is not derived from the live image, but from an earlier fluoroscopic image. A DRR is a 2D image generated from a 3D dataset

The angiographic image and the bone image form an image pair. They show the same situation of the patient, but comprise different information.

The subsequent explanation assumes that the bone image is a fluoroscopic image, but also encompasses the case in which the bone image is derived from the fluoroscopic image discussed below or is a DRR.

The angiographic image and the bone image are captured using a medical imaging system such as an X-ray imaging system. The angiographic image is taken under administration of a contrast agent, while the bone image is taken without the administration of a contrast agent. It shall be pointed out that the present invention does not relate to capturing the bone image or the angiographic image, and in particular not to administering a contrast agent, but rather to image processing in order to combine angiographic information with a live fluoroscopic image.

The bone image and the angiographic image of an image pair are preferably taken with the same position of the medical imaging device relative to the patient, and thus, from the same position and the same viewing direction onto the patient. This means that the bone image and the angiographic image of an image pair are spatially aligned and thus congruent. They thus show the same part of the patient. The bone image and the angiographic image are preferably captured directly after each other, which means for example within 1 second, 2 seconds, 5 seconds or 10 seconds.

A raw angiographic image basically shows the same content as a fluoroscopic image, but in addition the vascular system due to the contrast agent. The angiographic image represented by the angiographic image data is typically generated by subtracting the corresponding fluoroscopic image, which is also referred to as mask image, from the raw angiographic image as captured, such that only the difference, which is caused by the contrast agent in the vascular system of the patient, remains in the angiographic image. This is referred to as digital subtraction angiography (DSA). However, the angiographic image can also be the raw angiographic image.

The raw angiographic image may be a combined image based on multiple single images. The contrast agent does typically not fill the entire vascular structure to be imaged at once. A small amount of contrast agent propagates through the vascular structure and is imaged by multiple images captured subsequentially over time. The images each showing a part of the vascular structure are then combined into the raw angiographic image.

Instead of combining the single images into the raw angiographic images and then subtracting the mask image, the mask image might be subtracted from the single images and the resulting subtracted single images might be combined into the angiographic DSA image.

Typically, the bone image and the angiographic of an image pair are captured by positioning the imaging device relative to the patient, capturing a fluoroscopic image, administering a contrast agent and capturing a raw angiographic image. The angiographic image is calculated from the fluoroscopic image and the raw angiographic image as described above. In an alternative, the fluoroscopic image can be captured after capturing the raw angiographic image.

The bone image of the image pair being a fluoroscopic image and the live fluoroscopic image are for example captured using the same medical imaging system. They may have the same image sizes, given as the number of pixels, or a ratio of image sizes between, for example, 0.7 and 1.3, between 0.8 and 1.2, between 0.9 and 1.1 , or between 0.95 and 1.05. They may be captured using the same imaging parameters, like cone beam opening angle, beam energy, collimation, or the like.

As mentioned above, the angiographic image may be a DSA image. The mask image used to generate the angiographic image may be used as the bone image.

In this document, the word "live" means that the point in time at which the live image is taken is later than the point in time at which the fluoroscopic image which forms the bone image of from which the bone image is derived is taken. The live fluoroscopic image is for example captured shortly before the data processing of the present invention is performed, preferably only a second or a fraction of a second before. The word "live" thus means that the current situation of the patient can be seen. It can further mean that a sequence of live fluoroscopic images can be captured and the data processing of the present invention is performed on each live fluoroscopic image of the sequence in real time, which for example means with a delay of for example one second or preferably less, such as a fraction of a second.

In other implementations, the bone image might be derived from a raw angiographic image. For example, the parts of the raw angiographic image showing bony structure are extracted to form the bone image. In yet other implementations, the bone image can be a generic image of the bony structure matched to situation of the patient as it is when the angiographic image is captured.

In this document, an image, like a bone image, an angiographic image or a live image, is represented by corresponding image data. The image and the image data are referred to synonymously unless the context implies that only one of the image and the image data is meant.

In a (for example fourth) exemplary step, the angiographic image data and the bone image data are aligned with the live image data. Aligning, or registering, those data means amending the image pair such that it is congruent with the live image. When the images are aligned, areas of the bone image in the image pair and the live fluoroscopic image which show the same part of the patient overlap each other. In other words, the angiographic image and the bone image are transformed to match the live image.

The alignment is performed by aligning the bone image and the live image since both of them show the bony structure of the patient. The alignment thus obtained is also used for the angiographic image since it spatially correlates to the bone image.

The registration can be calculated using known image fusion techniques, for example rigid image fusion. In one embodiment, the image pair and the live fluoroscopic image are shifted relative to each other along the imaging plane and rotated about an axis perpendicular to the imaging plane in order to align the image pair and the live fluoroscopic image.

The result of the registration of the image pair and the live fluoroscopic image can be that no registration has been found, which is for example the case if they do not overlap. In this case, a corresponding information or no data can be provided. It is possible to receive user input indicating a manual registration, for example if an automatic registration fails.

The result of this fourth step is that it is known which parts of the live fluoroscopic image overlap with which parts of the bone image in the image pair. It is thus known which part of the angiographic image of the image pair overlaps with which part of the live fluoroscopic image.

The registration in the fourth step can optionally involve an elastic fusion of the image pairs with the live fluoroscopic image. In this case, the bone image is deformed to match the live fluoroscopic image as good as possible. The same deformation is applied to the corresponding angiographic image.

In a (for example fifth) exemplary step, an overlap image region being an identical spatial region in both the angiographic image data and the bone image data is determined, wherein the overlap image region of the angiographic image data represents a vessel and the overlap image region of the bone image data represents a bone. The overlap image region thus means the region of the images in terms of a contiguous set of pixels, and said region is identical in the angiographic image and the bone image. It is understood that “represents a vessel” does not mean that the overlap image region represents all of the vessel, but represents part of the vessel only. In analogy, “represents a bone” does not mean that the overlap image region represents all of the bone, but represents part of the bone only

From now on, the angiographic image and the bone image are the aligned versions unless stated otherwise.

In other words, the pixels in the overlap image region of the angiographic image show a vascular structure while the same pixels in the overlap image region of the bone image show bony structure. So if the angiographic image and the bone image were overlaid, the overlap image region of the resulting image would cover both vascular and bony structures.

The angiographic image and the bone image typically have the same size in terms of the number of pixels in the horizontal and vertical directions. In this case, the same pixel in two images means the pixel at the same position in the two images. If the two images have different sizes, the same pixels in the two images are pixels at the same relative position as defined by the horizontal position of the pixel divided by the horizontal size of the image and the vertical position of the pixel divided by the vertical size of the image.

Determining the overlap image region may be automatic, for example using thresholds for determining whether or not a pixel of the (grayscale) angiographic image represents a vessel and/or a pixel of the (grayscale) bone image represents a bone. The overlap image region can then be a logical AND-combination of the thresholded images.

Determining the overlap image region may also involve user input data. The user input data might represent drawing data in which the user draws the overlap image region or corrects an automatically determined overlap image region. In a (for example sixth) exemplary step, the bone image data is altered in the overlap image region. Altering may involve any modification of the bone image data in the overlap image region such that the visibility of information in the live image and/or the angiographic image is increased when performing the following step.

The overlap image region is preferably smaller than the bone image. The size of the overlap image region is for example less than 50 percent, 25 percent or 10 percent of the size of bone image. This means that at least a part of the bone image remains visible after the alteration.

Further alterations may be performed outside the overlap image region, for example in a region in which the live image and/or the angiographic image shows an implant, like a stent.

In a (for example seventh) exemplary step, the aligned angiographic image data and the aligned bone image data are overlaid over the fluoroscopic image data. This means that the three images are combined, for example by averaging their pixel values pixel by pixel or by taking the largest pixel value out of the three pixel values of the three images for each pixel. Other suitable overlay methods may be employed, for example one suitable for colored images. The result of overlaying the angiographic image and the bone image over the fluoroscopic image is also referred to as overlay image represented by overlay image data.

The purpose of overlaying the angiographic image over the live image is that information on the vascular system is combined with the information in the live image. The purpose of overlaying the bone image over the live image is giving an easily recognizable impression of whether the alignment of the image pair with the live image is correct.

However, important information might get lost if the angiographic image and the original aligned bone image were overlaid over the live image. This in particular refers to information in the live image which gets obstructed by the overlay. The invention thus overlays the bone image that is altered in the overlap image region instead of the unaltered bone image. The bone image preferably remains unaltered outside of the overlap image region, which means that it is altered in the overlap image region only.

The overlay image may then be output, for example to a storage device that stores the overlay image data and/or a display unit that displays the overlay image.

The steps of acquiring the fluoroscopic image data, aligning the angiographic image data and the bone image data with the fluoroscopic image data, determining the overlap image region, altering the bone image data and overlaying the aligned angiographic image data and the aligned bone image data over the fluoroscopic image data may be repeated for multiple fluoroscopic, or live, images, such that the overlay can be calculated and displayed (almost) in real time.

It shall be noted that the numbering of the steps does not necessarily define the order in which the steps are performed. For example, the order in which the image data are acquired may be different or determining the overlap image region may be performed before aligning the angiographic image data and the bone image data with the fluoroscopic image data.

It shall be emphasized that the angiographic image and the bone image being taken “from the same viewing position” is not limited to exactly the same viewing position, but also allows a small offset as long as the resulting offset between the angiographic image and the bone image does not lead to a clinically relevant degradation of the overlay image. Depending on the application, an offset between the angiographic image and the bone image of 1 pixel or up to 2, 5 or 10 pixels may still mean that the images are taken from the same viewing position. The only effect of the offset is that the angiographic image is not absolutely exactly aligned with the live image, but this has no influence on the step of altering the bone image data in the overlap image region.

As mentioned above, the angiographic image can be a generic image. A generic angiographic image is for example a DRR generated from a 3D image dataset representing vascular information, for example a computed tomography angiography (CTA) or a magnetic resonance angiography (MRA).

In one embodiment, both the bone image and the angiographic image are generic, for example generated from the same 3D image dataset which shows both the bony and the vascular structure. This 3D image dataset is for example segmented, which means that each voxel has assigned information which kind of structure it shows. So each voxel can for example be labeled as “bone”, “vessel” or “other”. The advantage of generating both the bone image and the angiographic image from the same 3D dataset is that they are perfectly aligned with each other and generated from the same “virtual” viewing position onto the 3D image dataset.

In one embodiment, the angiographic image is amended based on information on the vascular system in a 3D image dataset representing vascular information. In one implementation, a DRR which matches the angiographic image is generated from the 3D image dataset. Missing parts of the vascular system in the angiographic image can then be supplemented from the DRR or noise can be removed from the angiographic image, for example by removing or attenuating pixels in which the angiographic image shows a vessel, but the DRR does not.

In one embodiment, the bone image is amended based on a 3D image dataset in analogy.

In one embodiment, image processing is applied to the angiographic image, in particular before the step of determining the overlap image region is performed.

One example of image processing is inverting the angiographic image. “Inverting” for example means reversing the grayscale of the angiographic image, thus making light pixels dark and vice versa. While dark pixels typically show the vascular system on light background, this is inverted such that image information comprised in the fluoroscopic image is not obstructed when the angiographic image is overlaid to form the overlay image. Another example of image processing is broadening the vessel or vessels shown in the angiographic image. This is also referred to as image dilation or image erosion, depending on whether the vascular system is bright or dark in the angiographic image. This has the technical effect that the overlap image region becomes potentially larger. This means that the visibility of a larger part of the live image and/or the angiographic image in the overlay image is achieved. This example of image processing can for example reduce the effect of an offset between the angiographic image and the bone image since the broadened version of a vessel which is offset relative to its correct position (relative to the bone image) may also cover the vessel at its correct position.

In one embodiment, altering the bone image data in the overlap image region is attenuating the bone image data in the overlap image region. Attenuating for example means reducing the pixel values of the bone image in the overlap image region by a factor, wherein the factor is for example selectable by a user. The result of the attenuation is that the bone image becomes partly or fully transparent in the overlap image region.

In one example, attenuating the bone image in the overlap image region is by 100 percent, which means that the bone image becomes fully transparent within the overlap image region. If the overlap image region is understood as a binary mask, the bone image may be multiplied with the overlap image region.

Other options for altering the bone image data in the overlap image region may for example involve one or more of a modification of the color scheme of the bone image, amending a solid bone outline into a broken bone outline or replacing fluoroscopic image information by a bone outline. In the latter case, the altered bone image is a fluoroscopic image outside of the overlap image region and a bone outline within the overlap image region.

It shall be noted that the alteration of the bone image in the overlap image region does not necessarily modify the bone image data themselves, but can be performed while overlaying the bone image over the live image. In one embodiment, the method further comprises the step of extending the overlap image region before performing the step of altering the bone image data. Originally, the overlap image region only encompasses exactly those pixels which show a vascular structure in the angiographic image and bony structure in the bone image. The overlap image region is thus typically irregular in shape and might be rather small. In the present embodiment, the overlap image region is enlarged while still encompassing the original overlap image region. The shape of the extended overlap image region might be regular, like circular, rectangular, polygonal, oval or elliptical.

In one embodiment, the method further comprises the step of adding a halo to the overlap image region, either with or without extending the overlay image region beforehand. The halo is a shine towards the outside of the overlap image region in which the intensity of the alteration of the bone image data gradually decreases. In this case, the overlap image region may not be a binary mask, but fades out over some pixels to the outside, starting from the original border of the overlap image region.

In one embodiment, the method further comprises the step of determining a location of a region of interest of a medical device in at least one of the fluoroscopic image data, that is the live image, the bone image data and the angiographic image data, wherein the medical device is at least partly located in the vascular system of the patient, setting a device image region relative to the location of the region of interest in the bone image data and combining the device image region with the overlap image region before performing the step of altering the bone image data. Altering the bone image is performed in the combined region including the device image region and the overlap image region.

The device image region may only comprise those pixels which are determined to show the medical device. However, the device image region may also be extended. The shape of the device image region after extension might be regular, like circular, rectangular, polygonal, oval or elliptical.

Setting the device image region relative to the location of the region of interest may mean that the region of interest lies within the device image region. The device image region may be centered about the location of the region of interest. However, the region of interest may be off-center to the device image region. Compared to the center of the device image region, the region of interest may be located in a direction opposite to a movement direction of the region of interest. In this case, the larger part of the device image region covers a region into which the region of interest is supposed to move.

This embodiment is particularly advantageous if a medical instrument, or a part thereof, is located in the vascular system and shall be clearly visible in the overlay image. The region of interest includes a part of the medical device or the complete medical device.

The medical device can for example be a stent. In this case, the region of the interest is for example the whole stent. The region of interest can be determined in the bone image data if the stent was already placed when the bone image was captured. As an alternative or in addition, the region of interest can be determined in the fluoroscopic image data such that it represents the current position of the stent, for example if the stent is in the process of being placed.

As another example, the medical device can be an endovascular object, for example an endovascular implant like a stent or a device like a guidewire. The endovascular tool is inserted from the outside into the vascular system of the patient. The region of interest of an endovascular tool is for example the distal end of the endovascular tool. This may include the end only or a distal end region of the endovascular tool, like the most distal centimeter, two centimeters or five centimeters. In this example, the region of interest of the endovascular tool is preferably determined in the fluoroscopic image data, which means in the live image which represents the latest situation.

The method may determine regions of interest of multiple medical devices and/or multiple regions of interest of a single medical device. There are thus multiple regions of interest and the method sets corresponding multiple device image regions. The multiple device image regions are then combined with the overlap image region.

In one implementation, the location of the region of interest is determined by image analysis of the fluoroscopic image, the bone image or the angiographic image, as applicable. Image analysis for example uses known images of the medical device to find and localize the medical device, or a part thereof, in the fluoroscopic image or the bone image, respectively. The image analysis may be performed on the live image and one or more previous live images to track motion of the medical device. The tracked motion can be used to set the device image region, for example by locating it relative to the region of interest such that the device image region is centered about a location that is supposed to lie on the predicted future path of the region of interest of the medical instrument.

In one implementation, the location of the region of interest is determined by tracking the medical device. In one example, the medical device is a stent which is applied using an application tool and the application tool is tracked, for example using optical or electromagnetic tracking using markers on the application tool. In another example, the medical device is an endovascular tool and the distal tip of the endovascular tool is tracked by measuring the penetration depth of the endovascular tool and the path taken by the endovascular tool within the vascular system of the patient. The penetration depth means the length of the part of the endovascular tool which is within the body of the patient.

In one implementation, the method is applied to multiple subsequent live images. In this case, the region of interest of the medical device is preferably determined in the fluoroscopic image data and combined with the overlap image region independently for each live image. The device image region thus moves together with the region of interest of the medical device.

However, in one implementation, the method comprises the step of freezing the device image region upon reception of user input data. The frozen device image region is combined with the overlap image region even if the region of interest of the medical device has moved in subsequent live images. In one example, also a new device image region is set in the bone image data relative to the location of the region of interest of the medical device in the current live image. The method then involves combining the frozen device image region and the new device image region with the overlap image region before performing the step of altering the bone image data.

Combining the device image region and the overlap image region may involve determining an intersection of those regions. Alternatively, it may involve adding those regions, such that the combination covers all pixels comprised in at least one of the device image region and the overlap image region.

In one embodiment, the method further comprises the step of adding enlargement image data representing an enlargement of an enlargement area of the fluoroscopic image and of the aligned angiographic image to the overlay of the fluoroscopic image data, the angiographic image data and the bone image data, wherein the enlargement area is an area defined relative to a region of interest of a medical device located at least partly in the vascular system of the patient.

In other words, the overlay image data is supplemented by the enlargement image data such that the content in the enlargement area is zoomed in or magnified. The enlargement image data is for example overlaid over the overlay image data.

The position of the enlargement image data relative to the overlay image data can be such that it covers all or part of the enlargement area. However, it can be such that it does not overlap with the enlargement area, such that both the enlargement area and its magnification are visible.

The enlargement image data is preferably positioned such that it does not overlap any part of the overlap image region. The enlargement image data can be positioned such that it is at least partly outside of the overlay image data. This means that the size, in terms of the horizontal and/or vertical number of pixels, of the combination of the overlay image data and the enlargement image data is larger than the size of the overlay image data only.

As above, the medical device can for example be a stent or an endovascular tool. The region of interest can be the complete region covered by the stent or a region about the distal end of the endovascular tool. The term “relative to” means the same as explained above with regards to the device image region.

In one implementation, the enlargement image data is added only if a speed of movement of the medical device in the vascular system is below a threshold. This means that the enlargement area is only enlarged if movement of the medical device is moved slowly. While the medical device is typically moved rather quickly if it is still further away from its desired location, it is moved slower when it approaches the desired location, where it might be helpful to enlarge the enlargement area. The threshold is for example 0.5, 1 , 2, 5, 10, 50 100 or 200 millimeters per second.

In one embodiment, the method further comprises the steps of determining a subtraction image region in the fluoroscopic image data, the subtraction image region being an image region defined relative to a region of interest of a medical device at least partly located in the vascular system of the patient, and of subtracting prior fluoroscopic image data from the fluoroscopic image in the subtraction image region, wherein the prior fluoroscopic image data represents a fluoroscopic image of the patient taken at a point in time before the fluoroscopic image is taken and which is aligned with the fluoroscopic image data.

In this embodiment, bony structures shown in the live fluoroscopic image are removed therefrom to increase visibility of vascular structures in the overlay image. The subtraction image region may be identical to the overlap image region, but may also differ from the overlap image region. It may for example be smaller or larger than the overlap image region. In one implementation, the subtraction image region overlaps with a part or all of the overlap image region.

The prior fluoroscopic image may be a fluoroscopic bone image which was taken in combination with a corresponding angiographic image. However, it might also be any other suitable fluoroscopic image, such as a dedicated image captured at the start of an intervention or a previous live fluoroscopic image.

In addition or as an alternative, the subtraction image region may be a region defined relative to the overlap image region rather than relative to a region of interest of a medical device. It may for example be identical to the overlap image region. The method may acquire two or more image pairs, each image pair comprising a bone image and an angiographic image. In this case, the image pair whose bone image matches best with the live fluoroscopic image is selected. The angiographic image and the bone image of the selected image pair are aligned with the live image.

The image pairs might show different parts of the patient, wherein two image pairs might show overlapping parts of the patient. In this case, a larger area of the patient can be imaged than with just a single image pair. The live image typically shows the part of the patient which is of interest at the time being, and the best image pair is selected for aligning the corresponding angiographic image with the live image. The subsequent processing is performed with the selected image pair as described above.

The method may determine multiple overlap image regions. The bone image data is then altered in each of the multiple overlap image regions before it is overlaid over the live image. The same alteration may be applied to each of the overlay image regions, or different alterations may be applied. Each one of the multiple overlap image regions may be extended individually as described above. The multiple overlap image regions may be combined with one or more device image regions.

In a second aspect, the invention is directed to a computer program comprising instructions which, when the program is executed by at least one computer, causes the at least one computer to carry out method according to the first aspect. The invention may alternatively or additionally relate to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, such as an electromagnetic carrier wave carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the steps of the method according to the first aspect. The signal wave is in one example a data carrier signal carrying the aforementioned computer program. A computer program stored on a disc is a data file, and when the file is read out and transmitted it becomes a data stream for example in the form of a (physical, for example electrical, for example technically generated) signal. The signal can be implemented as the signal wave, for example as the electromagnetic carrier wave which is described herein. For example, the signal, for example the signal wave is constituted to be transmitted via a computer network, for example LAN, WLAN, WAN, mobile network, for example the internet. For example, the signal, for example the signal wave, is constituted to be transmitted by optic or acoustic data transmission. The invention according to the second aspect therefore may alternatively or additionally relate to a data stream representative of the aforementioned program, i.e. comprising the program.

In a third aspect, the invention is directed to a computer-readable storage medium on which the program according to the second aspect is stored. The program storage medium is for example non-transitory.

In a fourth aspect, the invention is directed to at least one computer (for example, a computer), comprising at least one processor (for example, a processor), wherein the program according to the second aspect is executed by the processor, or wherein the at least one computer comprises the computer-readable storage medium according to the third aspect.

In a fifth aspect, the invention is directed to a medical system, comprising: a) the at least one computer according to the fourth aspect; b) at least one electronic data storage device storing at least the angiographic image data and the bone image data; and c) a medical imaging device configured to capture a live fluoroscopic image.

Alternatively or additionally, the invention is directed to a for example non-transitory computer-readable program storage medium storing a program for causing the computer according to the fourth aspect to execute the data processing steps of the method according to the first aspect.

For example, the disclosed method is not a method for treatment of the human or animal body by surgery or therapy. For example, the invention does not involve or in particular comprise or encompass an invasive step which would represent a substantial physical interference with the body requiring professional medical expertise to be carried out and entailing a substantial health risk even when carried out with the required professional care and expertise. The method is rather a data processing method performed by a computer, which is a method to operate a computer to perform the steps of the method described above.

For example, the invention does not comprise a step of administering a contrast agent or inserting a medical device into the body of the patient. More particularly, the invention does not involve or in particular comprise or encompass any surgical or therapeutic activity. The invention is instead directed as applicable to image processing in order to combine a live image with an angiographic image and a bone image. For this reason alone, no surgical or therapeutic activity and in particular no surgical or therapeutic step is necessitated or implied by carrying out the invention.

DEFINITIONS

In this section, definitions for specific terminology used in this disclosure are offered which also form part of the present disclosure.

Computer-implemented method

The method in accordance with the invention is for example a computer-implemented method. For example, all the steps or merely some of the steps (i.e. less than the total number of steps) of the method in accordance with the invention can be executed by a computer (for example, at least one computer). An embodiment of the computer implemented method is a use of the computer for performing a data processing method. An embodiment of the computer implemented method is a method concerning the operation of the computer such that the computer is operated to perform one, more or all steps of the method.

The computer for example comprises at least one processor and for example at least one memory in order to (technically) process the data, for example electronically and/or optically. The processor being for example made of a substance or composition which is a semiconductor, for example at least partly n- and/or p-doped semiconductor, for example at least one of II-, III-, IV-, V-, Vl-sem iconductor material, for example (doped) silicon and/or gallium arsenide. The calculating or determining steps described are for example performed by a computer. Determining steps or calculating steps are for example steps of determining data within the framework of the technical method, for example within the framework of a program. A computer is for example any kind of data processing device, for example electronic data processing device. A computer can be a device which is generally thought of as such, for example desktop PCs, notebooks, netbooks, etc., but can also be any programmable apparatus, such as for example a mobile phone or an embedded processor. A computer can for example comprise a system (network) of "sub-computers", wherein each sub-computer represents a computer in its own right. The term "computer" includes a cloud computer, for example a cloud server. The term computer includes a server resource. The term "cloud computer" includes a cloud computer system which for example comprises a system of at least one cloud computer and for example a plurality of operatively interconnected cloud computers such as a server farm. Such a cloud computer is preferably connected to a wide area network such as the world wide web (WWW) and located in a so-called cloud of computers which are all connected to the world wide web. Such an infrastructure is used for "cloud computing", which describes computation, software, data access and storage services which do not require the end user to know the physical location and/or configuration of the computer delivering a specific service. For example, the term "cloud" is used in this respect as a metaphor for the Internet (world wide web). For example, the cloud provides computing infrastructure as a service (laaS). The cloud computer can function as a virtual host for an operating system and/or data processing application which is used to execute the method of the invention. The cloud computer is for example an elastic compute cloud (EC2) as provided by Amazon Web Services™. A computer for example comprises interfaces in order to receive or output data and/or perform an analogue-to-digital conversion. The data are for example data which represent physical properties and/or which are generated from technical signals. The technical signals are for example generated by means of (technical) detection devices (such as for example devices for detecting marker devices) and/or (technical) analytical devices (such as for example devices for performing (medical) imaging methods), wherein the technical signals are for example electrical or optical signals. The technical signals for example represent the data received or outputted by the computer. The computer is preferably operatively coupled to a display device which allows information outputted by the computer to be displayed, for example to a user. One example of a display device is a virtual reality device or an augmented reality device (also referred to as virtual reality glasses or augmented reality glasses) which can be used as "goggles" for navigating. A specific example of such augmented reality glasses is Google Glass (a trademark of Google, Inc.). An augmented reality device or a virtual reality device can be used both to input information into the computer by user interaction and to display information outputted by the computer. Another example of a display device would be a standard computer monitor comprising for example a liquid crystal display operatively coupled to the computer for receiving display control data from the computer for generating signals used to display image information content on the display device. A specific embodiment of such a computer monitor is a digital lightbox. An example of such a digital lightbox is Buzz®, a product of Brainlab AG. The monitor may also be the monitor of a portable, for example handheld, device such as a smart phone or personal digital assistant or digital media player.

The invention also relates to a computer program comprising instructions which, when on the program is executed by a computer, cause the computer to carry out the method or methods, for example, the steps of the method or methods, described herein and/or to a computer-readable storage medium (for example, a non-transitory computer- readable storage medium) on which the program is stored and/or to a computer comprising said program storage medium and/or to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, such as an electromagnetic carrier wave carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the method steps described herein. The signal wave is in one example a data carrier signal carrying the aforementioned computer program. The invention also relates to a computer comprising at least one processor and/or the aforementioned computer-readable storage medium and for example a memory, wherein the program is executed by the processor.

Within the framework of the invention, computer program elements can be embodied by hardware and/or software (this includes firmware, resident software, micro-code, etc.). Within the framework of the invention, computer program elements can take the form of a computer program product which can be embodied by a computer-usable, for example computer-readable data storage medium comprising computer-usable, for example computer-readable program instructions, "code" or a "computer program" embodied in said data storage medium for use on or in connection with the instructionexecuting system. Such a system can be a computer; a computer can be a data processing device comprising means for executing the computer program elements and/or the program in accordance with the invention, for example a data processing device comprising a digital processor (central processing unit or CPU) which executes the computer program elements, and optionally a volatile memory (for example a random access memory or RAM) for storing data used for and/or produced by executing the computer program elements. Within the framework of the present invention, a computer-usable, for example computer-readable data storage medium can be any data storage medium which can include, store, communicate, propagate or transport the program for use on or in connection with the instruction -executing system, apparatus or device. The computer-usable, for example computer-readable data storage medium can for example be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or a medium of propagation such as for example the Internet. The computer-usable or computer-readable data storage medium could even for example be paper or another suitable medium onto which the program is printed, since the program could be electronically captured, for example by optically scanning the paper or other suitable medium, and then compiled, interpreted or otherwise processed in a suitable manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product and any software and/or hardware described here form the various means for performing the functions of the invention in the example embodiments. The computer and/or data processing device can for example include a guidance information device which includes means for outputting guidance information. The guidance information can be outputted, for example to a user, visually by a visual indicating means (for example, a monitor and/or a lamp) and/or acoustically by an acoustic indicating means (for example, a loudspeaker and/or a digital speech output device) and/or tactilely by a tactile indicating means (for example, a vibrating element or a vibration element incorporated into an instrument). For the purpose of this document, a computer is a technical computer which for example comprises technical, for example tangible components, for example mechanical and/or electronic components. Any device mentioned as such in this document is a technical and for example tangible device. Acquiring data

The expression "acquiring data" for example encompasses (within the framework of a computer implemented method) the scenario in which the data are determined by the computer implemented method or program. Determining data for example encompasses measuring physical quantities and transforming the measured values into data, for example digital data, and/or computing (and e.g. outputting) the data by means of a computer and for example within the framework of the method in accordance with the invention. A step of “determining” as described herein for example comprises or consists of issuing a command to perform the determination described herein. For example, the step comprises or consists of issuing a command to cause a computer, for example a remote computer, for example a remote server, for example in the cloud, to perform the determination. Alternatively or additionally, a step of “determination” as described herein for example comprises or consists of receiving the data resulting from the determination described herein, for example receiving the resulting data from the remote computer, for example from that remote computer which has been caused to perform the determination. The meaning of "acquiring data" also for example encompasses the scenario in which the data are received or retrieved by (e.g. input to) the computer implemented method or program, for example from another program, a previous method step or a data storage medium, for example for further processing by the computer implemented method or program. Generation of the data to be acquired may but need not be part of the method in accordance with the invention. The expression "acquiring data" can therefore also for example mean waiting to receive data and/or receiving the data. The received data can for example be inputted via an interface. The expression "acquiring data" can also mean that the computer implemented method or program performs steps in order to (actively) receive or retrieve the data from a data source, for instance a data storage medium (such as for example a ROM, RAM, database, hard drive, etc.), or via the interface (for instance, from another computer or a network). The data acquired by the disclosed method or device, respectively, may be acquired from a database located in a data storage device which is operably to a computer for data transfer between the database and the computer, for example from the database to the computer. The computer acquires the data for use as an input for steps of determining data. The determined data can be output again to the same or another database to be stored for later use. The database or database used for implementing the disclosed method can be located on network data storage device or a network server (for example, a cloud data storage device or a cloud server) or a local data storage device (such as a mass storage device operably connected to at least one computer executing the disclosed method). The data can be made "ready for use" by performing an additional step before the acquiring step. In accordance with this additional step, the data are generated in order to be acquired. The data are for example detected or captured (for example by an analytical device). Alternatively or additionally, the data are inputted in accordance with the additional step, for instance via interfaces. The data generated can for example be inputted (for instance into the computer). In accordance with the additional step (which precedes the acquiring step), the data can also be provided by performing the additional step of storing the data in a data storage medium (such as for example a ROM, RAM, CD and/or hard drive), such that they are ready for use within the framework of the method or program in accordance with the invention. The step of "acquiring data" can therefore also involve commanding a device to obtain and/or provide the data to be acquired. In particular, the acquiring step does not involve an invasive step which would represent a substantial physical interference with the body, requiring professional medical expertise to be carried out and entailing a substantial health risk even when carried out with the required professional care and expertise. In particular, the step of acquiring data, for example determining data, does not involve a surgical step and in particular does not involve a step of treating a human or animal body using surgery or therapy. In order to distinguish the different data used by the present method, the data are denoted (i.e. referred to) as "XY data" and the like and are defined in terms of the information which they describe, which is then preferably referred to as "XY information" and the like.

Image registration

Image registration is the process of transforming different sets of data into one coordinate system. The data can be multiple photographs and/or data from different sensors, different times or different viewpoints. It is used in computer vision, medical imaging and in compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from these different measurements.

Marker

It is the function of a marker to be detected by a marker detection device (for example, a camera or an ultrasound receiver or analytical devices such as CT or MRI devices) in such a way that its spatial position (i.e. its spatial location and/or alignment) can be ascertained. The detection device is for example part of a navigation system. The markers can be active markers. An active marker can for example emit electromagnetic radiation and/or waves which can be in the infrared, visible and/or ultraviolet spectral range. A marker can also however be passive, i.e. can for example reflect electromagnetic radiation in the infrared, visible and/or ultraviolet spectral range or can block X-ray radiation. To this end, the marker can be provided with a surface which has corresponding reflective properties or can be made of metal in order to block the X-ray radiation. It is also possible for a marker to reflect and/or emit electromagnetic radiation and/or waves in the radio frequency range or at ultrasound wavelengths. A marker preferably has a spherical and/or spheroid shape and can therefore be referred to as a marker sphere; markers can however also exhibit a cornered, for example cubic, shape.

Imaging methods

In the field of medicine, imaging methods (also called imaging modalities and/or medical imaging modalities) are used to generate image data (for example, two- dimensional or three-dimensional image data) of anatomical structures (such as soft tissues, bones, organs, etc.) of the human body. The term "medical imaging methods" is understood to mean (advantageously apparatus-based) imaging methods (for example so-called medical imaging modalities and/or radiological imaging methods) such as for instance computed tomography (CT) and cone beam computed tomography (CBCT, such as volumetric CBCT), X-ray tomography, magnetic resonance tomography (MRT or MRI), conventional X-ray, sonography and/or ultrasound examinations, and positron emission tomography. For example, the medical imaging methods are performed by the analytical devices. Examples for medical imaging modalities applied by medical imaging methods are: X-ray radiography, magnetic resonance imaging, medical ultrasonography or ultrasound, endoscopy, elastography, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques as positron emission tomography (PET) and Single-photon emission computed tomography (SPECT), as mentioned by Wikipedia. The image data thus generated is also termed “medical imaging data”. Analytical devices for example are used to generate the image data in apparatusbased imaging methods. The imaging methods are for example used for medical diagnostics, to analyze the anatomical body in order to generate images which are described by the image data. The imaging methods are also for example used to detect pathological changes in the human body. However, some of the changes in the anatomical structure, such as the pathological changes in the structures (tissue), may not be detectable and for example may not be visible in the images generated by the imaging methods. A tumor represents an example of a change in an anatomical structure. If the tumor grows, it may then be said to represent an expanded anatomical structure. This expanded anatomical structure may not be detectable; for example, only a part of the expanded anatomical structure may be detectable. Primary/high- grade brain tumors are for example usually visible on MRI scans when contrast agents are used to infiltrate the tumor. MRI scans represent an example of an imaging method. In the case of MRI scans of such brain tumors, the signal enhancement in the MRI images (due to the contrast agents infiltrating the tumor) is considered to represent the solid tumor mass. Thus, the tumor is detectable and for example discernible in the image generated by the imaging method. In addition to these tumors, referred to as "enhancing" tumors, it is thought that approximately 10% of brain tumors are not discernible on a scan and are for example not visible to a user looking at the images generated by the imaging method.

Mapping

Mapping describes a transformation (for example, linear transformation) of an element (for example, a pixel or voxel), for example the position of an element, of a first data set in a first coordinate system to an element (for example, a pixel or voxel), for example the position of an element, of a second data set in a second coordinate system (which may have a basis which is different from the basis of the first coordinate system). In one embodiment, the mapping is determined by comparing (for example, matching) the color values (for example grey values) of the respective elements by means of an elastic or rigid fusion algorithm. The mapping is embodied for example by a transformation matrix (such as a matrix defining an affine transformation).

Elastic fusion, image fusion/morphing, rigid

Image fusion can be elastic image fusion or rigid image fusion. In the case of rigid image fusion, the relative position between the pixels of a 2D image and/or voxels of a 3D image is fixed, while in the case of elastic image fusion, the relative positions are allowed to change.

In this application, the term "image morphing" is also used as an alternative to the term "elastic image fusion", but with the same meaning.

Elastic fusion transformations (for example, elastic image fusion transformations) are for example designed to enable a seamless transition from one dataset (for example a first dataset such as for example a first image) to another dataset (for example a second dataset such as for example a second image). The transformation is for example designed such that one of the first and second datasets (images) is deformed, for example in such a way that corresponding structures (for example, corresponding image elements) are arranged at the same position as in the other of the first and second images. The deformed (transformed) image which is transformed from one of the first and second images is for example as similar as possible to the other of the first and second images. Preferably, (numerical) optimization algorithms are applied in order to find the transformation which results in an optimum degree of similarity. The degree of similarity is preferably measured by way of a measure of similarity (also referred to in the following as a "similarity measure"). The parameters of the optimization algorithm are for example vectors of a deformation field. These vectors are determined by the optimization algorithm in such a way as to result in an optimum degree of similarity. Thus, the optimum degree of similarity represents a condition, for example a constraint, for the optimization algorithm. The bases of the vectors lie for example at voxel positions of one of the first and second images which is to be transformed, and the tips of the vectors lie at the corresponding voxel positions in the transformed image. A plurality of these vectors is preferably provided, for instance more than twenty or a hundred or a thousand or ten thousand, etc. Preferably, there are (other) constraints on the transformation (deformation), for example in order to avoid pathological deformations (for instance, all the voxels being shifted to the same position by the transformation). These constraints include for example the constraint that the transformation is regular, which for example means that a Jacobian determinant calculated from a matrix of the deformation field (for example, the vector field) is larger than zero, and also the constraint that the transformed (deformed) image is not self-intersecting and for example that the transformed (deformed) image does not comprise faults and/or ruptures. The constraints include for example the constraint that if a regular grid is transformed simultaneously with the image and in a corresponding manner, the grid is not allowed to interfold at any of its locations. The optimizing problem is for example solved iteratively, for example by means of an optimization algorithm which is for example a first-order optimization algorithm, such as a gradient descent algorithm. Other examples of optimization algorithms include optimization algorithms which do not use derivations, such as the downhill simplex algorithm, or algorithms which use higher-order derivatives such as Newton-like algorithms. The optimization algorithm preferably performs a local optimization. If there is a plurality of local optima, global algorithms such as simulated annealing or generic algorithms can be used. In the case of linear optimization problems, the simplex method can for instance be used.

In the steps of the optimization algorithms, the voxels are for example shifted by a magnitude in a direction such that the degree of similarity is increased. This magnitude is preferably less than a predefined limit, for instance less than one tenth or one hundredth or one thousandth of the diameter of the image, and for example about equal to or less than the distance between neighboring voxels. Large deformations can be implemented, for example due to a high number of (iteration) steps.

The determined elastic fusion transformation can for example be used to determine a degree of similarity (or similarity measure, see above) between the first and second datasets (first and second images). To this end, the deviation between the elastic fusion transformation and an identity transformation is determined. The degree of deviation can for instance be calculated by determining the difference between the determinant of the elastic fusion transformation and the identity transformation. The higher the deviation, the lower the similarity, hence the degree of deviation can be used to determine a measure of similarity.

A measure of similarity can for example be determined on the basis of a determined correlation between the first and second datasets.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, the invention is described with reference to the appended figures which give background explanations and represent specific embodiments of the invention. The scope of the invention is however not limited to the specific features disclosed in the context of the figures, wherein

Fig. 1 illustrates a flow diagram of an exemplary method according to the invention;

Fig. 2 shows a bone image;

Fig. 3 shows an angiographic image;

Fig. 4 shows the combination of the bone image and the angiographic image;

Fig. 5 shows a live image;

Fig. 6 shows an overlay of the bone image and the angiographic image over the live image;

Fig. 7 shows an overlap image region in the bone image;

Fig. 8 shows an extended overlap image region;

Fig. 9 shows an overlay of an attenuated bone image and the angiographic image over the live image;

Fig. 10 shows an overlay of a modified bone image and the angiographic image over the live image;

Fig. 11 shows a second live image;

Fig. 12 shows a device image region in the bone image;

Fig. 13 shows the combination of the overlay image region and the device image region; Fig. 14 shows an overlay of a modified bone image and the angiographic image over the second live image;

Fig. 15 shows the overlay image with a subtracted bone region;

Fig. 16 shows the overlay image with an enlarged device image region; and

Fig. 17 shows a system for implementing the invention;

DESCRIPTION OF EMBODIMENTS

Figure 1 illustrates the basic steps of the method according to the first aspect, in which step S01 involves acquiring bone image data representing a bone image and angiographic image data representing an angiographic image. The bone image shows parts of the bone structure of a patient and the angiographic image shows parts of the vascular system of the patient. The bone image and the angiographic image are taken from the same viewing position onto the patient and have the same size in terms of the number of vertical and horizontal pixels. In the present example, both the bone image and the angiographic image were captured using a medical x-ray imaging system, wherein the angiographic image was taken under administration of a contrast agent.

Figure 2 shows the bone image which shows parts of a bone B and Figure 3 shows the angiographic image which shows parts of the vessel V, which is a vascular structure being part of the vascular system. Figure 4 shows a combination of the bone image and the angiographic image. This combination shows both the bone B and the vessel V.

Step S02 involves acquiring fluoroscopic image data representing a fluoroscopic image, which is also referred to as live image. The live image is shown in figure 5 and shows the same bone B as the bone image.

Step S03 involves aligning the bone image and the angiographic image with the live image. In particular, this step calculates a transformation which matches the bone image with the live image. Since the bone image and the angiographic image were taken from the same viewing position relative to the patient, the same transformation also matches the angiographic image to the live image. From now on, the bone image means the aligned bone image and the angiographic image means the aligned angiographic image.

Figure 6 shows an overlay of the bone image and the angiographic image over the live image. As can be seen, in particular the bone image occludes parts of the live image such that information in the live image might not be recognizable. However, the overlay allows to recognize whether or not the alignment in step S03 was acceptable. In the example shown in figure 6, the alignment is perfect because the aligned bone image completely covers the live image.

Optional step S04 involves processing one or both of the angiographic image and the bone image. Processing might include one or more of inverting the angiographic image, dilation/erosion of the angiographic image and amending one or both of the angiographic image and the bone image.

Step S05 involves determining an overlap image region OIR. This region means those pixels which show the bone B in the bone image and the vessel V in the angiographic image. In other words, the overlap image region is the region in the combination of the bone image and the angiographic image in which the bone B and the vessel V overlap. In figure 7, the overlap image region OIR is shown as a hatched area.

Step S06 involves extending the overlap image region. The initial overlap image region OIR only comprised the pixels common to the bone B and the vessel V. Step S06 extends this region. In the example shown in figure 8, the extended overlap image region eOlR has a circular shape and covers all of the initial overlap image region OIR. Step S06 is optional, but preferred.

Each of steps S07 to S09 in the workflow of figure 1 is optional and is described later after the description of the basic embodiment.

Step S10 involves altering the bone image data in the overlap image region or the extended overlap image region as applicable. In the present example shown in figure 9, the bone image data is attenuated by 100 percent in the (extended) overlap image region, such that the live image is not occluded by the bone image in this region.

Figure 10 shows an alternative to the alteration as shown in figure 9. In the example shown in figure 10, the bone image is attenuated by 100 percent in the (extended) overlap image region and a bone outline BA is added to the bone image. The bone outline in the present example is a dashed line indicating the outline of the bone B within the (extended) overlap image region of the bone image. The contour of the bone B is thus visible in the altered bone image while concealing only a small part of the live image.

Step S11 involves overlaying the altered bone image and the angiographic image over the live image, resulting in an overlay image. This step combines the live image, the altered bone image and the angiographic image. In one example, each of those three images is placed in a layer of the overlay image and the transparency of each layer is adjusted, for example based on user input.

Altering the bone image data in step S10 does not necessarily modify the actual bone image data, but may be done on the fly when overlaying the bone image data over the live image in step S11 .

In Step S12, the overlay image is displayed, for example on a display unit.

Step S13 involves determining whether or not the process is to be repeated. This determination is for example based on user input. If the process is not to be repeated, the method ends at step S14. If the process is to be repeated, the method branches to step S02 at which a new live image is acquired.

Optional step S07 involves determining a region of interest of a medical device, for example in the live image. The medical instrument is at least partly located in the vascular system of the patient. Figure 11 shows an exemplary live image showing both the bone B and the medical instrument M, which is an endovascular tool in the example shown. Step S07 involves identifying the medical instrument M in the live image, for example using image processing, and determining a region of interest of the medical instrument. In the shown case of an endovascular tool, the region of interest is for example the distal tip of the tool.

Step S07 further involves setting a device image region relative to the region of interest of the medical instrument M. Figure 12 shows an example in which a circular device image region DIR is set, wherein the region of interest of the medical instrument lies within the device image region DIR. The device image region DIR is set in the bone image and combined with the (extended) overlap image region OIR/eOIR as shown in figure 13.

The alteration of the bone image of step S10 is then performed in the combination of the (extended) overlap image region and the device image region. In the example shown in figure 14, the bone image is attenuated by 100 percent in the combination of the (extended) overlap image region and the device image region. However, in an alternative, the bone image may be altered differently in the (extended) overlap image region and in the device image region in step S10. In the overlay image shown in figure 14, the medical instrument M is clearly visible in the vessel V.

Optional step S08 involves subtracting a prior fluoroscopic image from a part of the live image. This step includes setting a subtraction image region in the live image relative to the region of interest of the medical instrument M. The subtraction image region is for example identical to the device image region DIR. Step S08 further includes subtraction of the prior fluoroscopic image from the live image in the subtraction image region. This suppresses image information representing the bone in the subtraction image region of the live image, thus increasing visibility of the angiographic information in the overlay image.

The prior fluoroscopic image is aligned with the live image before subtracting if necessary. The prior fluoroscopic image is a fluoroscopic image which is older than the live image, and can be a previous live image or the bone image.

In the example shown in figure 15, the bone image is attenuated in the extended overlap image region and in the device image region. The prior fluoroscopic image is subtracted from the live image in the subtraction image region. There is thus a region in the overlay image in which no bone is visible even though the image information is present in the live image and the unaltered bone image. Subtraction of the prior bone image from the live image can be performed on the actual live image or when the overlay image is generated, which maintains the live image as it is.

Optional step S09 involves adding an enlargement of the device image region DIR to the overlay image. An example of the enlargement is shown in figure 16. This enlargement shows a zoomed-in version of the device image region of the overlay image. This might be combined with a previous subtraction of the prior fluoroscopic image from the live image in the subtraction image region.

Figure 17 is a schematic illustration of the medical system 1 according to the fifth aspect. The system is in its entirety identified by reference sign 1 and comprises a computer 2, an input device 6, a display device 7 and an imaging device 8. The input device is for example a keyboard, a mouse, a touch sensitive surface or any other means for receiving input by a user. The display device is for example a monitor, a projector or a display unit in augmented or virtual reality goggles. The imaging device 8 is for example an x-ray imaging device.

The computer 2 comprises a central processing unit 3, an interface 4 and a memory 5. The central processing unit 3 performs the steps of the data processing method. The memory 5 stores instructions that let the central processing unit 3 perform the data processing method as well as data to be processed, such as image data. The interface 4 is adapted to connect the computer 2 to external devices, like the input device 6, the display device 7 and the imaging device 8.

Claims

1. A computer-implemented data processing method for overlaying angiographic information over a fluoroscopic image, comprising the steps of:

- acquiring fluoroscopic image data representing the fluoroscopic image;

- acquiring angiographic image data representing an angiographic image;

- acquiring bone image data representing a bone image, wherein the bone image and the angiographic image show the same area of a patient and are taken from the same viewing position relative to the patient and wherein the fluoroscopic image shows at least a part of the area of the patient which is shown by the bone image and the angiographic image;

- aligning the angiographic image data and the bone image data with the fluoroscopic image data;

- determining an overlap image region (OIR) being an identical spatial region in both the aligned angiographic image data and the aligned bone image data, wherein the overlap image region (OIR) of the aligned angiographic image data represents a vessel and the overlap image region (OIR) of the aligned bone image data represents a bone;

- altering the bone image data in the overlap image region (OIR); and

- overlaying the aligned angiographic image data and the aligned bone image data over the fluoroscopic image data.

2. The method of claim 1 , wherein altering the bone image data in the overlap image region (OIR) is attenuating the bone image data in the overlap image region.

3. The method of claim 2, wherein attenuating the bone image data in the overlap image region (OIR) is by 100 percent.

4. The method of any one of claims 1 or 3, further comprising the step of extending the overlap image region (OIR) before performing the step of altering the bone image data.

5. The method of any one of claims 1 to 4, further comprising the step of determining a location of a region of interest of a medical device (M) in at least one of the fluoroscopic image data, the bone image data and the angiographic image data, wherein the medical device (M) is at least partly located in the vascular system (V) of the patient, setting a device image region (DIR) relative to the location of the region of interest in the bone image data and combining the device image region (DIR) with the overlap image region (OIR) before performing the step of altering the bone image data.

6. The method of claim 5, wherein the location of the region of interest is determined by image analysis of the respective image.

7. The method of claim 5 or 6, wherein the location of the region of interest is determined by tracking the medical device (M).

8. The method of any one of claims 5 to 7, wherein the medical device (M) is an endovascular object.

9. The method of any one of claims 5 to 8, further comprising the step of freezing the device image region (DIR) upon reception of user input data.

10. The method of any one of claims 1 to 9, further comprising the step of adding enlargement image data representing an enlargement of an enlargement area of the fluoroscopic image and of the aligned angiographic image to the overlay of the fluoroscopic image data, the angiographic image data and the bone image data, wherein the enlargement area is an area defined relative to a region of interest of a medical device (M) located at least partly in the vascular system of the patient.

11 . The method of claim 10, wherein the enlargement image data is added only if a speed of movement of the medical device (M) in the vascular system is below a threshold.

12. The method of any one of claims 1 to 11 , further comprising the steps of determining a subtraction image region in the fluoroscopic image data, the subtraction image region being an image region defined relative to a region of interest of a medical device (M) at least partly located in the vascular system of the patient, and of subtracting prior fluoroscopic image data from the fluoroscopic image in the subtraction image region, wherein the prior fluoroscopic image data represents a fluoroscopic image of the patient which is taken at a point in time before the fluoroscopic image is taken and which is aligned with the fluoroscopic image data.

13. The method of claim 12, wherein the subtraction image region differs from the overlap image region (OIR).

14. The method of any one of claims 1 to 13, wherein the bone image is a generic bone image, like a bone outline.

15. A computer program comprising instructions which, when the program is executed by a computer (2), cause the computer (2) to carry out the method according to any one of the preceding claims; and/or a computer-readable storage medium on which the program is stored; and/or a computer (2) comprising at least one processor (3) and/or the program storage medium, wherein the program is executed by the processor (3); and/or a data carrier signal carrying the program; and/or a data stream comprising the program.