WO2024110335A1 - Providing projection images - Google Patents

Abstract

A computer-implemented method of providing a projection image representing an interventional device (110), is provided. The method includes inputting X-ray projection image data (120a) into a neural network (150). The X-ray projection image data (120a) represents the interventional device (110) from a first perspective (130a) of a projection X-ray imaging system (140) with respect to the interventional device. In response to the inputting, the neural network generates predicted X-ray projection image data (120pb) representing the interventional device (110) from a second perspective (130b) of the projection X-ray imaging system (140) with respect to the interventional device. The neural network (150) is trained to generate the predicted X-ray projection image data (120pb) using training data comprising a plurality of X-ray projection training images (120a' 1..n) representing the interventional device (110) from the first perspective (130a), and for each X-ray projection training image (120a' 1..n), a corresponding ground truth X-ray projection training image (120b' 1..n) representing the interventional device (110) from the second perspective (130b).

Description

PROVIDING PROJECTION IMAGES

TECHNICAL FIELD

The present disclosure relates to providing projection images representing an interventional device. A computer-implemented method, a computer program product, and a system, are disclosed.

BACKGROUND

Many interventional medical procedures are carried out using projection X-ray imaging systems. The two-dimensional “projection” images provided by such imaging systems facilitate the navigation of interventional devices such as guidewires and catheters within the anatomy. However, it is challenging to mentally visualize the three-dimensional shape of interventional devices from projection images. This hampers the navigation of interventional devices.

In order to overcome this limitation, biplane projection X-ray imaging systems have been developed. A biplane projection X-ray imaging system can simultaneously acquire projection images that represent an interventional device from two different, often orthogonal, perspectives. The information that is provided by the different perspectives facilitates a user to mentally visualize its three-dimensional shape. However, such imaging systems have a large footprint. The acquisition of images from a second perspective also increases the amount of X-ray radiation dose that is delivered to a patient.

Alternative approaches to overcoming the challenge of mentally visualizing the three- dimensional shape of interventional devices from projection images are also available. One approach is to acquire multiple projection images of the interventional device from different perspectives using a projection X-ray imaging system. However, this approach suffers from the drawback of needing to reposition the projection X-ray imaging system. It also increases the amount of X-ray radiation dose that is delivered to a patient. Another approach involves tracking the 3D shape of the interventional device. Various tracking systems may be used to track the 3D shape. The shape information augments the projection images generated by a projection X-ray imaging system. However, this approach suffers from increased procedural complexity due to the need to set-up the tracking system, and to register the coordinate system of the tracking system to that of the projection X-ray imaging system.

Consequently, there remains a need for improvements that facilitate an understanding of the three-dimensional shape of interventional devices from X-ray projection images.

A document WO 2022/106377 Al discloses a computer-implemented method of providing a neural network for predicting a three-dimensional shape of an interventional device disposed within a vascular region. The method includes: training a neural network to predict, from received X-ray image data and received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region. The training includes constraining the adjusting of parameters of the neural network such that the three- dimensional shape of the interventional device predicted by the neural network fits within the three-dimensional shape of the vascular region represented by the received volumetric image data. This document also discloses a computer-implemented method of predicting a three-dimensional shape of an interventional device disposed within a vascular region. The method comprises: receiving volumetric image data representing a three- dimensional shape of the vascular region; receiving X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region; and inputting the received X-ray image data and the received volumetric image data into a neural network trained to predict, from the received X-ray image data and the received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region, and in response to the inputting, predicting, from the received X-ray image data and the received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region, using the neural network.

SUMMARY

According to one aspect of the present disclosure, a computer-implemented method of providing a projection image representing an interventional device is provided, the method includes: receiving X-ray projection image data representing the interventional device from a first perspective of a projection X-ray imaging system with respect to the interventional device; inputting the X-ray projection image data into a neural network; and in response to the inputting, generating, using the neural network, predicted X-ray projection image data representing the interventional device from a second perspective of the projection X-ray imaging system with respect to the interventional device, the second perspective being different to the first perspective; and wherein the neural network is trained to generate the predicted X-ray projection image data representing the interventional device from the second perspective, using training data comprising a plurality of X-ray projection training images representing the interventional device from the first perspective, and for each X-ray projection training image, a corresponding ground truth X-ray projection training image representing the interventional device from the second perspective.

In the method, predicted X-ray projection image data is generated from X-ray projection image data representing the interventional device from a first perspective of a projection X-ray imaging system. The predicted X-ray projection image data represents the interventional device from a second perspective of the projection X-ray imaging system. Consequently, the method provides X-ray projection data for the interventional device from a different perspective. The two perspectives facilitate a user to visualize the three-dimensional shape of the interventional device. Moreover, since the predicted X-ray projection image data for the second perspective, is generated without the need to actually acquire X-ray projection image data from this perspective, the method facilitates a user to visualize the three- dimensional shape of the interventional device without increasing the amount of X-ray radiation dose that is delivered to a patient. In contrast to the method disclosed in WO 2022/106377 Al, neither the training of the neural network, nor the predictions made by the neural network, require volumetric image data.

Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples, which is made with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flowchart illustrating an example of a computer-implemented method of providing a projection image representing an interventional device, in accordance with some aspects of the present disclosure.

Fig. 2 is a schematic diagram illustrating an example of a system 200 for providing a projection image representing an interventional device, in accordance with some aspects of the present disclosure.

Fig. 3 is an example of X-ray projection image data 120a representing an interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device, in accordance with some aspects of the present disclosure.

Fig. 4 is an example of a predicted X-ray projection image 120^pb representing an interventional device 110 from a second perspective 130b of a projection X-ray imaging system 140 with respect to the interventional device, in accordance with some aspects of the present disclosure.

Fig. 5 is a first example of a neural network 150 that is trained to generate predicted X- ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, using training data comprising X-ray projection training images 120a’ representing the interventional device 110 from a first perspective 130a, and a corresponding ground truth X-ray projection training images 120b’ representing the interventional device 110 from the second perspective 130b, in accordance with some aspects of the present disclosure.

Fig. 6 is a second example of the training of a neural network 150 to generate predicted X-ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, using training data comprising X-ray projection training images 120a’ representing the interventional device 110 from a first perspective 130a, and a corresponding ground truth X-ray projection training images 120b’ representing the interventional device 110 from the second perspective 130b, in accordance with some aspects of the present disclosure.

Fig. 7 is an example of a neural network 150 that is trained to generate constrained predicted X-ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, and wherein the predicted X-ray projection image data 120^pb is constrained by angiographic image data 170b, E, in accordance with some aspects of the present disclosure. Fig. 8 is a schematic diagram including predicted X-ray projection image data 120^pb, 120^pc respectively representing an interventional device 110 from a second perspective 130b, and from a third perspective 130c, of a projection X-ray imaging system 140, in accordance with some aspects of the present disclosure.

DETAILED DESCRIPTION

Examples of the present disclosure are provided with reference to the following description and Figs. In this description, for the purposes of explanation, numerous specific details of certain examples are set forth. Reference in the specification to “an example”, “an implementation” or similar language means that a feature, structure, or characteristic described in connection with the example is included in at least that one example. It is also to be appreciated that features described in relation to one example may also be used in another example, and that all features are not necessarily duplicated in each example for the sake of brevity. For instance, features described in relation to a computer implemented method, may be implemented in a computer program product, and in a system, in a corresponding manner.

In the following description, reference is made to computer-implemented methods that involve providing a projection image representing an interventional device. In some examples, the interventional device is an intravascular device. For instance, reference is made to examples in which the interventional device represented in the projection image is a guide wire. However, it is to be appreciated that the computer-implemented methods disclosed herein may similarly be used to provide projection images that represent other types of interventional devices. For instance, the interventional device may be used to perform a medical procedure on another part of the anatomy, including the lungs and associated airways. By way of some examples, the interventional device may alternatively be a catheter, a thrombectomy device, an intravascular ultrasound “IVUS” imaging device, an endobronchial ultrasound “EBUS” device, an Optical Coherence Tomography “OCT” imaging device, a blood pressure device and/or flow sensor device, a TEE probe, and so forth.

It is noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including computer-readable instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product can be provided by dedicated hardware, or hardware capable of running the software in association with appropriate software. When provided by a processor, the functions of the method features can be provided by a single dedicated processor, or by a single shared processor, or by a plurality of individual processors, some of which can be shared. The functions of one or more of the method features may for instance be provided by processors that are shared within a networked processing architecture such as a client/server architecture, a peer-to-peer architecture, the Internet, or the Cloud. The explicit use of the terms “processor” or “controller” should not be interpreted as exclusively referring to hardware capable of running software, and can implicitly include, but is not limited to, digital signal processor “DSP” hardware, read only memory “ROM” for storing software, random access memory “RAM”, a non-volatile storage device, and the like. Furthermore, examples of the present disclosure can take the form of a computer program product accessible from a computer-usable storage medium, or a computer-readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable storage medium or a computer readable storage medium can be any apparatus that can comprise, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or a semiconductor system or device or propagation medium. Examples of computer-readable media include semiconductor or solid state memories, magnetic tape, removable computer disks, random access memory “RAM”, read-only memory “ROM”, rigid magnetic disks and optical disks. Current examples of optical disks include compact diskread only memory “CD-ROM”, compact disk-read/write “CD-R/W”, Blu-Ray™ and DVD.

As mentioned above, there remains a need for improvements that facilitate an understanding of the three-dimensional shape of interventional devices from X-ray projection images.

Fig. 1 is a flowchart illustrating an example of a computer-implemented method of providing a projection image representing an interventional device, in accordance with some aspects of the present disclosure. Fig. 2 is a schematic diagram illustrating an example of a system 200 for providing a projection image representing an interventional device, in accordance with some aspects of the present disclosure. The system 200 illustrated in Fig. 2 includes one or more processors 210. It is noted that operations described in relation to the method illustrated in Fig. 1 , may also be performed by the one or more processors 210 of the system 200 illustrated in Fig. 2. Likewise, operations described in relation to the one or more processors 210 of the system 200, may also be performed in the method described with reference to Fig. 1.

With reference to Fig. 1, the computer-implemented method of providing a projection image representing an interventional device 110, includes: receiving SI 10 X-ray projection image data 120a representing the interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device; inputting S120 the X-ray projection image data 120a into a neural network 150; and in response to the inputting, generating SI 30, using the neural network 150, predicted X- ray projection image data 120^pb representing the interventional device 110 from a second perspective 130b of the projection X-ray imaging system 140 with respect to the interventional device, the second perspective 130b being different to the first perspective 130a; and wherein the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, using training data comprising a plurality of X-ray projection training images 120a’ i .._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i..n, a corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b.

With reference to the method illustrated in Fig. 1, X-ray projection image data 120a is received in the operation SI 10. The X-ray projection image data 120a represents an interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device.

The X-ray projection image data 120a that is received in the operation SI 10 may be generated by a projection X-ray imaging system. Projection X-ray imaging systems typically include a support arm such as a so-called “C-arm”, or an “0-arm”, that supports an X-ray source-detector arrangement. Projection X-ray imaging systems may alternatively include a support arm with a different shape to these examples. Projection X-ray imaging systems may alternatively include a C-less system where the X-ray source and detector are unpaired and can be arranged independently of each other. Projection X-ray imaging systems typically generate X-ray projection image data with the support arm held in a static position with respect to an imaging region during the acquisition of image data. The X-ray projection image data 120a may be generated by the projection X-ray imaging system 140 illustrated in Fig. 2, for example. By way of an example, the X-ray projection image data may be generated by the Philips Azurion 7 X-ray imaging system marketed by Philips Healthcare, Best, The Netherlands.

In general, the X-ray projection image data 120a that is received in the operation SI 10 may represent one or more static images, or it may represent a temporal sequence of images. The temporal sequence of images may be generated substantially in real-time. Thus, the X-ray projection image data 120a may represent the interventional device 110 in real-time.

In general, the X-ray projection image data 120a that is received in the operation SI 10 may represent any region of interest in the anatomy. For example, the region of interest may be a portion of the brain, or the heart, or the lungs, and so forth. In some examples, the interventional device 110 is disposed in a vasculature, and the X-ray projection image data 120a also represents the vasculature.

The X-ray projection image data 120a that is received in the operation SI 10 may be received from various sources. For example, the X-ray projection image data 120a may be received from a projection X-ray imaging system, such as the projection X-ray imaging system 140 illustrated in Fig. 2, or it may be received from a computer readable storage medium, or from the Internet or the Cloud, for example. The X-ray image data may be received by the one or more processors 210 illustrated in Fig. 2. The X-ray projection image data 120a may be received via any form of data communication, including wired, optical, and wireless communication. By way of some examples, when wired or optical communication is used, the communication may take place via signals transmitted on an electrical or optical cable, and when wireless communication is used, the communication may for example be via RF or optical signals.

As mentioned above, the X-ray projection image data 120a that is received in the operation SI 10 represents an interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device. Fig. 3 is an example of X-ray projection image data 120a representing an interventional device 110 from a first perspective 130a of a projection X- ray imaging system 140 with respect to the interventional device, in accordance with some aspects of the present disclosure. The X-ray projection image data 120a illustrated on the left-hand side of Fig. 3 includes an interventional device 110 in the form of a guidewire that is disposed in the brain. The corresponding first perspective 130a of the projection X-ray imaging system 140 with respect to the interventional device is illustrated on the right-hand side of Fig. 3. The first perspective 130a may in general be any perspective with respect to the interventional device 110. The first perspective 130a may be defined in any coordinate system. For instance, the perspective may for instance be defined in a spherical, or in a cartesian coordinate system. The first perspective 130a of the projection X-ray imaging system 140 is defined with respect to the interventional device 110. A known transformation between the orientation of the interventional device and the patient may be used to further define the first perspective 130a with respect to a patient. For instance, in the example illustrated in Fig. 3, the first perspective 130a also represents a so-called anteroposterior “AP” view of the brain.

Returning to the method illustrated in Fig. 1, in the operation SI 20, the X-ray projection image data 120a is inputted into a neural network 150. The neural network 150 is trained to generate predicted X-ray projection image data 120^pb representing the interventional device 110 from a second perspective 130b, using training data comprising a plurality of X-ray projection training images 120a’ i.._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i.._n, a corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b.

The neural network 150 may be provided by various types of architectures. For example, the neural network 150 may have a convolutional neural network “CNN”, or an autoencoder, or a transforming autoencoder “TAE”, or a generative adversarial network “GAN”, or capsule network, or a regression network architecture. The neural network may also be provided by a combination of such architectures. In examples in which the X-ray projection image data 120a that is inputted into a neural network 150 represents a temporal sequence of images, the neural network may include recurrent neural network “RNN” or long short-term memory “LSTM” cells. Such cells may be used to generate outputs that are based on both current, and also historic, images in the temporal sequence. The use of such cells allows the neural network 150 to capture temporal information from the X-ray projection image data 120a.

In one example, the neural network 150 has a TAE architecture. This example is described with reference to Fig. 5, which is a first example of a neural network 150 that is trained to generate predicted X-ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, using training data comprising X-ray projection training images 120a’ representing the interventional device 110 from a first perspective 130a, and a corresponding ground truth X-ray projection training images 120b’ representing the interventional device 110 from the second perspective 130b, in accordance with some aspects of the present disclosure.

In the example described with reference to Fig. 5, the neural network 150 comprises a transforming autoencoder. The transforming autoencoder includes an encoder 150_e and a decoder 150d- The encoder 150_e, and the decoder 150d may include one or more convolutional layers. The encoder 150_e is trained to learn a latent space representation, LR, of the inputted X-ray projection image data 120a such that when the latent space representation of the inputted X-ray projection image data 120a is transformed by the orientation data 160, T and decoded by the decoder 150d, the decoded latent space representation represents the interventional device 110 from the second perspective 130b. Consequently, at inference, an instance of X-ray projection image data 120a is inputted into the neural network 150. The encoder 150_e generates a latent space representation, LR, of the inputted X-ray projection image data 120a. The transform, T, is then applied to the latent space representation to provide a transformed latent space representation, TLR. The transformed latent space representation, TLR, is then inputted into the decoder 150d, and the decoder 150d outputs a decoded latent space representation that represents the interventional device 110 from the second perspective 130b.

In the example described above with reference to Fig. 5, the transform, T, is a spatial transform that represents the change in perspective between the first perspective 130a, and the second perspective 130b. The transform, T, is associated with the training data because the training data includes X-ray projection training images 120a’ i.._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i.._n, a corresponding ground truth X- ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b. In general, the neural network 150 may be trained using any transform, T, and the transform, T, may represent any change in perspective between the first perspective 130a, and the second perspective 130b. By way of an example, the first perspective 130a and the second perspective 130b may be mutually orthogonal. This change in perspective is useful in many medical procedures.

An example of the transform, T, is given in Equation 1 below for a transform which represents a rotation of the perspective of the projection X-ray imaging system in the x-y plane through 90 degrees around the y-axis in cartesian space: cos 90 0 sin 90

. = . . (90) = • 0 1 0 • Equation 1

• sin 90 0 cos 90 If predictions are desired for a single change in perspective, the neural network 150 may be trained using a set of X-ray projection training images that represent the interventional device 110 from the first perspective 130a and from the second perspective 130b, and the corresponding transform, T. At inference, the same transform, T, is applied to the latent space representations, LR, by the neural network. The transform, T, may be hidden from a user and applied to the latent space representations LR within the neural network at inference, or it may be applied as an input to the neural network at inference.

If predictions are desired for multiple changes in perspective, multiple neural networks may be trained, each neural network being trained using training data, and the corresponding transform, for a different change in perspective. Alternatively, a single neural network may be trained to make predictions for multiple changes in perspective. In this case, the neural network 150 is trained using training data that includes for each change in perspective: a set of X-ray projection training images representing the interventional device 110 from the first perspective 130a and from the second perspective 130b, and the corresponding transform.

In the case that a single neural network is trained to make predictions for multiple changes in perspective, during training, the transform, T, is applied as an input to the neural network. The neural network then learns to base its predictions on the desired transform, T. At inference, the transform, T, for the desired change in perspective, is inputted into the neural network, and predictions are made for that transform. In this case, the transform, T, may be selectable automatically, or based on user input. If the transform is selectable automatically, the neural network may be controlled so as to make sequential predictions for different changes in perspective. Thus, the neural network may be controlled to make near-real-time predictions for multiple different changes in perspective.

In the examples described above wherein a transform, T, is inputted into the neural network, the transform, T, is inputted into the neural network in the form of orientation data. In these examples, the method described with reference to Fig. 1 also includes: receiving orientation data 160, T defining the second perspective 130b of the projection X-ray imaging system 140; and inputting the received orientation data 160, T into the neural network 150; and wherein the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, based on the orientation data 160, T.

In some examples, prior to inputting the X-ray projection image data 120a into the neural network 150, a segmentation operation is performed on the X-ray projection image data 120a in order to identify the interventional device. This is illustrated in the lower portion of Fig. 5 via the label “Segment”. Various segmentation algorithms may be used for this purpose, including model-based segmentation, watershed-based segmentation, region growing, level sets, graphcuts, and so forth. A neural network may also be trained to segment the X-ray projection image data 120a in order to identify the interventional device. Identifying the interventional device in this manner prior to inputting the X-ray projection image data 120a into the neural network, facilitates the neural network to focus on the interventional device, and consequently improves the predictions that are made by the neural network. In one example, the X-ray projection image data 120a that is inputted into the neural network represents only the interventional device. Data that represents only the interventional device may be extracted from the segmented X-ray projection image data 120a. Thus, features representing tissue, bone, and the vasculature, may be removed from the X-ray projection image data 120a that is inputted into the neural network 150. Such features contain larger variations and can hamper the predictions of the neural network 150, and consequently by removing such features, the reliability of the neural network’s predictions may be improved.

In general, the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, by: receiving the training data; and for each of a plurality of the X-ray projection training images 120a’ i.._n in the training data representing the interventional device 110 from the first perspective 130a: inputting the X-ray projection training image 120a’ i.._n into the neural network 150; generating a predicted X-ray projection image 120^pbi.._n representing the interventional device 110 from the second perspective 130b, using the neural network 150; and adjusting parameters of the neural network 150 based on a difference between the predicted X-ray projection image 120^pbi.._n and the corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b; and repeating the inputting, the generating, and the adjusting, until a stopping criterion is met. The training data that is used in these operations includes a plurality of X-ray projection training images 120a’ i.._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i.._n, a corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b. The X-ray projection training images 120aT .._n in the training data may represent the same type of interventional device as that for which the predictions are to be made at inference. The X-ray projection training images 120a’ i..n may represent the interventional device in the same type of region of interest as that in which the interventional device is disposed at inference. The training data may represent the interventional device 110 during procedures on different subjects. For instance, the plurality of X-ray projection training images 120aT.._n may include images from some tens, or hundreds, or thousands, or more, medical procedures that have been performed on different subjects. The subjects may have a range of different ages, body mass index, gender, and so forth. Thus, the training data may represent the interventional device in a variety of different shapes.

In the example neural network 150 illustrated in Fig. 5, the inputting of X-ray projection training images 120a’ i.._n into the neural network 150 during training is illustrated on the left-hand side of Fig. 5. During training, the encoder 150_e generates a latent space representation LR of each inputted X- ray projection training image 120a’ i.._n. The transform, T, is then applied to the latent space representation to provide a transformed latent space representation, TLR. The transformed latent space representation, TLR, is then inputted into the decoder 150d, and the decoder 150d generates a predicted X-ray projection image 120^pbi.._n representing the interventional device 110 from the second perspective 130b. Parameters of the neural network 150 are then adjusted based on a difference between the predicted X-ray projection image 120^pbi.._n and the corresponding ground truth X-ray projection training image 120bT.._n representing the interventional device 110 from the second perspective 130b. The difference is calculated using a loss function, as illustrated on the right-hand side of Fig. 5. Loss functions such as the LI norm, the L2 norm, negative log likelihood, and so forth may be used for this purpose. Loss functions may also be used that force the neural network 150 to learn distributions from which the latent space representation may be sampled to be similar to a reference distribution. For example, the values of loss functions such as the Kullback Leibler “KL” divergence, may be used for this purpose, and which use a standard gaussian distribution as the reference distribution.

If predictions are desired for multiple different perspectives at inference, then the training operations described above are performed using training data that includes for each change in perspective: a set of X-ray projection training images representing the interventional device 110 from the first perspective 130a and from the second perspective 130b, and the corresponding transform, T. Thus, in this case, the transform, T, that corresponds to the change in perspective between the first perspective 130a and the second perspective 130b, is also inputted into the neural network during training.

During training, the operations of inputting the X-ray projection training image 120a’ i.._n into the neural network 150, generating a predicted X-ray projection image 120^pbi.._n, and adjusting parameters of the neural network 150, in the training method described above, are repeated until a stopping criterion is met. Training is terminated when the value of the loss function satisfies a stopping criterion. This indicates that the trained neural network 150 is capable of predicting X-ray projection images representing the interventional device 110 from the second perspective 130b, with an acceptable level of accuracy.

As described above, the training of the neural network 150 involves inputting a training dataset into the neural network, and iteratively adjusting the neural network’s parameters until the trained neural network provides an accurate output. Training is often performed using a Graphics Processing Unit “GPU” or a dedicated neural processor such as a Neural Processing Unit “NPU” or a Tensor Processing Unit “TPU”. Training often employs a centralized approach wherein cloud-based or mainframe-based neural processors are used to train a neural network. Following its training with the training dataset, the trained neural network may be deployed to a device for analyzing new input data during inference. The processing requirements during inference are significantly less than those required during training, allowing the neural network to be deployed to a variety of systems such as laptop computers, tablets, mobile phones and so forth. Inference may for example be performed by a Central Processing Unit “CPU”, a GPU, an NPU, a TPU, on a server, or in the cloud. The process of training the neural network 150 described above therefore includes adjusting the parameters of its encoder 150_e and its decoder 150d- The parameters, or more particularly the weights and biases, control the operation of activation functions in the neural network. In supervised learning, the training process automatically adjusts the weights and the biases, such that when presented with the input data, the neural network accurately provides the corresponding expected output data. In order to do this, the value of the loss functions, or errors, are computed based on a difference between predicted output data and the expected output data. The value of the loss function may be computed using functions such as the negative log-likelihood loss, the mean absolute error (or LI norm), the mean squared error, the root mean squared error (or L2 norm), the Huber loss, or the (binary) cross entropy loss. Other loss functions like the Kullback-Leibler divergence may additionally be used when training a variational autoencoder to ensure that the distribution of latent space encodings generated from X-ray projection training images 120a’ i.._n, is similar to a standard Gaussian distribution with mean 0 and standard deviation of 1. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function satisfies a stopping criterion. Sometimes, training is terminated when the value of the loss function satisfies one or more of multiple criteria.

Various methods are known for solving the loss minimization problem such as gradient descent, Quasi-Newton methods, and so forth. Various algorithms have been developed to implement these methods and their variants including but not limited to Stochastic Gradient Descent “SGD”, batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax “optimizers”. These algorithms compute the derivative of the loss function with respect to the model parameters using the chain rule. This process is called backpropagation since derivatives are computed starting at the last layer or output layer, moving toward the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted in order to minimize the error function. That is, adjustments to model parameters are made starting from the output layer and working backwards in the network until the input layer is reached. In a first training iteration, the initial weights and biases are often randomized. The neural network then predicts the output data, which is likewise, random. Backpropagation is then used to adjust the weights and the biases. The training process is performed iteratively by making adjustments to the weights and biases in each iteration. Training is terminated when the error, or difference between the predicted output data and the expected output data, is within an acceptable range for the training data, or for some validation data. Subsequently the neural network may be deployed, and the trained neural network makes predictions on new input data using the trained values of its parameters. If the training process was successful, the trained neural network accurately predicts the expected output data from the new input data.

Returning to the method illustrated in Fig. 1, at inference, the method continues with the operation SI 30, in which, in response to the inputting in the operation SI 20, predicted X-ray projection image data 120^pb is generated using the neural network. The predicted X-ray projection image data 120^pb represents the interventional device 110 from the second perspective 130b of the projection X-ray imaging system 140 with respect to the interventional device. With reference to the example neural network 150 illustrated in Fig. 5, this operation is performed using the neural network 150 that has been trained using the techniques described above.

The predicted X-ray projection image data 120^pb is then outputted. Fig. 4 is an example of a predicted X-ray projection image 120^pb representing an interventional device 110 from a second perspective 130b of a projection X-ray imaging system 140 with respect to the interventional device, in accordance with some aspects of the present disclosure. As compared to the image representation of the corresponding X-ray projection image data 120a in Fig. 3 that was inputted into the neural network 150, the second perspective 130b is rotated by 90 degrees with respect to that in Fig. 3. The example illustrated in Fig. 4 represents a so-called lateral view of the brain. As may be appreciated, by providing the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, i.e. by providing the image representation illustrated in Fig. 4, a user can better visualize the three-dimensional shape of the interventional device. Moreover, since the predicted X-ray projection image data for the second perspective, is generated without the need to actually acquire X-ray projection image data from this perspective, the method facilitates a user to visualize the three- dimensional shape of the interventional device without increasing the amount of X-ray radiation dose that is delivered to a patient.

Fig. 6 is a second example of the training of a neural network 150 to generate predicted X-ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, using training data comprising X-ray projection training images 120a’ representing the interventional device 110 from a first perspective 130a, and a corresponding ground truth X-ray projection training images 120b’ representing the interventional device 110 from the second perspective 130b, in accordance with some aspects of the present disclosure. The neural network 150 illustrated in Fig. 6 is a capsule network implementation of the neural network illustrated in Fig. 5 and may be used as an alternative to the neural network illustrated in Fig. 5. Capsule neural networks are described further in a document by Hinton, G. E. et al., “Transforming Auto-Encoders”, Artificial Neural Networks and Machine Learning - ICANN 2011, Lecture Notes in Computer Science, 6791: 44-51. In the neural network illustrated in Fig. 6, each segment encapsulated in the oval boundary is a capsule Ci.j that contains a neural network, such as the neural network 150 illustrated in Fig. 5. The encoder 150_e of the neural network on the left-hand side of each of the capsules additionally outputs a confidence measure (denoted by p in Fig. 6) in the range [0,1] in its generated output. The decoder 150_e of the neural network on the right-hand side of each of the capsules generates an output. The confidence measure for each encoder within a capsule is multiplied by the output of the decoder of the capsule, and the outputs of the capsules are combined to provide a combined output for the capsules. The confidence measure, p, may also be outputted such that a user can modulate their trust in the system accordingly. The training of the neural network is performed in the same manner as described above for the example illustrated in Fig. 5, and all capsules Ci..j are trained simultaneously. As compared to the neural network illustrated in Fig. 5, the use of the capsule implementation illustrated in Fig. 6 has the advantage of gaining a consensus from the outputs of each of the capsules. Each capsule may focus on different parts of the image (e.g., a capsule may focus on only the device tip while another may focus on a larger extent of the distal end of the device). This is achieved by an additional set of weights that are used to weight the inputs to each of the capsules. The outputs from each of these capsules must agree in order to generate a combined output that produces a low error when compared to the ground truth.

Variations of the above-described method are also contemplated, as described in the examples below.

In one example, the interventional device 110 is disposed in a vasculature, and the X-ray projection image data 120a also represents the vasculature. In this case, if the X-ray projection training images 120a’ i.._n, and the ground truth X-ray projection training images 120b’ i.._n, also represent the vasculature, the neural network 150 may be trained to generate predicted X-ray projection image data 120^pb that also represents the vasculature. The interventional device 110 may be disposed in any part of the vasculature. For instance, it may be disposed in the brain, or in the peripheral vasculature, such as in the arm, or in the leg. The vasculature provides valuable context to the shape of the interventional device, and consequently by providing predictions of the vasculature in the predicted X-ray projection image data 120^pb, a user may obtain an improved understanding of its shape. In this example, a GAN may be used as the neural network 150. The neural network in this example is trained in the same manner as described above with reference to Fig. 5.

In another example, the X-ray projection image data 120a, the predicted X-ray projection image data 120^pb, the X-ray projection training images 120a’ i.._n, and the ground truth X-ray projection training images 120bT.._n, each comprise temporal sequences of images representing the interventional device 110. In this example, the neural network 150 is trained 150 to generate the predicted X-ray projection image data 120^pb for a current X-ray image in the inputted X-ray projection image data 120a, using the current X-ray image in the inputted X-ray projection image data 120a and one or more historic X-ray images in the inputted X-ray projection image data 120a. In this example, the encoder 150_e and decoder 150d illustrated in Fig. 5 may include RNN, or LSTM, cells, as described above.

In another example, the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, is constrained using angiographic image data 170b representing the vasculature. In this example, the interventional device 110 is disposed in a vasculature, and the method described with reference to Fig. 1 also includes: receiving angiographic image data 170b representing the vasculature; and constraining the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, using the received angiographic image data 170b, such that the predicted X-ray projection image data 120^pb represents the interventional device within the vasculature. In this example, the angiographic image data may be provided in the form of a preoperative computed tomography “CT” angiography image representing the vasculature. The CT image may be obtained from a CT imaging system. Alternatively, the angiographic image data may be provided in the form of an intra-operative 3D rotational angiography “3DRA” image that is obtained using a projection X-ray imaging system. The 3DRA image may be obtained from a cone beam CT “CBCT” imaging system. Alternatively, the angiographic image data may be provided in the form of an intraoperative projection image obtained from a biplane projection X-ray imaging system that represents the vasculature from the second perspective 130b. Such data is often obtained prior to, or during medical investigations, and includes the second perspective 130b of the vasculature. In some examples, the angiographic image data is obtained subsequent to the injection of a contrast agent into the vasculature in order to enhance the visibility of the vasculature, and prior to navigation of a device in the vasculature. If the angiographic image data is generated by a biplane projection X-ray imaging system, image data that represents the vasculature from the second perspective 130b may be provided directly by one of its biplane images. Similarly, if the angiographic image data is provided by a 3DRA image, image data that represents the vasculature from the second perspective 130b may be provided by selecting data from a perspective that matches the second perspective 130b. If the angiographic image data is provided by a CT image, image data that represents the vasculature from the second perspective 130b may be provided in the form of a digitally reconstructed radiograph “DRR” generated by arranging a virtual source and a virtual detector at the second perspective 130b with respect to the CT image, and projecting virtual X-rays emitted by the virtual X-ray source through the CT data and onto the virtual X-ray detector. Thus, in this case, the angiographic image data comprises a computed tomography, CT, image representing the vasculature, and wherein the method includes: projecting the angiographic image data to provide a projection of the CT image representing the vasculature from the second perspective 130b; and the constraining the predicted X-ray projection image data 120^pb, is performed using the projection of the CT image.

In this example, the constraining operation is performed on the predicted X-ray projection image data 120^pb after it has been outputted by the neural network 150. In other words, it may be described as a post-processing operation that is applied to the output of the neural network. The constraining operation may be performed by deforming the interventional device in the predicted X-ray projection image data 120^pb such that it represents the interventional device within the vasculature. In other words, the shape of the interventional device is deformed such that it fits within the vasculature. Known deformation techniques such as the use of displacements fields may be used for this purpose. Displacement fields can be used to specify in which direction a pixel in an image must move in order to generate a new image. Displacement fields may, therefore, be used to define how pixels belonging to the interventional device should be moved such that they fit within the vasculature. The deformation may also be performed subject to mechanical constraints of the interventional device. Thus, limits on factors such as the connectivity between sections of the interventional device, and the amount of curvature of the interventional device, may also be applied in this operation in order to ensure that the prediction of the interventional device shape is valid.

In another example, rather than constraining the predicted X-ray projection image data 120^pb after it has been outputted by the neural network 150, the neural network 150 is trained to generate constrained predicted X-ray projection image data 120^. In this example, the neural network 150 is trained to generate constrained predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b wherein the predicted X-ray projection image data 120^pb is constrained by the angiographic image data 170b, and the training data further comprises an angiographic image representing the vasculature.

This example is described with reference to Fig. 7, which is an example of a neural network 150 that is trained to generate constrained predicted X-ray projection image data 120^pb representing an interventional device 110 from a second perspective 130b, and wherein the predicted X- ray projection image data 120^pb is constrained by angiographic image data 170b, L, in accordance with some aspects of the present disclosure. In this example, X-ray projection image data 120a is inputted into the neural network 150, as illustrated on the left-hand side of Fig. 7. In the illustrated example, the interventional device has been segmented from the X-ray projection image data 120a that is labelled “Device navigation on AP”, and only the X-ray projection image data 120a that represents the interventional device is inputted into the neural network. The X-ray projection image data 120a may be provided by a projection X-ray imaging system. For example it may be provided by one of the planes of a biplane projection X-ray imaging system. In the illustrated example, biplane angiographic images Ii and I2 are acquired prior to inference. The biplane angiographic images represent the vasculature in which the interventional device will be disposed at inference. The biplane angiographic image Ii is acquired from the first perspective 130a, i.e. the perspective from which projections are to be made with the X-ray projection image data 120a. The biplane angiographic image I2 is acquired from the second perspective 130b, and therefore represents the shape of the vasculature as seen from the perspective for which the predicted X-ray projection image data 120^pb is to be generated by the neural network 150. In this example, angiographic image data 170b in the form of the biplane angiographic image I2, is inputted into the neural network 150 and used to generate the constrained predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b. The constrained predicted X-ray projection image data 120^pb may be overlaid onto the biplane angiographic image I2 in order to provide anatomical context to the shape of the interventional device as seen from the second perspective 130b. Such an overlaid image is illustrated in the image labelled “Device navigation on virtual biplane”

The training data that is used to train the neural network 150 in this example includes the X-ray projection training images 120a’ i.._n, the corresponding ground truth X-ray projection training images 120b’ i.._n, and also angiographic image data 170b. The neural network implicitly learns to associate the curvature of the vasculature from the inputted angiographic image data 170b with the curvature of the interventional device. Additional loss functions may be used to penalize predictions where the interventional device 110 does not overlap with the vasculature in angiographic image data 170b in order to force the neural network to learn to associate the curvature of the vasculature from the inputted angiographic image data 170b with the curvature of the interventional device.

In a related example, the angiographic image data 170b in the previous example comprises a CT image representing the vasculature, and the CT image also represents a portion of the interventional device 110. In this example, the angiographic image data 170b is projected in order to provide a projection of the of both the vasculature, and the interventional device, from the second perspective 130b. The predictions of the neural network are also constrained such that the predicted shape of the interventional device in the predicted X-ray projection image data 120^pb, matches the shape of the interventional device in the projection of the angiographic image data 170b. Thus, in this example, the shape of the interventional device from the second perspective, as determined from the angiographic image data 170b, is used by the neural network as a constraint in its predictions.

In this example, angiographic image data comprises a computed tomography, CT, image representing the vasculature, and also at least a portion of the interventional device 110, and the method described with reference to Fig. 7 includes: projecting the angiographic image data to provide a projection of the CT image representing the vasculature from the second perspective 130b; and the constraining the predicted X-ray projection image data 120^pb, is performed using the projection of the CT image.

The method also includes: segmenting the portion of the interventional device 110 in the angiographic image data; projecting the segmented portion of the interventional device to provide a projection of the segmented portion of the interventional device from the second perspective 130b; and constraining the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, using the projected segmented portion of the interventional device, such that the predicted X-ray projection image data 120^pb corresponds to the projected segmented portion of the interventional device.

This example is illustrated in Fig. 8, which is a schematic diagram including predicted X- ray projection image data 120^pb, 120^pc respectively representing an interventional device 110 from a second perspective 130b, and from a third perspective 130c, of a projection X-ray imaging system 140, in accordance with some aspects of the present disclosure. In the illustrated example, the upper-left portion of Fig. 8 illustrates a CT image representing the vasculature, and a portion of the interventional device 110. The CT image is a so-called prior CT image that is generated prior to the live X-ray projection image 120a illustrated in the upper right-hand portion of Fig. 8. The central arrow in the CT image shows the first perspective 130a of a projection X-ray imaging system, and from which perspective, the current live X-ray projection image 120a illustrated on the upper right-hand side of Fig. 8 is generated. Referring now to the image in the upper right-hand side of Fig. 8. The live X-ray projection image 120a is a current image that is generated by a projection X-ray imaging system from the orientation 130a that is illustrated in the CT image. The path A-B-C-D illustrates the extent of the interventional device in the current projection image 120a. The first portion A-B of the path A-B-C-D also corresponds to the projected shape of the interventional device that is obtained by projecting the CT image from the orientation 130a. The second, dashed, portion of the path A-B-C-D, i.e. B-C-D illustrates the further extent of the interventional device in the current projection image 120a. The predictions of the neural network are illustrated for the second perspective 130b, and for the third perspective 130c, in the lower right-hand portion of Fig. 8, and in the lower left-hand portion of Fig. 8, respectively. In the example illustrated in Fig. 8, predictions are made for both the second perspective 130b, as well as a third perspective 130c. However, it is noted that predictions may alternatively be made for only one perspective, i.e. the second perspective 130b. The image in the lower right-hand portion of Fig. 8 shows the predicted shape of the interventional device from the second perspective 130b along the path A-D’. This path includes the portion A-B, and which corresponds to the projected shape of the interventional device that is obtained by projecting the CT image from the perspective 130b, and also the portion B-D’ which shows the predicted shape of the interventional device that is predicted by the neural network. The lower left-hand portion of Fig. 8 shows the projected, and predicted shape of the interventional device in a similar manner. In this image, the portions A-B, and B-D’, respectively represent the projected, and predicted shape of the interventional device for the third perspective 130c.

In another example, device type data 180 that is indicative of a type of the interventional device 110 represented in the X-ray projection image data 120a, is inputted into the neural network during training, and also at inference. In this example, the method described with reference to Fig. 1 also includes: receiving device type data 180 indicative of a type of the interventional device 110 represented in the X-ray projection image data 120a; inputting the received device type data into the neural network 150; and wherein the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, based on the device type data 180.

The training of the neural network using the device type data allows the neural network to make predictions at instance for different types of interventional devices. In this example, the device type data may refer to a category of the interventional device, such as “guidewire” or “IVUS imaging device”, or it may refer to a more specific identifier, such as a supplier, or a model number of the interventional device. Examples in which the device type 180 is inputted into the neural network are illustrated in Fig. 5 and in Fig. 7. The device type data may be selected manually by a user, as indicated in these examples and in which the device type is user-selectable from a menu of options. Alternatively, the device type data may be detected automatically. In the latter case, the device type may be detected by performing an object detection operation on the X-ray projection image data 120a using a feature detector. Alternatively, the device type may be detected automatically before the device is inserted in the patient, from video data generated during a medical procedure using object detection techniques.

In another example, device constraint data is used to constrain the predictions of the neural network. In this example, the method described with reference to Fig. 1 includes: receiving device constraint data defining a mechanical and/or dimension constraint for the interventional device 110 represented in the X-ray projection image data 120a; and constraining the predicted X-ray projection image data 120^pb, using the received device constraint data, such that the predicted X-ray projection image data 120^pb represents the interventional device 110 within the mechanical and/or dimension constraints defined by the device constraint data.

An example of a mechanical constraint that may be applied in this example is a limit on the shape of the interventional device, such as an amount of curvature. For example, it may be specified that the curvature of the interventional device is below a specified limit. Another example of a mechanical constraint is a connectedness of interventional device. For example, it may be specified that the interventional device is fully connected along its length. An example of a dimension constraint is a length of the interventional device. For example, it may be specified that the length of the interventional device is within a specified range. The constraining of the predicted X-ray projection image data 120^pb may be performed using various techniques. For instance, the constraining may be performed within the neural network by penalizing predictions that deviate from the constraint. Alternatively, a post-processing operation may be performed on the predicted X-ray projection image data 120^pb that is generated by the neural network 150. In this approach, the predicted X-ray projection image data 120^pb that is generated by the neural network 150 may be deformed using displacement fields, as described above, and in which any violations of the constraint in the neural network’s predictions are rectified by identifying the location in which the constraint is violated, and applying a deformation to the predicted X-ray projection image data 120^pb that ensures that the constraint is met. In another post-processing approach, a projection of a 3D computer aided design “CAD” model that represents the interventional device from the first perspective 130a may be deformed such that the prediction of the shape of the interventional device from the second perspective 130b satisfies the mechanical and/or dimension constraint. By applying such constraints, it may be ensured that the predictions of the interventional device shape are valid. Such constraints may be applied in combination with the constraint described above in which the interventional device is deformed such that it fits within the vasculature.

In another example, a confidence value is calculated for the predicted X-ray projection image data 120^pb. In this example, the method described with reference to Fig. 1 includes: calculating a confidence value p for the predicted X-ray projection image data 120^pb; and outputting an indication of the confidence value p.

The confidence value, p, may be calculated using various techniques. For example, the neural network 150 may calculate the confidence value using the dropout technique. The dropout technique involves iteratively inputting the same data into the neural network 150 and determining the neural network’ s output whilst randomly excluding a proportion of the neurons from the neural network in each iteration. The outputs of the neural network are then analyzed to provide mean and variance values. The mean value represents the final output, and the magnitude of the variance indicates whether the neural network is consistent in its predictions, in which case the variance is small and the confidence value is relatively higher, or whether the neural network was inconsistent in its predictions, in which case the variance is larger and the confidence value is relatively lower. The confidence may alternatively be calculated using e.g. the Kullback-Leibler “KL” divergence, between the distribution that the latent space representation, LR, is sampled from, and the distribution over the current trained encodings. This divergence indicates how well the input sequence is represented by the learned encodings. Alternatively, the confidence value may be calculated based on the sharpness of images representing the predicted X-ray projection image data 120^pb. For instance, a blurry image is indicative of low confidence, whereas a sharper image is indicative of higher confidence. A low value of confidence may indicate that the trained neural network 150 is not suitable for processing the inputted X-ray projection image data 120a. For example, the input data may be out-of-distribution as compared to the data that was used to train the neural network.

The indication of the confidence value p may be outputted in various ways. For example, the confidence value p may be outputted as a numerical value. Alternatively, a graphical representation of the confidence value p may be generated by multiplying an image representation of the predicted X-ray projection image data 120^pb by the confidence value p. This results in a relatively higher contrast image if confidence is relatively higher, and relatively lower contrast image if confidence is relatively lower. A graphical representation of the confidence value p may alternatively be provided as a heatmap image. The outputting of the confidence value p in this example allows a user to modulate their trust in the predictions of the neural network accordingly.

As mentioned above, a source of training data for training the neural network 150 is the projection images from historic procedures that have been performed using biplane projection X-ray imaging systems. Such procedures typically yield a single pair of perspectives of the interventional device per procedure. This limits the amount of training data that is available for training the neural network. In one example, the neural network is trained using synthetic X-ray projection training images. In this example, the training data that is used to train the neural network 150 comprises a plurality of synthetic X-ray projection training images representing the interventional device 110 from the first perspective 130a, and for each synthetic X-ray projection training image, a corresponding ground truth synthetic X- ray projection training image representing the interventional device from the second perspective 130b. The synthetic X-ray projection training images are generated by projecting 3D tracking data representing a shape of the interventional device from each of the first perspective 130a, and the second perspective 130b, respectively. In this example, the operation of projecting the 3D tracking data to provide X-ray projection training images representing the interventional device from the first perspective 130a and the second perspective 130b, may be performed by arranging a virtual source and a virtual detector at the desired perspective 130a, 130b with respect to the 3D tracking data, and projecting virtual X-rays emitted by the virtual X-ray source through the 3D tracking data and onto the virtual X-ray detector.

Since, in this example, synthetic X-ray projection training images are generated by projecting 3D tracking data representing a shape of the interventional device, a dataset from one historic procedure can be used to generate multiple pairs of perspectives of training data. This increases the amount of training data that is provided per procedure, and also facilitates the generation of training data with any desired pair of perspectives 130a, 130b. In this example, the tracking data may be provided by a tracking system such as an electromagnetic tracking system, or an optical fiber-based tracking system, or a dielectric mapping system. An example of an electromagnetic tracking system is disclosed in the document US 2020/397510 Al. An example of an optical fiber-based tracking system that uses a strain sensor to determine the position of an interventional device is disclosed in the document US 2012/323115 Al.

In another example, a computer program product, is provided. The computer program product comprises instructions which when executed by one or more processors, cause the one or more processors to carry out a method of providing a projection image representing an interventional device 110. The method comprises: receiving SI 10 X-ray projection image data 120a representing the interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device; inputting S120 the X-ray projection image data 120a into a neural network 150; and in response to the inputting, generating SI 30, using the neural network 150, predicted X- ray projection image data 120^pb representing the interventional device 110 from a second perspective 130b of the projection X-ray imaging system 140 with respect to the interventional device, the second perspective 130b being different to the first perspective 130a; and wherein the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, using training data comprising a plurality of X-ray projection training images 120a’ i .._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i.._n, a corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b.

In another example, a system 200 for providing a projection image representing an interventional device 110, is provided. The system includes one or more processors 210 configured to: receive X-ray projection image data 120a representing the interventional device 110 from a first perspective 130a of a projection X-ray imaging system 140 with respect to the interventional device; input the X-ray projection image data 120a into a neural network 150; and in response to the input, generate using the neural network 150, predicted X-ray projection image data 120^pb representing the interventional device 110 from a second perspective 130b of the projection X-ray imaging system 140 with respect to the interventional device, the second perspective 130b being different to the first perspective 130a; and wherein the neural network 150 is trained to generate the predicted X-ray projection image data 120^pb representing the interventional device 110 from the second perspective 130b, using training data comprising a plurality of X-ray projection training images 120a’ i.._n representing the interventional device 110 from the first perspective 130a, and for each X-ray projection training image 120a’ i..n, a corresponding ground truth X-ray projection training image 120b’ i.._n representing the interventional device 110 from the second perspective 130b.

An example of the system 200 is illustrated in Fig. 2. It is noted that the system 200 may also include one or more of: an interventional device 110, a medical imaging system for generating the X- ray projection image data 120a, such as for example the projection X-ray imaging system 140 illustrated in Fig. 2; a patient bed 220, a monitor 230 for displaying the predicted X-ray projection image data 120^pb, and a user interface device (not illustrated in Fig. 2) that is configured to receive user input relating to the methods described above, such as a mouse, a touchscreen, a keyboard, a joystick, and so forth.

The above examples are to be understood as illustrative of the present disclosure, and not restrictive. Further examples are also contemplated. For instance, the examples described in relation to computer-implemented methods, may also be provided by the computer program product, or by the computer-readable storage medium, or by the system 200, in a corresponding manner. It is to be understood that a feature described in relation to any one example may be used alone, or in combination with other described features, and may be used in combination with one or more features of another of the examples, or a combination of other examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims. In the claims, the word “comprising” does not exclude other elements or operations, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. Any reference signs in the claims should not be construed as limiting their scope.

Claims

CLAIMS:

1. A computer-implemented method of providing a projection image representing an interventional device (110), the method comprising: receiving (SI 10) X-ray projection image data (120a) representing the interventional device (110) from a first perspective (130a) of a projection X-ray imaging system (140) with respect to the interventional device; inputting (S120) the X-ray projection image data (120a) into a neural network (150); and in response to the inputting, generating (S130), using the neural network (150), predicted X-ray projection image data (120^pb) representing the interventional device (110) from a second perspective (130b) of the projection X-ray imaging system (140) with respect to the interventional device, the second perspective (130b) being different to the first perspective (130a); and wherein the neural network (150) is trained to generate the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), using training data comprising a plurality of X-ray projection training images (120a’ i.._n) representing the interventional device (110) from the first perspective (130a), and for each X-ray projection training image (120a’ i..n), a corresponding ground truth X-ray projection training image (120b’ i.._n) representing the interventional device (110) from the second perspective (130b).

2. The computer-implemented method according to claim 1 , wherein the method further comprises: receiving orientation data (160, T) defining the second perspective (130b) of the projection X-ray imaging system (140); and inputting the received orientation data (160, T) into the neural network (150); and wherein the neural network (150) is trained to generate the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), based on the orientation data (160, T).

3. The computer-implemented method according to claim 2, wherein the neural network (150) comprises a transforming autoencoder comprising an encoder (150_e) and a decoder (150d), and wherein the encoder ( 150_e) is trained to learn a latent space representation (LR) of the inputted X-ray projection image data (120a) such that when the latent space representation of the inputted X-ray projection image data (120a) is transformed by the orientation data (160, T) and decoded by the decoder (150d), the decoded latent space representation represents the interventional device (110) from the second perspective (130b).

4. The computer-implemented method according to any one of claims 1 - 3, wherein the interventional device (110) is disposed in a vasculature, and wherein the received X-ray projection image data (120a), the predicted X-ray projection image data (120^pb), the X-ray projection training images (120a’ i..n), and the ground truth X-ray projection training images (120b’ i.._n), further represent the vasculature.

5. The computer-implemented method according to any one of claims 1 - 4, wherein the received X-ray projection image data (120a), the predicted X-ray projection image data (120^pb), the X-ray projection training images (120a’ i.._n), and the ground truth X-ray projection training images (120b’ i.._n), each comprise temporal sequences of images representing the interventional device (110); and wherein the neural network (150) is trained (150) to generate the predicted X-ray projection image data (120^pb) for a current X-ray image in the inputted X-ray projection image data (120a), using the current X-ray image in the inputted X-ray projection image data (120a) and one or more historic X-ray images in the inputted X-ray projection image data (120a).

6. The computer-implemented method according to any previous claim, wherein the interventional device (110) is disposed in a vasculature, and wherein the method further comprises: receiving angiographic image data (170b) representing the vasculature; and constraining the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), using the received angiographic image data (170b), such that the predicted X-ray projection image data (120^pb) represents the interventional device within the vasculature.

7. The computer-implemented method according to claim 6, wherein the neural network (150) is trained to generate constrained predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b) wherein the predicted X-ray projection image data (120^pb) is constrained by the angiographic image data (170b), and wherein the training data further comprises an angiographic image representing the vasculature.

8. The computer-implemented method according to claim 6 or claim 7, wherein the received angiographic image data comprises a computed tomography, CT, image representing the vasculature, and wherein the method further comprises: projecting the angiographic image data to provide a projection of the CT image representing the vasculature from the second perspective (130b); and wherein the constraining the predicted X-ray projection image data (120^pb), is performed using the projection of the CT image.

9. The computer-implemented method according to claim 8, wherein the received angiographic image data further represents at least a portion of the interventional device (110), and wherein the method further comprises: segmenting the portion of the interventional device (110) in the angiographic image data; projecting the segmented portion of the interventional device to provide a projection of the segmented portion of the interventional device from the second perspective (130b); and constraining the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), using the projected segmented portion of the interventional device, such that the predicted X-ray projection image data (120^pb) corresponds to the projected segmented portion of the interventional device.

10. The computer-implemented method according to any previous claim, wherein the method further comprises: receiving device type data (180) indicative of a type of the interventional device (110) represented in the X-ray projection image data (120a); inputting the received device type data into the neural network (150); and wherein the neural network (150) is trained to generate the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), based on the device type data (180).

11. The computer-implemented method according to any previous claim, wherein the method further comprises: receiving device constraint data defining a mechanical and/or dimension constraint for the interventional device (110) represented in the X-ray projection image data (120a); and constraining the predicted X-ray projection image data (120^pb), using the received device constraint data, such that the predicted X-ray projection image data (120^pb) represents the interventional device (110) within the mechanical and/or dimension constraints defined by the device constraint data.

12. The computer-implemented method according to any previous claim, wherein the method further comprises: calculating a confidence value (p) for the predicted X-ray projection image data (120^pb); and outputting an indication of the confidence value (p).

13. The computer-implemented method according to any previous claim, wherein the training data comprises a plurality of synthetic X-ray projection training images representing the interventional device (110) from the first perspective (130a), and for each synthetic X-ray projection training image, a corresponding ground truth synthetic X-ray projection training image representing the interventional device from the second perspective (130b), and wherein the synthetic X-ray projection training images are generated by projecting 3D tracking data representing a shape of the interventional device from each of the first perspective (130a), and the second perspective (130b), respectively.

14. The computer-implemented method according to any previous claim, wherein the first perspective (130a) and the second perspective (130b) are mutually orthogonal.

15. The computer-implemented method according to any one of claims 1 - 14, wherein the neural network (150) is trained to generate the predicted X-ray projection image data (120^pb) representing the interventional device (110) from the second perspective (130b), by: receiving the training data; and for each of a plurality of the X-ray projection training images (120a’ i.._n) in the training data representing the interventional device (110) from the first perspective (130a): inputting the X-ray projection training image (120a’ i.._n) into the neural network (150); generating a predicted X-ray projection image (120^pbi.._n) representing the interventional device (110) from the second perspective (130b), using the neural network (150); and adjusting parameters of the neural network (150) based on a difference between the predicted X-ray projection image (120^pbi.._n) and the corresponding ground truth X-ray projection training image (120bT.._n) representing the interventional device (110) from the second perspective (130b); and repeating the inputting, the generating, and the adjusting, until a stopping criterion is met.