WO2025255036A1

WO2025255036A1 - Systems and methods of generating synthetic image contrasts

Info

Publication number: WO2025255036A1
Application number: PCT/US2025/031944
Authority: WO
Inventors: Riccardo Lattanzi; Patricia Johnson; Narges Sharif RAZAVIAN
Original assignee: New York University NYU
Current assignee: New York University NYU
Priority date: 2024-06-05
Filing date: 2025-06-02
Publication date: 2025-12-11
Anticipated expiration: 2026-12-05

Abstract

The systems and methods of the present disclosure include a method. The method can include performing, by one or more processors, a scan using a magnetic resonance imaging (MRI) system. The method can include generating, by the one or more processors, based on signals from the scan, a T1 -weighted image and a T2-weighted image. The method can include generating, by the one or more processors, using a machine learning model, based on the Tl- weighted image and the T2-weighted image, at least one synthetic image contrast, wherein the T1 -weighted image and the T2-weighted image are input in the machine learning model, and the machine learning model outputs the at least one synthetic image contrast.

Description

SYSTEMSAND METHODS OF GENERATING SYNTHETIC

IMAGE CONTRASTS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Patent App. No. 63/656,523 filed on June 5, 2024, and U.S. Provisional Patent App. No. 63/662,375 filed on June 20, 2024, the disclosures of which are incorporated herein by reference in their entireties for all purposes.

TECHNICAL FIELD

[0002] The present disclosure relates generally to generating synthetic image contrasts.

BACKGROUND

[0003] Magnetic Resonance Imaging (MRI) exams are a common diagnostic tool in healthcare. However, these exams can be time-consuming due to a single MRI exam acquiring multiple image contrasts as part of protocol to evaluate different medical conditions. MRI exams can include multiple pulse sequences sequentially acquired with each pulse sequence generating images with different contrasts. Basic image contrasts obtained with different pulse sequences come from physical properties of the tissue, namely proton density (the density of proton spins), T1 and T2 relaxation times. Other more advanced image contrasts can come from different tissue properties, for example, magnetic susceptibility or the rate of diffusion of water molecules. The basic MRI contrasts are T1 -weighted images and T2-weighted images which are typically acquired in all imaging protocols. T1 -weighted images and T2-weighted images can be rapidly acquired, whereas other contrasts (e.g., diffusion-weighted, susceptibility-weighted, fluid-attenuated inversion recovery (FLAIR), etc.) are acquired with pulse sequences that require longer MRI scans. The overall duration of an MRI imaging exam can be shortened by synthetically generating additional image contrasts from T1 -weighted and T2-weighted images using artificial intelligence (Al). Existing technologies attempt to generate synthetic MRI contrasts from quantitative maps of tissue properties, but these are often noisy and cannot be efficiently acquired at image resolutions typically used in clinical settings. SUMMARY

[0004] One aspect of the present disclosure is directed towards a method. The method can include performing, by one or more processors, a scan using a magnetic resonance imaging (MRI) system. The method can include generating, by the one or more processors, based on signals from the scan, a T1 -weighted image and a T2-weighted image. The method can include generating, by the one or more processors, using a machine learning model, based on the T1 -weighted image and the T2-weighted image, at least one synthetic image contrast, wherein the T1 -weighted image and the T2-weighted image are input in the machine learning model, and the machine learning model outputs the at least one synthetic image contrast.

[0005] In various implementations, the machine learning model further include an encoder-decoder architecture in which a latent space is a deterministic output of the encoder. The machine learning model can include a generative model. The machine learning model can include a diffusion model. The diffusion model can include spatial conditioning controls. The synthetic image contrast can include at least one of fluid attenuated inversion recovery (FLAIR), susceptibility-weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phase-sensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

[0006] In various implementations, the method can include receiving, by the one or more processors, training data including a training T1 -weighted image, a training T2-weighted image, and a training synthetic image contrast. The method can include generating, by the one or more processors, using the machine learning model, a model synthetic image contrast based on the training T1 -weighted image and the training T2-weighted image. The method can include determining, by the one or more processors, at least one loss between the model synthetic image contrast and the training synthetic image contrast. The method can include updating, by the one or more processors, weights of the machine learning model based on the loss. The loss can be a mean squared error. At least one of the T1 -weighted image or the T2- weighted image can be a noise-corrupted image. The method can include predicting, by the one or more processors, conditional distributions of the noise-corrupted image. The method can include generating, by the one or more processors, a non-noise-corrupted image based on the conditional distributions. [0007] In various implementations, the method can include generating, by the one or more processors, based on the signals from the scan, a proton density image, wherein the synthetic image contrast is generated based on the T1 -weighted image, the T2-weighted image, and the proton density image. The machine learning model can include one or more channels and to input the plurality of T1 -weighted images and T2-weighted images into the machine learning model. The method can include combining, by the one or more processors, the T1 -weighted image and the T2-weighted image into a single image. The method can include inputting, by the one or more processors, the single image into the machine learning model. The method can include generating, by the one or more processors, based on the single image, the synthetic image contrast.

[0008] At least one aspect of the present disclosure is directed to a system. The system can include one or more processors. The one or more processors can be configured to receive one or more signals from a magnetic resonance imaging (MRI) system. The one or more processors can be configured to generate a T1 -weighted image and a T2-weighted image based on the one or more signals. The one or more processors can be configured to combine the T1 -weighted image and the T2-weighted image into a combined image. The one or more processors can be configured to generate, using a diffusion model, a synthetic image contrast, wherein the combined image is input into the diffusion model and the diffusion model outputs the synthetic image contrast.

[0009] In various implementations, the diffusion model can include at least one of a conditional diffusion, cross-domain attention diffusion, latent diffusion, or cycle consistent diffusion model. The one or more processors can be configured to generate a proton density image based on the one or more signals. The one or more processors can be configured to combine the proton density image, the T1 -weighted image, and the T2-weighted image into the combined image. The synthetic image contrast can include at least one of fluid attenuated inversion recovery (FLAIR), susceptibility-weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phase-sensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

[0010] At least one aspect of the present disclosure is directed to a system. The system can include one or more processors. The one or more processors can be configured to generate at least one of a T1 -weighted image, a T2-weighted image, or a proton density based on signals generated from a magnetic resonance imaging (MRI) scan. The one or more processors can be configured to generate, using a machine learning model, at least one synthetic image contrast by inputting the at least one of the T1 -weighted image, the T2- weighted image, or the proton density image into the machine learning model, the machine learning model outputting the at least one synthetic image contrast.

[0011] In various implementations, the one or more processors are further configured to generate, using the machine learning model, a model synthetic image contrast based on at least one of a training T1 -weighted image, a training T2-weighted image, or training proton density image. The one or more processors can be configured to determine at least one loss between the model synthetic image contrast and a training synthetic image contrast corresponding to the at least one of the training T1 -weighted image, the training T2-weighted image, or the proton density image. The one or more processors can be configured to update weights of the machine learning model based on the loss.

[0012] In various implementations, the training T1 -weighted image, the training T2- weighted image, and the proton density image can correspond to one MRI scan. The loss can be at least one of a mean squared error or a structural similarity index measure between the model synthetic image contrast and the training synthetic image contrast. The machine learning model can be at least one of a encoder-decoder model, a generative model, a diffusion model, an autoencoder model, or a convolutional neural network.

BRIEF DESCRIPTION OF THE FIGURES

[0013] The foregoing and other features of the present disclosure will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several implementations in accordance with the disclosure and are therefore not to be considered limiting of its scope, the disclosure will be described with additional specificity and detail through use of the accompanying drawings.

[0014] FIG. 1 is a schematic diagram of an example of a system for generating different synthetic image contrasts.

[0015] FIG. 2 is a flow diagram of an example of a method for generating different synthetic image contrasts. [0016] FIG. 3 is a flow diagram of an example of a method for generating different synthetic image contrasts.

[0017] FIG. 4 depicts example T1 -weighted, T2-weighted, and FLAIR images with their respective acquisition times.

[0018] FIG. 5 is a diagram of an example of a system for generating a synthetic image contrast (FLAIR) from T1 -weighted and T2-weighted images.

[0019] FIG. 6 is a diagram of an example of a system for generating a synthetic image contrast (FLAIR) from T1 -weighted and T2-weighted images.

[0020] Reference is made to the accompanying drawings throughout the following detailed description. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative implementations described in the detailed description, drawings, and claims are not meant to be limiting. Other implementations may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and made part of this disclosure.

DETAILED DESCRIPTION

[0021] Magnetic resonance imaging (MRI) is a medical imaging technique used to visualize internal structures in a body. MRI uses static magnetic fields to cause certain atoms in the body to align in a same direction and radio frequency waves to perturb atoms from an equilibrium position. As the atoms return to their original equilibrium position after turning off the radio frequency wave, radio frequency signals are detected and sent back to a computer where the computer converts the information into detailed contrast images. Contrast images highlight different tissues or structures in the body based off of radiofrequency signal intensity.

[0022] T1 -weighted and T2-weighted images are two types of contrast images produced by MRI examinations. Thus, T1 -weighted images provide contrast between different types of soft tissues by making tissues with short T1 relaxation times appear bright and long T1 relaxation times appear dark. T1 relaxation time is a characteristic property of atomic nuclei in a magnetic field. The T1 relaxation time is the time it takes for an atomic nucleus to return to approximately 63% of its original equilibrium position with respect to the static magnetic field generated by an MRI machine, after turning off the radiofrequency wave. T2-weighted images provides contrast between tissues with varying T2 relaxation times where short T2 relaxation times appear dark and long T2 relaxation times appear bright. The T2 relaxation time is another characteristic property of atomic nuclei in the magnetic field. The T2 relaxation time is the time it takes for an atomic nucleus to lose coherence to approximately 37% of the maximum value of an atomic nucleus’ coherence. The nucleus’ coherence refers to its ability to maintain a constant phase relationship with other nuclei which evidently decays due to interactions with nearby atoms and molecules.

[0023] MRI can include other image contrasts which use longer acquisition times compared to T1 -weighted and T2-weighted images. Additional image contrasts (e.g., advanced image contrasts) are used to provide additional information beyond information that the T1 -weighted and the T2-weighted images provide. Additional image contrasts can include fluid-attenuated inversion recovery (FLAIR), susceptibility-weighted imaging, diffusion- weighted imaging, perfusion-weighted imaging, and functional MRI. For example, FLAIR can selectively suppress signal from cerebrospinal fluid (CSF) to provide contrast between lesions, abnormalities, and surrounding tissues, in the brain. Particular image contrasts can measure microscopic motion of water, differences in magnetic susceptibility, passage of contrast agents through tissue, etc. T1 -weighted images and T2-weighted images are typically obtained in every MRI examination and can be gathered in under one minute. Other image contrasts can take significantly longer than T1 -weighted and T2-weighted images to obtain. Multiple image contrasts can be acquired as part of MRI protocol to evaluate changes in tissue such as edema, inflammation, blood, cyst, tumors, etc. Acquiring multiple image contrasts can increase the duration of the MRI examination.

[0024] Artificial intelligence (Al) is the simulation of human intelligence through computer processes. Machine learning is a subset of Al that involves algorithms and statistical models to perform tasks without explicit instructions. Machine learning includes supervised deep learning which is a technique to train machine learning models. Machine learning models can be trained on data to detect patterns, make decisions and predictions, and output products given feedback. Generative modeling is a type of machine learning model and determines patterns and probability distributions in a dataset to create new samples with similar characteristics to the given data. Some generative modeling techniques include autoencoders, variational autoencoders (VAEs), generative adversarial networks (GANs), PixelCNN, and PixelRNN. Generative modeling can be applied to image data to produce images based off of parameters and training of the generative model. For example, supervised deep learning is a subset of machine learning that utilizes neural networks with multiple layers to learn relationships in data with specific inputs and outputs. Supervised deep learning models iteratively adjust parameters to minimize error between the model’s predictions and the true output provided by the dataset.

[0025] Diffusion models are a class of generative models that gradually transforms a random noise distribution into a data distribution. Diffusion models can involve a series of steps that reverse a diffusion process. In the context of image-to-image translation, these models can convert an image from one modality to another by modeling the conditional distribution of target images given source images (e.g., receive an initial image and output another image). Specific examples of diffusion models include cross-domain attention diffusion, latent diffusion, and cycle consistent diffusion. Cross-domain attention diffusion incorporates attention mechanisms into diffusion models to better handle cross-domain image translation, such as translating between modalities that have different characteristics. Crossdomain attention diffusion models use attention to focus on relevant features in the source domain while generating the corresponding image in the target domain which enhances the model's ability to capture complex cross-domain relationships. Latent diffusion models operate in a compressed latent space rather than directly in the pixel space which can be advantageous for handling high-dimensional data like medical images. By transforming images into a lower-dimensional latent space before diffusion, these models can efficiently learn to translate between image modalities with potentially reduced computational costs and improved handling of complex image structures. Cycle consistent diffusion models extends the idea of cycle consistency, commonly used in non-diffusion translation models (e.g., CycleGAN), to the framework of diffusion models. Cycle consistent diffusion models ensure that an image from the original modality can be translated to another modality and then back again with minimal loss, enhancing the robustness and accuracy of the translation between modalities such as different types of scans. Diffusion models can be controlled by adding spatial conditioning controls, for example, additional input images (e.g., as in ControlNet). [0026] Implementations described herein relate generally to generating synthetic image contrasts from T1 -weighted and T2-weighted MR images using machine learning models. The present disclosure aims to address the problem of slow MRI exams by using Al to generate synthetic image contrasts (e.g., FLAIR, susceptibility- weighted, etc.) from basic Tl- weighted and T2-weighted images. The present disclosure could significantly reduce the duration of MRI protocols and have a significant impact on clinical workflow. For example, scan times for MRIs of the brain could be reduced from 8-10 minutes to 2 minutes or less.

Deep Learning for Synthetic Image Generator Models

[0027] FIG. 1 depicts a system 100 of generating synthetic image contrasts such as, but not limited to, advanced image contrasts. The synthetic image contrast can include at least one of a fluid attenuated inversion recovery (FLAIR), susceptibility-weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phasesensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

[0028] The system 100 can include one or more processors 102 and memory 104, which can be implemented as one or more processing circuits. The processor 102 may be a general purpose or specific purpose processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable processing components. The processor 102 may be configured to execute computer code or instructions stored in memory (e.g., fuzzy logic, etc.) or received from other computer readable media (e.g., CDROM, network storage, a remote server, etc.) to perform one or more of the processes described herein. The memory 104 may include one or more data storage devices (e.g., memory units, memory devices, computer-readable storage media, etc.) configured to store data, computer code, executable instructions, or other forms of computer- readable information. The memory 104 may include random access memory (RAM), readonly memory (ROM), hard drive storage, temporary storage, non-volatile memory, flash memory, optical memory, or any other suitable memory for storing software objects and/or computer instructions. The memory 104 may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. The memory 104 may be communicab ly connected to the processor 102 and may include computer code for executing (e.g., by processor 102) one or more of the processes described herein. The memory 104 can include various modules (e.g., circuits, engines) for completing processes described herein. The one or more processors 102 and memory 104 may include various distributed components that may be communicatively coupled by wired or wireless connections; for example, various portions of system 100 may be implemented using one or more client devices remote from one or more server devices. The system 100 can include any one or more rules, heuristics, logic, code, functions, machine learning models, neural networks, algorithms, or various combinations thereof to implement one or more components of the system 100, such as any one or more of model trainer 106, training data 108, image generator 110, and models 112. The system 100 and/or various components thereof can execute various operations described herein and/or combinations thereof as one or more tasks. For example, the model trainer 106 can cause the processor 102 to execute a training task; the image generator 110 can cause the processor 102 to execute an image generation task.

[0029] The system 100 can include at least one model trainer 106. The model trainer 106 can be used to train various machine learning models as described herein (e.g., models 112), such as to provide training data as input to a machine learning model, cause the machine learning model to generate estimated outputs responsive to the inputs, and update the machine learning model (e.g., one or more parameters of the machine learning model) according to an evaluation of the estimated outputs. For example, the model trainer 106 can perform unsupervised and/or self-supervised and/or supervised learning processes to update the machine learning model. For example, the model trainer 106 can use any of various objective functions and/or cost functions to evaluate the estimated outputs of the machine learning model, and techniques including but not limited to gradient descent to perform the updating of the machine learning model.

[0030] The system 100 can include training data 108. The system 100 can retrieve the training data 108 from and/or store the training data 108 in one or more data sources that can be maintained by the system 100 and/or be remote from the system 100.

[0031] The training data 108 can include or be retrieved from one or more data sources, such as data sources that include data structures that represent T1 -weighted, T2-weighted, other image contrasts (e.g., advanced image contrasts), and proton density images. For example, the data structure can include a plurality of sequence elements, each sequence element corresponding to at least one of a corresponding T1 -weighted, T2-weighted, other image contrasts, and proton density images from a singular MRI examination. The training data 108 can include any number of MRI exams and corresponding image contrasts spanning a wide range of pathology and patient demographics. For example, the training data 108 can include MRI exams for a knee and a brain of various patients and corresponding images.

[0032] The image contrasts included in the training data 108 can be generated by various scans performed during the MRI exam. For example, to obtain a FLAIR image, the MRI system can apply an inversion pulse, and generate the FLAIR image based on the signals received from the inversion pulse. The image contrasts can be generated using specific scans to generate the image contrasts included in the training data 108.

[0033] The training data 108 can include input features and corresponding target outputs. The input features can include at least one of T1 -weighted, T2-weighted, or proton density images. Target outputs can include other corresponding image contrasts obtained during the MRI examination. The target outputs can correspond to the input features based on the MRI examinations the images were generated based on. The training data 108 can be preprocessed. For example, the training data 108 can be cleaned to remove missing values. The training data 108 can be normalized to standardize the data to have consistent scale and distribution. The training data 108 can be included in or provided to the model trainer 106 for the model trainer 106 to use to train models.

[0034] The model trainer 106 can apply supervised deep learning to generative models to perform image-to-image synthesis to generate synthetic image contrasts from T1 -weighted and T2-weighted images. The model trainer 106 can apply self-supervised deep learning to generative models to perform image-to-image synthesis to generate synthetic image contrasts from T1 -weighted and T2-weighted images. The model trainer 106 can apply unsupervised deep learning to generative models to perform image-to-image synthesis to generate different synthetic image contrasts from T1 -weighted and T2-weighted images. For example, the model trainer 106 can train the generative model to cause the generative model to learn patterns and probability distributions within a dataset. The model trainer 106 can train the generative model to iteratively minimize difference between the generative model’s predicted output and the target output of the training data 108 by adjusting the generative model’s parameters. The model trainer 106 can update weights of the generative model based off a calculated mean squared error between the output synthetic image and the target image. The weights can be updated iteratively to minimize the mean squared error and can be adjusted by an optimization algorithm. The model trainer 106 can employ a neural network (e.g., the optimization algorithm) to optimize the generative model to make accurate predictions of additional image contrasts based off the corresponding T1 -weighted images and T2-weighted images in the training data 108. The model trainer 106 can use varying algorithms and methods to train the generative models based off of the parameters of the MRI examinations and desired image contrasts. The model trainer 106 can adjust supervised deep learning parameters to train the generative models based off of desired target output and MRI examination parameters.

[0035] The system 100 can include one or more models 112, such as machine models, machine learning models, encoder-decoder models, and/or generative models. The model 112 can be VAEs. The models 112 can be diffusion models. The models 112 can include conditional diffusion models, cross-domain attention diffusion, latent diffusion, and cycle consistent diffusion models. The models 112 can transform a random noise distribution to a data distribution by reversing the diffusion process. The models 112 can be controlled by adding spatial conditioning controls, for example, additional input images (e.g., as in ControlNet). The spatial conditioning control can guide the output of the models 112, such as instructing the model 112 controlling semantic layout of the synthetic image contrast. The models 112 can convert an image from one modality to another by modeling the conditional distribution of target images given source images.

[0036] In various implementations, the models 112 can be convolutional neural network (CNN), such as a U-net model. The models 112 can be a masked autoencoder (MAE) model. The models 112 can be a combination of the U-net model and the MAE model. In various implementations, the model trainer 106 can receive normalized T1 -weighted and T2- weighted images from the training data 108, combine the images into a two-channel input, and input the images into the model 112. The model trainer 106 can input the images into the MAE model and the U-net model simultaneously (e.g., in parallel, etc.). Outputs of the MAE model and the U-net model can be fused (e.g., combined, etc.). The model 112 can include a decoder, and the decoder can receive the fused outputs and output a synthetic advanced image contrast. The model trainer 106 can determine a loss based on a difference between the synthetic advanced image contrast, and the advanced image contrast corresponding to the training T1 -weighted images and T2- weighted images in the training data 108. The loss can include at least one of a mean squared error or a structural similarity index measure loss (SSIM), and can update weights of the model 112 based on the loss. [0037] The model 112 can start with a noise-corrupted version of the target image contrast and iteratively refine the image over multiple steps. For example, signals provided by the MRI scan can include noise, and the image contrasts can be generated with noise. The model 112 can predict conditional distributions iteratively to gradually transform the original noise- corrupted image into a high-quality image. The model 112 can also start with a non-noise corrupted version of the target image contrast. The model 112 can then generate the synthetic image contrast with standard spatial resolution (e.g., the high-quality image) given Tl- weighted and T2-weighted images and, in some implementations, proton density images. The models 112 can be models that have been trained by the model trainer 106 on the training data 108.

[0038] The model 112 can include a model input with one or more channels. The model input can be represented as a tensor. The tensor can include different aspects of the training data 108 such as batch size, height, and width of the image data. The tensor can include the T1 -weighted image, the T2-weighted image, and the proton density image. The T1 -weighted and T2-weighted images can be stacked and input into the model 112. The Tl-weighted and T2-weighted images can be combined into a single image and input into the model 112. The Tl-weighted, T2-weighted images, and the proton density images can be combined into a single image and input into the model 112. The model 112 can be configured based on initial Tl-weighted and T2-weighted images to generate the synthetic image contrast, for example, an advanced synthetic image contrast. The model 112 can include an encoder-decoder architecture in which a latent space is a deterministic output of the encoder. For example, the latent space can include encoded features or patterns of the single image.

Generative Model-Based Different Synthetic Image Contrast Generation

[0039] Referring further to FIG. 1, the system 100 can include the image generator 110. The image generator 110 can generate images, such as to predict one or more image contrasts (e.g., given at least the corresponding Tl-weighted and T2-weighted images). For example, the image generator 110 can be or include one or more models 112 which can generate one or more image contrasts, such as in response to receiving an indication of the Tl-weighted, T2-weighted, and/or proton density images.

[0040] The image generator 110 can generate synthetic image contrasts using the models 112. The image generator 110 can receive the model 112 from the model trainer 106 following convergence of weights of the model 112. The image generator 110 can translate between differing image modalities with different characteristics. Image modalities can differ based off of the type of scan performed by the MRI. The image generator 110 can generate synthetic image contrasts based off initial T1 -weighted, T2-weighted, and/or proton density images. The initial images can be T1 -weighted, T2-weighted, and/or proton density images generated based on signals generated during the MRI scan. The image generator 110 can adjust its parameters based off of parameter information of the MRI examination and a desired type of synthetic image contrast (e.g., FLAIR, susceptibility-weighted, diffusion-weighted, etc.).

[0041] FIG. 2 is a flow diagram of a method 200 for generating synthetic image contrasts. The method 200 can be performed using various systems described herein. Various steps in the method 200 may be repeated, omitted, performed in various orders, or otherwise modified. Various steps in the method 200 may be run concurrently, in parallel, or individually.

[0042] At 202, the method 200 can include training models on training data (e.g., training data 108 with supervised deep learning). In some implementations, the models can be trained via self-supervised and/or unsupervised deep learning. The models can include generative models with supervised deep learning to generate synthetic image contrasts.

[0043] At 204, the method 200 can include generating synthetic image contrasts using the trained models. The synthetic image contrasts are generated based on the latest weights of the model (e.g., the generative model). The system 100, can generate, by one or more processors 102 using a machine model (e.g., the model 112), based on initial T1 -weighted, T2-weighted, and/or proton density images and the machine learning model (e.g., the model 112 configured based on deep learning), a synthetic image contrast. 204 can include generating a plurality of synthetic image contrasts based on a plurality of MRI examinations with a corresponding plurality of T1 -weighted, T2-weighted, and/or proton density images. 204 can also include adjusting the T1 -weighted and T2-weighted images to have a same image resolution and align the T1 -weighted and T2-weighted images.

[0044] FIG. 3 is a flow diagram of a method 300 for generating synthetic image contrasts. The method 300 can be performed using various systems described herein. Various steps in the method 300 may be repeated, omitted, performed in various orders, or otherwise modified. Various steps in the method 300 may be run concurrently, in parallel, or individually. [0045] At 302, the method 300 can include performing a scan. The scan can be performed using an MRI system. The scan can be performed on a body part of a patient, such as a brain, knee, ankle, or another body part of the patient.

[0046] At 304, the method 300 can include generating a T1 -weighted image and a T2- weighted image. The T1 -weighted image and the T2-weighted image can be generated based on signals from the scan. In situations where at least one of the T1 -weighted image or the T2- weighted image is a noise-corrupted image, the method 300 can include predicting conditional distributions of the noise-corrupted image. The method 300 can include generating a non-noise-corrupted image based on the conditional distributions. The method 300 can include generating based on the signals from the scan, a proton density image. In various implementations, at least one of the T1 -weighted image and the T2-weighted image can be generated in parallel with performing the scan.

[0047] At 306, the method 300 can include generating at lest one synthetic image contrast. The synthetic image contrast can be generated using the T1 -weighted image, the T2-weighted image, and a machine learning model. The T1 -weighted image and the T2-weighted image can be input in the machine learning model, and the machine learning model can output the at least one synthetic image contrast. The machine learning model can include an encoderdecoder architecture in which a latent space is a deterministic output of the encoder. The machine learning model can include a generative model. The machine learning model can include a diffusion model. The diffusion model can include spatial conditioning controls. The synthetic image contrast can include at least one of fluid attenuated inversion recovery (FLAIR), susceptibility-weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phase-sensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

[0048] In various implementations, the synthetic image contrast can be generated based on the T1 -weighted image, the T2-weighted image, and the proton density image. The machine learning model can include one or more channels and to input the plurality of Tl- weighted images and T2-weighted images into the machine learning model, the method 300 can include combining the T1 -weighted image and the T2-weighted image into a single image. The method 300 can include inputting the single image into the machine learning model. The method 300 can include generating, based on the single image, the synthetic image contrast. [0049] In various implementations, the method 300 can include receiving training data including a training T1 -weighted image, a training T2-weighted image, and a training synthetic image contrast. The method 300 can include generating, using the machine learning model, a model synthetic image contrast based on the training T1 -weighted image and the training T2-weighted image. The method 300 can include determining at least one loss between the model synthetic image contrast and the training synthetic image contrast. The method 300 can include updating weights of the machine learning model based on the loss. The loss can be a mean squared error.

Examples

[0050] The system 100 (e.g., the supervised deep learning technique for generating synthetic image contrasts via generative models), can be demonstrated by synthesizing a FLAIR contrast based on T1 -weighted and T2-weighted images of the brain as seen in FIG. 4 and FIG. 5.

[0051] FIG. 4 depicts example T1 -weighted, T2-weighted, and FLAIR images obtained from an MRI examination. FIG. 4 also depicts acquisition times for each image contrast. The images of FIG. 4 can be included in the training data 108.

[0052] FIG. 5 depicts an example system 500 for generating advanced synthetic image contrasts. In various implementations, the system 500 is an example of the system 100. In various implementations, the system 100 includes the system 500. In various implementations, the system 500 includes the system 100.

[0053] The system 500 can include at least one image receiver 502. The image receiver 502 can receive images 504. The image receiver 502 can receive images 504 from, for example, training data 108. The images 504 can include at least corresponding Tl-weighted, T2-weighted, and synthetic contrast images. The image receiver 502 can register the received images 504. For example, the image receiver 502 can align the images 504 into a common coordinate space such that the images 504 are aligned for pixel-to-pixel comparison. As another example, the T2-weighted image is rotated to match a rotation of the Tl-weighted image.

[0054] The image receiver 502 can normalize the received images 504. In various implementations, the image receiver 502 can normalize the received images 504 prior to, in parallel with, or after registering the images 504. The image receiver 502 can adjust pixel intensity values such that the received images 504 are within a same range or distribution to normalize the images. The image receiver 502 can normalize the images 504 by min-max scaling (e.g., rescaling pixels to [0, 1] range, etc.), Z-score (e.g., subtract mean, divide by standard deviation, etc.), or histogram equalization (e.g., enhance contrast, etc.).

[0055] The system 500 can include at least one of the model trainer 106. The model trainer 106 can include the model 112, which can be a conditional diffusion model. The model 112 can be an image-to-image translation model, such as Palette which can be implemented by Pytorch. The model trainer 106 can receive the registered and normalized images 504 from the image receiver 502. The model trainer 106 can combine (e.g., stack, etc.) Tl-weightd and T2-weighted images 505. The T1 -weighted and T2-weighted images 505 can be included in the images 504. The model trainer 106 an input the combined images 504 into the model 112, and the model 112 can output a synthetic image contrast 506. Following output of the synthetic image contrast 506, the model trainer 106 can compare the synthetic image contrast 506 with a target synthetic image contrast 508. The target synthetic image contrast 508 can be included in the images 504, and can correspond to the T1 -weighted image and the T2- weighted image input into the model 112.

[0056] The model trainer 106 can determine at least a mean squared error between the synthetic image contrast 506 and the target synthetic image contrast 508. Based on the mean square error, the model trainer 106 can determine at least one loss. The model trainer 106 can use the loss to update weights of the model 112. The weights of the model 112 can be updated to minimize the mean squared error between the synthetic image contrast 506 and the target synthetic image contrast 508.

[0057] The system 500 can include at least one of the image generator 110. The model trainer 106 can update weights of the model 112 until convergence, and the image generator 110 can receive the model 112 from the model trainer 106. The image generator 110 can receive images 510. The images 510 can be images generated from an MRI scan, and may not be included in the images 504. In various implementations, images 510 are included in images 504. The model 112 can receive the images 510 as an input, and the images 510 can be combined prior to input into the model 112. The model 112 can generate the synthetic image contrast 506. The synthetic image contrast 506 can be at least one of a FLAIR, susceptibility -weighted, STIR, diffusion-weighted, DIR, PSIR, or MTC image. [0058] FIG. 6 is an example system 600 for generating synthetic image contrasts. The system 600 can be an example of the system 100. In various implementations, the system 600 is an alternative to the system 500. In various implementations, the system 100 includes the system 500. In various implementations, the system 500 includes the system 100.

[0059] The system 600 can include at least one of the image receiver 502, at least one of the model trainer 106, and at least one of the image generator 110. In the system 600, the model 112 can include a combination of an MAE and U-net models. The model 112 can include at least one MAE 602 and at least one U-net 604. The model 112 can receive the combined Tl-weighted and T2-weighted images 505 as an input into the MAE 602 and the U-net 604. The MAE 602 and the U-net 604 can receive images 505 simultaneously or in parallel.

[0060] The model 112 can include at least one U-net bottleneck 606. Outputs of the MAE 602 and the U-net 604 can be provided to a U-net bottleneck 606. Outputs of the MAE 602 and the U-net 604 can be encoded. The output of the U-net 605 can be a segmentation map. The segmentation map can segment images 505 according to a classification, such as tumor and background. The output of the MAE 602 can be a reconstructed image of the images 505 which can include predictions for missing (e.g., masked, etc.) pixels. The U-net bottleneck 606 can combine the outputs, and can extract features of the output. The features extracted by the U-net bottleneck 606 can represent a context of the output. For example, the context can include a feature map of images 505. The feature map of the images 505 can indicate features, such as edges, textures, and shapes within the image 505.

[0061] The model 112 can include at least one decoder 608. The U-net bottleneck 606 can provide an output to the decoder 608. The decoder 608 can upsample the output of the U- net bottleneck 606 to a higher special resolution, concatenate features, and apply convolution layers to generate the synthetic image contrast 506.

Definitions.

[0062] Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements can be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

[0063] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including” “comprising” “having” “containing” “involving” “characterized by” “characterized in that” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0064] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular can also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein can also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element can include implementations where the act or element is based at least in part on any information, act, or element.

[0065] Any implementation disclosed herein can be combined with any other implementation or implementation, and references to “an implementation,” “some implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation can be included in at least one implementation or implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation can be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0066] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements. [0067] Systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. Further relative parallel, perpendicular, vertical or other positioning or orientation descriptions include variations within +/-10% or +/-10 degrees of pure vertical, parallel or perpendicular positioning. References to “approximately,” “about” “substantially” or other terms of degree include variations of +/- 10% from the given measurement, unit, or range unless explicitly indicated otherwise. Coupled elements can be electrically, mechanically, or physically coupled with one another directly or with intervening elements. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.

[0068] The term “coupled” and variations thereof includes the joining of two members directly or indirectly to one another. Such joining may be stationary (e.g., permanent or fixed) or moveable (e.g., removable or releasable). Such joining may be achieved with the two members coupled directly with or to each other, with the two members coupled with each other using a separate intervening member and any additional intermediate members coupled with one another, or with the two members coupled with each other using an intervening member that is integrally formed as a single unitary body with one of the two members. If “coupled” or variations thereof are modified by an additional term (e.g., directly coupled), the generic definition of “coupled” provided above is modified by the plain language meaning of the additional term (e.g., “directly coupled” means the joining of two members without any separate intervening member), resulting in a narrower definition than the generic definition of “coupled” provided above. Such coupling may be mechanical, electrical, or fluidic.

[0069] The present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations. The implementations of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Implementations within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine- readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machineexecutable instructions or data structures, and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0070] Although the figures show a specific order of method steps, the order of the steps may differ from what is depicted. Also, two or more steps may be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps.

[0071] In various implementations, the steps and operations described herein may be performed on one processor or in a combination of two or more processors. For example, in some implementations, the various operations could be performed in a central server or set of central servers configured to receive data from one or more devices (e.g., edge computing devices/controllers) and perform the operations. In some implementations, the operations may be performed by one or more local controllers or computing devices (e.g., edge devices), such as controllers dedicated to and/or located within a particular building or portion of a building. In some implementations, the operations may be performed by a combination of one or more central or offsite computing devices/servers and one or more local controllers/computing devices. All such implementations are contemplated within the scope of the present disclosure. Further, unless otherwise indicated, when the present disclosure refers to one or more computer-readable storage media and/or one or more controllers, such computer-readable storage media and/or one or more controllers may be implemented as one or more central servers, one or more local controllers or computing devices (e.g., edge devices), any combination thereof, or any other combination of storage media and/or controllers regardless of the location of such devices.

[0072] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Such references used in conjunction with “comprising” or other open terminology can include additional items.

[0073] Modifications of described elements and acts such as variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, orientations can occur without materially departing from the teachings and advantages of the subject matter disclosed herein. For example, elements shown as integrally formed can be constructed of multiple parts or elements, the position of elements can be reversed or otherwise varied, and the nature or number of discrete elements or positions can be altered or varied. Other substitutions, modifications, changes and omissions can also be made in the design, operating conditions and arrangement of the disclosed elements and operations without departing from the scope of the present disclosure.

[0074] References herein to the positions of elements (e.g., “top,” “bottom,” “above,” “below”) are merely used to describe the orientation of various elements in the FIGURES. It should be noted that the orientation of various elements may differ according to other exemplary implementations, and that such variations are intended to be encompassed by the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: performing, by one or more processors, a scan using a magnetic resonance imaging (MRI) system; generating, by the one or more processors, based on signals from the scan, a Tl- weighted image and a T2-weighted image; and generating, by the one or more processors, using a machine learning model, based on the T1 -weighted image and the T2-weighted image, at least one synthetic image contrast, wherein the T1 -weighted image and the T2-weighted image are input in the machine learning model, and the machine learning model outputs the at least one synthetic image contrast.

2. The method of claim 1, wherein the machine learning model further comprises an encoder-decoder architecture in which a latent space is a deterministic output of the machine learning model.

3. The method of claim 1, wherein the machine learning model comprises a generative model.

4. The method of claim 1, wherein the machine learning model comprises a diffusion model.

5. The method of claim 4, wherein the diffusion model includes spatial conditioning controls.

6. The method of claim 1, the synthetic image contrast includes at least one of fluid attenuated inversion recovery (FLAIR), susceptibility -weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phase-sensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

7. The method of claim 1, further comprising: receiving, by the one or more processors, training data comprising a training Tl- weighted image, a training T2-weighted image, and a training synthetic image contrast; generating, by the one or more processors, using the machine learning model, a model synthetic image contrast based on the training T1 -weighted image and the training T2-weighted image; determining, by the one or more processors, at least one loss between the model synthetic image contrast and the training synthetic image contrast; and updating, by the one or more processors, weights of the machine learning model based on the loss.

8. The method of claim 6, wherein the loss is a mean squared error.

9. The method of claim 1, wherein at least one of the T1 -weighted image or the T2- weighted image is a noise-corrupted image, the method further comprising: predicting, by the one or more processors, conditional distributions of the noise- corrupted image; and generating, by the one or more processors, a non-noise-corrupted image based on the conditional distributions.

10. The method of claim 1, wherein the method further comprises generating, by the one or more processors, based on the signals from the scan, a proton density image, wherein the synthetic image contrast is generated based on the Tl-weighted image, the T2-weighted image, and the proton density image.

11. The method of claim 1, wherein the machine learning model includes one or more channels and to input the of Tl-weighted image and T2-weighted image into the machine learning model, the method further comprises: combining, by the one or more processors, the Tl-weighted image and the T2- weighted image into a single image; inputting, by the one or more processors, the single image into the machine learning model; and generating, by the one or more processors, based on the single image, the synthetic image contrast.

12. A system, comprising: one or more processors configured to: receive one or more signals from a magnetic resonance imaging (MRI) system; generate a T1 -weighted image and a T2-weighted image based on the one or more signals; combine the T1 -weighted image and the T2-weighted image into a combined image; and generate, using a diffusion model, a synthetic image contrast, wherein the combined image is input into the diffusion model and the diffusion model outputs the synthetic image contrast.

13. The system of claim 12, wherein the diffusion model comprises at least one of a conditional diffusion, cross-domain attention diffusion, latent diffusion, or cycle consistent diffusion model.

14. The system of claim 12, the one or more processors further configured to: generate a proton density image based on the one or more signals; and combine the proton density image, the T1 -weighted image, and the T2-weighted image into the combined image.

15. The system of claim 12, wherein the synthetic image contrast includes at least one of fluid attenuated inversion recovery (FLAIR), susceptibility -weighted, short tau inversion recovery (STIR), diffusion-weighted, double inversion recovery (DIR), phase-sensitive inversion recovery (PSIR), or magnetization transfer contrast (MTC) image.

16. A system, comprising one or more processors configured to: generate at least one of a T1 -weighted image, a T2-weighted image, or a proton density based on signals generated from a magnetic resonance imaging (MRI) scan; and generate, using a machine learning model, at least one synthetic image contrast by inputting the at least one of the T1 -weighted image, the T2-weighted image, or the proton density image into the machine learning model, the machine learning model outputting the at least one synthetic image contrast.

17. The system of claim 16, the one or more processors further configured to: generate, using the machine learning model, a model synthetic image contrast based on at least one of a training T1 -weighted image, a training T2-weighted image, or training proton density image; determine at least one loss between the model synthetic image contrast and a training synthetic image contrast corresponding to the at least one of the training Tl- weighted image, the training T2-weighted image, or the proton density image; and update weights of the machine learning model based on the loss.

18. The system of claim 17, wherein the training T1 -weighted image, the training T2- weighted image, and the proton density image correspond to one MRI scan.

19. The system of claim 17, wherein the loss is at least one of a mean squared error or a structural similarity index measure between the model synthetic image contrast and the training synthetic image contrast.

20. The system of claim 16, wherein the machine learning model is at least one of a encoder-decoder model, a generative model, a diffusion model, a masked autoencoder, or a convolutional neural network.