WO2022126614A1

WO2022126614A1 - Manifold optimization-based deep learning method for dynamic magnetic resonance imaging

Info

Publication number: WO2022126614A1
Application number: PCT/CN2020/137655
Authority: WO
Inventors: 梁栋; 朱燕杰; 柯子文; 崔卓须; 刘新; 郑海荣
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2020-12-18
Filing date: 2020-12-18
Publication date: 2022-06-23

Abstract

A manifold optimization-based deep learning method and apparatus for dynamic magnetic resonance (MR) imaging, a device, and a storage medium. The method comprises: creating a fixed rank-based manifold space, and deploying an entire optimization process into a neural network to obtain a manifold optimization-based deep model (110); for a dynamic MR image in which there is an interrelation between frames, constructing an image reconstruction model on a nonlinear manifold space (120); designing an iterative reconstruction algorithm on a corresponding manifold (130); and deploying the iterative reconstruction algorithm as a deep neural network (140). The solution avoids a complicated parameter adjustment process, and greatly shortens the reconstruction time; in addition, an original complicated nonlinear optimization process in a linear space is converted into a linear optimization process in a manifold space, and the reconstruction performance is expected to be further improved.

Description

Deep learning method for magnetic resonance dynamic imaging based on manifold optimization

technical field

The present invention relates to the technical field of magnetic resonance imaging, in particular to a deep learning method, apparatus, device and storage medium for magnetic resonance dynamic imaging based on manifold optimization.

Background technique

Dynamic MR Imaging (dMRI) is a non-invasive imaging technique, such as cardiac cine imaging, which can be used to evaluate cardiac function, abnormal ventricular wall motion, etc., providing rich information for clinical diagnosis of the heart. However, due to the limitations of magnetic resonance physics and hardware, and the duration of cardiac motion cycles, magnetic resonance cardiac cine imaging is often limited in time and space resolution, and cannot accurately assess cardiac function in some cardiac diseases, such as arrhythmia. Therefore, it is particularly important to improve the temporal and spatial resolution of magnetic resonance cardiac cine imaging by using fast imaging methods under the premise of ensuring imaging quality.

At present, traditional parallel imaging or compressed sensing technologies do not utilize big data priors, and this iterative optimization method is often time-consuming and difficult to select parameters. The deep learning-based neural network methods (DC-CNN, CRNN, DIMENSION) can avoid the iterative solution step and speed up the reconstruction time. However, such methods are generally constructed in linear Euclidean (image) space, so it is difficult to accurately describe the inherent nonlinear and non-local dependencies of images, which limits the improvement of reconstruction performance.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a deep learning method, apparatus, device and storage medium for magnetic resonance dynamic imaging based on manifold optimization.

In a first aspect, an embodiment of the present application provides a deep learning method for magnetic resonance dynamic imaging based on manifold optimization, the method comprising:

Establish a popular space based on a fixed rank, and expand a dynamic optimization process in the popular space, and obtain a popular optimization-based deep model by expanding the entire optimization process into a neural network;

For dynamic MR images with interrelationships between frames, an image reconstruction model is constructed on the nonlinear manifold space;

Design an iterative reconstruction algorithm on the corresponding manifold;

Expand into deep neural networks.

In one of the embodiments, the constructing the image reconstruction model comprises:

The model is designed using the manifold space metric.

In one embodiment, the designing an iterative reconstruction algorithm on the corresponding manifold includes:

The iterative reconstruction algorithm is designed according to the manifold structure, that is, the gradient of the objective function is calculated in the tangent vector space and iteratively updated along the manifold geodesic in the direction of the negative gradient.

In one embodiment, the expanding into a deep neural network includes:

Replace the corresponding operator or iterative rule in the iterative reconstruction algorithm on the corresponding manifold with the network module;

Train the loaded neural network module;

Learn the prior information contained in the data itself from the training data.

In a second aspect, an embodiment of the present application further provides a deep learning device for magnetic resonance dynamic imaging based on manifold optimization, the device comprising:

The establishment unit is used to establish a popular space based on a fixed rank, and expand a dynamic optimization process in the popular space, and obtain a depth model based on popular optimization by expanding the entire optimization process into a neural network;

The construction unit is used to construct the image reconstruction model on the nonlinear manifold space for the dynamic MR image with the relationship between the frames;

A design unit for designing an iterative reconstruction algorithm on the corresponding manifold;

Unfolding units for unwinding into deep neural networks.

The model is designed using the manifold space metric.

In one embodiment, the expanding into a deep neural network includes:

The replacement unit is used to replace the corresponding operator or iterative rule in the iterative reconstruction algorithm on the corresponding manifold with the network module;

The training unit is used to train the loaded neural network module;

The learning unit is used to learn the prior information contained in the data itself from the training data.

In a third aspect, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the program as described in the present application when the processor executes the program. A method as in any one of the descriptions of the embodiments.

In a fourth aspect, an embodiment of the present application further provides a computer device, a computer-readable storage medium, on which a computer program is stored, and the computer program is used for: when the computer program is executed by a processor, the computer program is executed as described in the present application. A method as in any one of the descriptions of the embodiments.

Beneficial effects of the present invention:

The deep learning method for magnetic resonance dynamic imaging based on manifold optimization provided by the present invention, for dynamic MR images with obvious dependencies between frames, on the (adaptive) nonlinear manifold space obtained by learning , build a suitable image reconstruction model, design the corresponding iterative reconstruction algorithm on the manifold, and then expand it into a deep neural network to realize the design of a learning iterative reconstruction method that is conducive to characterizing the internal dependencies of the image, and is expected to further improve the reconstruction results.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 shows a schematic flowchart of a deep learning method for magnetic resonance dynamic imaging based on manifold optimization provided by an embodiment of the present application;

FIG. 2 shows a schematic flowchart of a deep learning method for magnetic resonance dynamic imaging based on manifold optimization provided by another embodiment of the present application;

FIG. 3 shows an exemplary structural block diagram of a deep learning apparatus 300 for magnetic resonance dynamic imaging based on manifold optimization according to an embodiment of the present application;

FIG. 4 shows an exemplary structural block diagram of a deep learning apparatus 400 for magnetic resonance dynamic imaging based on manifold optimization according to another embodiment of the present application;

FIG. 5 shows a schematic structural diagram of a computer system suitable for implementing the terminal device according to the embodiment of the present application.

Detailed ways

In order to make the above objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", " Back, Left, Right, Vertical, Horizontal, Top, Bottom, Inner, Outer, Clockwise, Counterclockwise, Axial , "radial", "circumferential" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying the indicated device or Elements must have a particular orientation, be constructed and operate in a particular orientation and are therefore not to be construed as limitations of the invention.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

In the present invention, unless otherwise expressly specified and limited, the terms "installed", "connected", "connected", "fixed" and other terms should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between the two elements, unless otherwise specified limit. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

In the present invention, unless otherwise expressly specified and limited, a first feature "on" or "under" a second feature may be in direct contact between the first and second features, or the first and second features indirectly through an intermediary touch. Also, the first feature being "above", "over" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is level higher than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature has a lower level than the second feature.

It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for the purpose of illustration only and do not represent the only embodiment.

Please refer to FIG. 1 . FIG. 1 shows a schematic flowchart of a deep learning method for magnetic resonance dynamic imaging based on manifold optimization provided by an embodiment of the present application.

As shown in Figure 1, the method includes:

Step 110, establish a popular space based on a fixed rank, and develop a dynamic optimization process in the popular space, and obtain a popular optimization-based deep model by expanding the entire optimization process into a neural network;

Step 120, constructing an image reconstruction model on the nonlinear manifold space for the dynamic MR images with the mutual relationship between the frames;

Step 130, designing an iterative reconstruction algorithm on the corresponding manifold;

Step 140, expand into a deep neural network.

Using the above technical solution, for dynamic MR images with obvious dependencies between frames, a suitable image reconstruction model is constructed on the (adaptive) nonlinear manifold space obtained by learning, and a corresponding image reconstruction model on the manifold is designed. The iterative reconstruction algorithm is then expanded into a deep neural network to realize the design of a learning iterative reconstruction method that is conducive to characterizing the internal dependencies of the image, and is expected to further improve the reconstruction results.

In some embodiments, constructing the image reconstruction model includes designing the model using a manifold space metric.

In some embodiments, designing an iterative reconstruction algorithm on the corresponding manifold includes: designing an iterative reconstruction algorithm according to the manifold structure, that is, calculating the gradient of the objective function in the tangent vector space, and iteratively updating along the manifold geodesic in the direction of negative gradient.

In some embodiments, please refer to FIG. 2, which is a schematic flowchart of a deep learning method for magnetic resonance dynamic imaging based on manifold optimization provided by another embodiment of the present application.

As shown in Figure 2, the expansion into a deep neural network includes:

Step 210, replacing the corresponding operator or iterative rule in the iterative reconstruction algorithm on the design corresponding manifold with a network module;

Step 220, train the mounted neural network module;

Step 230, learning the prior information contained in the data itself from the training data.

Specifically, the method of the present application first constructs a popular space based on a fixed rank, and expands the dynamic optimization process in this popular space. By expanding the entire optimization process into a neural network, we obtain a popular optimization-based deep model Manifold-Net . For dynamic MR images with obvious dependencies between frames, an appropriate image reconstruction model is constructed on the (adaptive) nonlinear manifold space obtained by learning, and an iterative reconstruction algorithm on the corresponding manifold is designed. According to this, it is expanded into a deep neural network to realize the design of a learning iterative reconstruction method that is conducive to characterizing the internal dependencies of images.

(1) Low-dimensional manifold representation method

Based on the quantity and quality of training data, different solutions for manifold representation are provided. When the amount of data is small and the quality is low, it is difficult to train redundant neural networks to represent manifolds, so a more "lightweight" representation model is considered. Specifically, the following two schemes are given. Scheme 1: directly use the existing manifold representation methods, such as the Laplace feature transformation and the kernel principal component analysis. Such methods have certain theoretical guarantees in theory, but unfortunately, the specific implementation strategy of such methods is to represent low-dimensional manifolds by intercepting part of the singular values and eigenvectors of a certain matrix, thus there is a certain loss of information. Option 2: Embed the dynamic MR image into an existing low-dimensional manifold by training a relatively redundant neural network, such as a fixed-rank matrix/tensor manifold space. The technical route of this method follows the rule of redundancy first and then low-dimensionality, which not only ensures that the "low-dimensional" prior is satisfied, but also tries to avoid information loss. On the other hand, when the training data is of sufficient quantity and high quality, the fully redundant parameterized representation can more accurately describe the information contained in the data. Therefore, plan 3 is considered: use deep neural networks to represent the necessary structures for designing iterative algorithms on manifold space, such as homeomorphic mapping from image space to manifold, tangent space on manifold, geodesics, etc., to fully mine the data itself information, adaptively select the "appropriate" manifold representation.

(2) Dynamic MR image reconstruction model and iterative algorithm design on manifold

In the past, in the reconstruction of dynamic MR image models based on low-dimensional manifolds, the "low-dimensionality" of the manifold is usually used as a regular term, and the designed iterative algorithm is still located in the image (Euclidean) space, using the metrics and iterative rules in the image space, its output Solutions often do not reflect well the dependencies portrayed by manifolds. This project considers designing dynamic MR image reconstruction models and corresponding iterative algorithms directly on the learned manifold space. Among them, the (variational) model design will use the manifold space metric, and the algorithm iterative design will follow the manifold structure, that is, the gradient of the objective function is calculated in the tangent vector space, and iteratively updated along the manifold geodesic in the direction of the negative gradient.

(3) Multi-space parallel reconstruction of dynamic MR image algorithm design

Combined with the relatively mature central distributed framework at home and abroad, that is, the computing target is allocated to each computing unit, and there is a central processor to aggregate the information obtained by each computing unit. In this project, the targets characterized by different properties or internal dependencies are allocated to different manifold spaces, and each computing unit calculates the computational targets in each space. The central processor is located in the image space, and realizes information transmission through the homeomorphic mapping from the image space to each manifold space. The central processor summarizes the calculation information in each manifold space (computing unit) and updates the overall image.

(4) The image reconstruction iterative algorithm is expanded into a deep neural network

Following the general rules of iterative algorithm expansion into neural network, some operators or iterative rules in the designed multi-space parallel reconstruction algorithm are replaced with network modules to achieve more adaptive iterative reconstruction. By training the loaded neural network module, the prior information contained in the data itself can be fully learned from the training data, and the image reconstruction process suitable for such data can be described more accurately. Furthermore, by leveraging the high-performance computing power of the Graphics Processing Unit (GPU), the resulting network is often more efficient than the original iterative algorithm. It is worth noting that, different from general distributed optimization algorithms, the central processor here summarizes the information of each computing unit in the form of network fusion to further improve work efficiency.

Further, referring to FIG. 3 , FIG. 3 shows an exemplary structural block diagram of a deep learning apparatus 300 for magnetic resonance dynamic imaging based on manifold optimization according to an embodiment of the present application.

As shown in Figure 3, the device includes:

The establishment unit 310 is used to establish a popular space based on a fixed rank, and develop a dynamic optimization process in the popular space, and obtain a depth model based on popular optimization by expanding the entire optimization process into a neural network;

The construction unit 320 is used for constructing an image reconstruction model on the nonlinear manifold space for the dynamic MR images with the mutual relationship between the frames;

a design unit 330, configured to design an iterative reconstruction algorithm on the corresponding manifold;

The expansion unit 340 is used to expand into a deep neural network.

Using the above device, for dynamic MR images with obvious dependencies between frames, a suitable image reconstruction model is constructed on the (adaptive) nonlinear manifold space obtained by learning, and a corresponding iteration on the manifold is designed. The reconstruction algorithm is then expanded into a deep neural network to realize the design of a learning iterative reconstruction method that is conducive to characterizing the internal dependencies of the image, and is expected to further improve the reconstruction results.

Further, please refer to FIG. 4 , which is an exemplary structural block diagram of a deep learning apparatus 400 for magnetic resonance dynamic imaging based on manifold optimization according to another embodiment of the present application.

As shown in Figure 4, the device includes:

A replacement unit 410, configured to replace the corresponding operator or iterative rule in the iterative reconstruction algorithm on the design corresponding manifold with a network module;

A training unit 420, configured to train the mounted neural network module;

The learning unit 430 is configured to learn the prior information contained in the data itself from the training data.

It should be understood that the units or modules described in the apparatuses 300-400 correspond to the respective steps in the methods described with reference to FIGS. 1-2. Therefore, the operations and features described above with respect to the method are also applicable to the apparatuses 300 - 400 and the units included therein, and will not be repeated here. The apparatuses 300-400 may be pre-implemented in a browser of the electronic device or other security applications, or may be loaded into the browser of the electronic device or its security application by means of downloading or the like. Corresponding units in the apparatuses 300-400 may cooperate with units in the electronic device to implement the solutions of the embodiments of the present application.

Referring to FIG. 5 below, it shows a schematic structural diagram of a computer system 500 suitable for implementing a terminal device or a server according to an embodiment of the present application.

As shown in FIG. 5, a computer system 500 includes a central processing unit (CPU) 501 which can be loaded into a random access memory (RAM) 503 according to a program stored in a read only memory (ROM) 502 or a program from a storage section 508 Instead, various appropriate actions and processes are performed. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504 .

The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc. ; and a communication section 509 including a network interface card such as a LAN card, a modem, and the like. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to FIGS. 1-2 may be implemented as computer software programs. For example, embodiments of the present disclosure include a manifold optimization-based deep learning method for magnetic resonance dynamic imaging comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising a method for executing a graph The program code of the method of 1-2. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 509 and/or installed from the removable medium 511 .

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units or modules involved in the embodiments of the present application may be implemented in a software manner, and may also be implemented in a hardware manner. The described unit or module may also be provided in the processor, for example, it may be described as: a processor includes a first sub-area generating unit, a second sub-area generating unit and a display area generating unit. Wherein, the names of these units or modules do not constitute a limitation on the units or modules themselves, for example, the display area generating unit may also be described as "used to generate unit of the display area of the text".

As another aspect, the present application also provides a computer-readable storage medium, and the computer-readable storage medium may be the computer-readable storage medium included in the aforementioned apparatus in the above-mentioned embodiments; computer-readable storage medium in the device. The computer-readable storage medium stores one or more programs used by one or more processors to execute the text generation method described in this application for a transparent window envelope.

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover the above technical features or Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in this application (but not limited to) with similar functions.

Claims

A deep learning method for magnetic resonance dynamic imaging based on manifold optimization, characterized in that the method comprises:

Establish a popular space based on a fixed rank, and expand a dynamic optimization process in the popular space, and obtain a popular optimization-based deep model by expanding the entire optimization process into a neural network;

For dynamic MR images with interrelationships between frames, an image reconstruction model is constructed on the nonlinear manifold space;

Design an iterative reconstruction algorithm on the corresponding manifold;

Expand into deep neural networks.
The deep learning method for magnetic resonance dynamic imaging based on manifold optimization according to claim 1, wherein the constructing the image reconstruction model comprises:

The model is designed using the manifold space metric.
The deep learning method for magnetic resonance dynamic imaging based on manifold optimization according to claim 1, wherein the designing an iterative reconstruction algorithm on the corresponding manifold comprises:

The iterative reconstruction algorithm is designed according to the manifold structure, that is, the gradient of the objective function is calculated in the tangent vector space and iteratively updated along the manifold geodesic in the direction of the negative gradient.
The deep learning method for magnetic resonance dynamic imaging based on manifold optimization according to claim 1, wherein the expanding into a deep neural network comprises:

Replace the corresponding operator or iterative rule in the iterative reconstruction algorithm on the corresponding manifold with the network module;

Train the loaded neural network module;

Learn the prior information contained in the data itself from the training data.
A deep learning device for magnetic resonance dynamic imaging based on manifold optimization, characterized in that the device comprises:

The establishment unit is used to establish a popular space based on a fixed rank, and expand a dynamic optimization process in the popular space, and obtain a depth model based on popular optimization by expanding the entire optimization process into a neural network;

The construction unit is used to construct the image reconstruction model on the nonlinear manifold space for the dynamic MR image with the relationship between the frames;

A design unit for designing an iterative reconstruction algorithm on the corresponding manifold;

Unfolding units for unwinding into deep neural networks.
The deep learning device for magnetic resonance dynamic imaging based on manifold optimization according to claim 5, wherein the constructing the image reconstruction model comprises:

The model is designed using the manifold space metric.
The deep learning device for magnetic resonance dynamic imaging based on manifold optimization according to claim 5, wherein the designing an iterative reconstruction algorithm on the corresponding manifold comprises:

The iterative reconstruction algorithm is designed according to the manifold structure, that is, the gradient of the objective function is calculated in the tangent vector space and iteratively updated along the manifold geodesic in the direction of the negative gradient.
The deep learning device for magnetic resonance dynamic imaging based on manifold optimization according to claim 5, wherein the expanding into a deep neural network comprises:

The replacement unit is used to replace the corresponding operator or iterative rule in the iterative reconstruction algorithm on the corresponding manifold with the network module;

The training unit is used to train the loaded neural network module;

The learning unit is used to learn the prior information contained in the data itself from the training data.
A computer device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, when the processor executes the program, any one of claims 1-4 is implemented. method described.
A computer-readable storage medium having a computer program stored thereon for:

The computer program, when executed by a processor, implements the method of any one of claims 1-4.