CN116797457A

CN116797457A - Method and system for simultaneously realizing super-resolution and artifact removal of magnetic resonance image

Info

Publication number: CN116797457A
Application number: CN202310576261.7A
Authority: CN
Inventors: 罗家佳; 左健为; 李晓伟; 冯飞; 张晨鑫; 孙秀丽; 王建六
Original assignee: Peking University
Current assignee: Peking University
Priority date: 2023-05-20
Filing date: 2023-05-20
Publication date: 2023-09-22
Anticipated expiration: 2043-05-20
Also published as: CN116797457B

Abstract

The invention provides a method and a system for simultaneously realizing super-resolution and artifact removal of magnetic resonance images, wherein magnetic resonance images are scanned to obtain a plurality of first-direction magnetic resonance image slices, interlayer distances among the plurality of magnetic resonance image slices are reduced through an interlayer super-resolution module, artifact stripes in the reconstructed first-direction magnetic resonance image slices are removed through an artifact removal module, the in-layer resolution of the artifact removed first-direction magnetic resonance image slices is improved through an in-layer super-resolution module, and a target magnetic resonance image is obtained according to reconstruction of the plurality of first-direction target magnetic resonance image slices. The method has the advantages that the optimization thought of the magnetic resonance image from whole to slice to whole is realized, the interlayer spacing between the magnetic resonance image slices is reduced in the optimization process, the artifact stripes of the magnetic resonance image slices are reduced, the resolution in the magnetic resonance image slices is improved in various modes, and the image quality of the magnetic resonance image is improved in all directions.

Description

Method and system for simultaneously realizing super-resolution and artifact removal of magnetic resonance image

Technical Field

The invention relates to the technical field of image processing, in particular to a method and a system for simultaneously realizing super-resolution and artifact removal of magnetic resonance images.

Background

Magnetic resonance imaging (Magnetic Resonance Imaging, MRI) is a non-invasive medical imaging technique that can provide high resolution three-dimensional images for aiding in the diagnosis and treatment of various diseases. However, achieving an improvement in MRI image quality under the limitations of clinical hardware equipment has been an important research direction in the field of medical imaging. Currently, the MRI image quality improving method mainly includes super resolution, artifact removal, noise removal, and the like.

MRI super-resolution technology is one of the most commonly used methods for enhancing the image quality of MRI images. MRI super-resolution is a technique that converts a low-resolution image into a high-resolution image, and can improve the definition and detail of the image. Clinically, if the interlayer spacing is to be reduced, the time of scanning must be increased, the time cost for the patient becomes very high, and the time of scanning is increased, so that the probability of movement of the patient is increased, and the risk of occurrence of artifacts is further increased. Therefore, how to reduce the interlayer spacing of MRI without changing the hardware limitations such as scan time and the like, and at the same time, improving the quality of MRI has not been an important issue that has not been solved well in clinic. Currently, MRI super-resolution techniques are mainly divided into two types, namely interpolation algorithms and reconstruction algorithms. Interpolation is a simple super-resolution technique that obtains a high resolution image by interpolating a low resolution image. The interpolation algorithm has the advantages of high calculation speed, but has the defects of unsatisfactory image quality, distortion, blurring and the like. The reconstruction algorithm is a more advanced super-resolution technique that obtains a high resolution image by reconstructing a low resolution image. The reconstruction algorithm has the advantage of improving the image quality, but has the disadvantage of large calculation amount and long calculation time.

Removal of artifacts and noise interference is also important for clinical MRI quality improvement. Artifacts in MRI images are caused by magnetic field inhomogeneities or magnetic field drift, and noise in MRI images is caused by magnetic resonance signal randomness, electronic noise, and other factors, and artifacts and noise interference affect image quality and diagnostic accuracy. Thus, de-artifacting, de-noising techniques are an important aspect of MRI image quality improvement. Currently, MRI degritting related techniques are mainly divided into two types: a physical model-based approach and a data-driven based approach. The physical model-based method describes the generation mechanism of the artifacts by establishing a mathematical model, so that the artifacts are removed. This method requires accurate measurement of physical parameters such as the distribution of the magnetic field and the drift of the magnetic field, and is therefore complex. The data driving-based method is to build a de-artifact model by learning a large number of MRI images by using a deep learning algorithm, so as to remove artifacts. This method requires a large amount of data to train, but can achieve a better artifact removal in a shorter time.

Although MRI image quality enhancement methods have been widely used, there are some drawbacks. Firstly, the MRI super-resolution technology is easy to generate distortion, blurring and other problems in the image reconstruction process, and due to the limitation of hardware, new artifacts which are difficult to expect are often brought after the MRI is subjected to interlayer super-resolution, and how to remove the incidental artifacts without changing the hardware setting, such as scanning time and the like, so that a good solution is not available at present. In addition, most of the current methods for improving the quality of the MRI are limited to a certain aspect, such as interlayer super resolution or intra-layer super resolution on the MRI, or specifically remove the artifacts or noise of the MRI, so that the current method for improving the quality of the MRI image in all directions is lacking, which limits the clinical application of related algorithms to a certain extent.

Disclosure of Invention

The invention provides a method and a system for simultaneously realizing super-resolution and artifact removal of a magnetic resonance image, which are used for solving the defects that in the prior art, the quality of the magnetic resonance image is improved in a single aspect, the quality of the magnetic resonance image cannot be comprehensively improved due to adverse effects on other aspects when the quality of the magnetic resonance image is improved, the interlayer resolution and the in-layer resolution of the magnetic resonance image are improved, artifact stripes generated by super-resolution are restrained, and the quality of the magnetic resonance image is simultaneously improved from multiple aspects.

The invention provides a method for simultaneously realizing super-resolution and artifact removal of a magnetic resonance image, which comprises the following steps:

acquiring a plurality of first-direction magnetic resonance image slices obtained by scanning the magnetic resonance image along a first direction;

inputting a plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, and outputting a plurality of reconstructed first-direction magnetic resonance image slices;

inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and outputting the reconstructed first-direction magnetic resonance image slices from which the artifacts are removed;

inputting the plurality of artifact-removed first-direction magnetic resonance image slices into a super-resolution module in the layer, and outputting a plurality of first-direction target magnetic resonance image slices;

Reconstructing a target magnetic resonance image according to a plurality of the first direction target magnetic resonance image slices;

the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the in-layer super-resolution module is used for improving the in-layer resolution of the first-direction magnetic resonance image slice after the artifact is removed

According to the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image provided by the invention, the interlayer super-resolution module comprises a preset residual error dense network;

inputting the plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, outputting the plurality of reconstructed first-direction magnetic resonance image slices, and comprising:

carrying out scanning reconstruction on the plurality of first-direction magnetic resonance image slices along a second direction to obtain a plurality of second-direction magnetic resonance image slices;

inputting a plurality of second-direction magnetic resonance image slices into the residual dense network, and improving the in-layer resolution of each second-direction magnetic resonance image slice to obtain a plurality of optimized second-direction magnetic resonance image slices;

Carrying out scanning reconstruction on the plurality of first-direction magnetic resonance image slices along a third direction to obtain a plurality of third-direction magnetic resonance image slices;

inputting a plurality of third-direction magnetic resonance image slices into the residual dense network, and improving the in-layer resolution of each third-direction magnetic resonance image slice to obtain a plurality of optimized third-direction magnetic resonance image slices;

projecting the plurality of optimized second-direction magnetic resonance image slices and the plurality of optimized third-direction magnetic resonance image slices to the first direction to obtain a plurality of reconstructed first-direction magnetic resonance image slices;

wherein the directions of the first direction, the second direction and the third direction are different from each other.

According to the method for simultaneously realizing the super-resolution and artifact removal of the magnetic resonance image, the residual dense network is obtained by the following steps:

inputting a first image sample obtained in advance into a preset initial residual dense network, and outputting a second image sample, wherein the resolution of the first image sample is lower than that of the second image sample;

calculating regularization function loss according to the first image sample and the second image sample;

And optimizing parameters of the initial ragged dense network according to the regularization function loss, returning to the step of re-executing the output second image sample until the regularization function loss meets a preset threshold, and determining the initial residual dense network as the residual dense network.

According to the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image, which is provided by the invention, the artifact removal module comprises a preset depth residual error learning network;

inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, outputting the reconstructed first-direction magnetic resonance image slices, and the method comprises the following steps:

and inputting each reconstructed first-direction magnetic resonance image slice into the depth residual error learning network, removing artifacts from the reconstructed first-direction magnetic resonance image slices, and outputting corresponding first-direction magnetic resonance image slices after removing the artifacts.

According to the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image provided by the invention, the preset depth residual error learning network is obtained by the following modes:

performing artifact simulation based on a third image sample acquired in advance to obtain an artifact image sample;

Inputting the artifact image sample into a preset initial depth residual error learning network, and outputting a fourth image sample;

calculating a mean square error loss according to the pixel value of the third image sample and the pixel value of the fourth sample;

and optimizing parameters of the initial depth residual error learning network according to the mean square error loss, returning to the step of re-executing the output fourth image sample until the mean square error loss meets a preset threshold value, and determining the initial depth residual error learning as the depth residual error learning network.

According to the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image, the intra-layer super-resolution module comprises a three-dimensional super-resolution generation countermeasure network, and the three-dimensional super-resolution generation countermeasure network comprises a generator and a discriminator;

the first direction magnetic resonance image slice after removing the artifacts is input into the intra-layer super-resolution module, and a plurality of first direction target magnetic resonance image slices are output, including:

inputting each first direction magnetic resonance image slice after artifact removal into a generator in a three-dimensional super-resolution generation countermeasure network, carrying out in-layer resolution improvement on the first direction magnetic resonance image slice after artifact removal, and outputting a corresponding first direction target magnetic resonance image slice;

The three-dimensional super-resolution generation countermeasure network comprises a generator and a discriminator, wherein the generator and the discriminator in the three-dimensional super-resolution generation countermeasure network are optimized according to comprehensive loss between the generator and the discriminator, and the comprehensive loss comprises generation countermeasure loss, visual perception loss and projection loss optimization.

According to the method for simultaneously realizing the super-resolution and artifact removal of the magnetic resonance image, the three-dimensional super-resolution generation countermeasure network is obtained by the following steps:

acquiring training data, wherein the training data comprises a fifth image sample, a fifth image sample noise distribution function, a sixth image sample and a sixth image sample distribution function, the fifth image sample is obtained by scanning a preset first magnetic resonance image sample along the first direction, and the sixth image sample is obtained by scanning a preset second magnetic resonance image sample along the first direction;

inputting the fifth image sample into a preset initial generator to generate a seventh image sample;

inputting the seventh image sample into the initial discriminator to obtain a seventh image sample discrimination result;

inputting the sixth image sample into the initial discriminator to obtain a sixth image sample discrimination result;

Calculating and generating a countermeasures loss according to the sixth image sample distinguishing result, the seventh image sample distinguishing result, the fifth image sample noise distribution function and the sixth image sample distribution function;

extracting fifth image sample characteristics corresponding to the fifth image sample and seventh image sample characteristics corresponding to the seventh image sample;

calculating a visual perception loss according to the fifth image sample feature and the seventh image sample feature;

calculating a projection loss according to the projection of the fifth image sample in the second direction, the projection of the fifth image sample in the third direction, the projection of the seventh image sample in the second direction and the projection of the seventh image sample in the third direction;

determining a composite loss from the generated fight loss, the visual perception loss, and the projection loss;

optimizing parameters of the initial generator and parameters of the initial discriminator according to the comprehensive loss, and returning to the step of re-executing the training data acquisition until the comprehensive loss meets a preset threshold, determining the initial generator as the generator, and determining the initial discriminator as the discriminator.

The invention also provides a system for simultaneously realizing magnetic resonance image super-resolution and artifact removal, which comprises:

an acquisition unit, configured to acquire a plurality of first-direction magnetic resonance image slices obtained by scanning a magnetic resonance image along a first direction;

the interlayer super-resolution unit is used for inputting a plurality of the first-direction magnetic resonance image slices into the interlayer super-resolution module and outputting a plurality of reconstructed first-direction magnetic resonance image slices;

the artifact removing unit is used for inputting the reconstructed first-direction magnetic resonance image slices into the artifact removing module and outputting the reconstructed first-direction magnetic resonance image slices;

the intra-layer super-resolution unit is used for inputting the plurality of the artifact-removed first-direction magnetic resonance image slices into the intra-layer super-resolution module and outputting a plurality of first-direction target magnetic resonance image slices;

a reconstruction unit, configured to reconstruct a target magnetic resonance image according to a plurality of the target magnetic resonance image slices in the first direction;

the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the in-layer super-resolution module is used for improving the in-layer resolution of the first-direction magnetic resonance image slice after the artifact is removed.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and operable on the processor, wherein the processor when executing the program implements the steps of the method for simultaneously implementing super-resolution and artifact removal of magnetic resonance images as described in any one of the above.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of simultaneously performing magnetic resonance image super resolution and artifact removal as described in any of the above.

The invention also provides a computer program product comprising a computer program which when executed by a processor performs the steps of the method of simultaneously performing magnetic resonance image super resolution and artifact removal as described in any of the preceding.

The method and the system for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image provided by the invention are used for scanning the magnetic resonance image to obtain a plurality of first-direction magnetic resonance image slices, reducing the interlayer spacing among the plurality of magnetic resonance image slices through an interlayer super-resolution module, removing artifact stripes in the reconstructed first-direction magnetic resonance image slices through an artifact removal module, improving the in-layer resolution of the artifact removed first-direction magnetic resonance image slices through an in-layer super-resolution module, and reconstructing according to the plurality of first-direction target magnetic resonance image slices to obtain the target magnetic resonance image. The method has the advantages that the optimization thought of the magnetic resonance image from whole to slice to whole is realized, the interlayer spacing between the magnetic resonance image slices is reduced in the optimization process, the artifact stripes of the magnetic resonance image slices are reduced, the resolution in the magnetic resonance image slices is improved in various modes, and the image quality of the magnetic resonance image is improved in all directions.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for simultaneously achieving magnetic resonance image super-resolution and artifact removal according to the present invention;

FIG. 2 is a schematic diagram of an artifact simulation result provided by the present invention;

FIG. 3 is a schematic diagram of a process for simultaneously implementing super-resolution and artifact removal of a magnetic resonance image according to the present invention;

fig. 4 is a schematic structural diagram of a residual dense network in an interlayer super resolution module provided by the invention;

FIG. 5 is a graph showing the comparison of the effect of the interlayer super resolution result provided by the invention;

fig. 6 is a schematic structural diagram of a depth residual learning network in an artifact removal module according to the present invention;

FIG. 7 is a graph comparing the effects of artifact removal results provided by the present invention;

FIG. 8 is a schematic diagram of the structure of the generator and discriminator in the intra-layer super resolution module provided by the invention;

FIG. 9 is a graph showing the effect of super-resolution results in layers provided by the present invention;

FIG. 10 is a schematic diagram of a system for simultaneously performing magnetic resonance image super-resolution and artifact removal according to the present invention;

fig. 11 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

For the convenience of understanding the present invention, the principle of the present invention will now be explained.

The technical route of the invention comprises three modules, namely an interlayer super-resolution module, an artifact removal module and an interlayer super-resolution module. And an RRDB (Residual-in-Residual Dense Block) Residual dense block is used as a backbone network of the interlayer super resolution of the MRI image in the interlayer super resolution module, so that the interlayer resolution among MRI slices of the MRI image layered scanning is improved through the interlayer resolution module, and the interlayer spacing is reduced. In order to eliminate artifacts caused by hardware limitations in the process of interlayer super resolution, a simulated artifact data set is constructed, a depth residual error learning network is trained, and then a model is migrated to an MRI image with real artifacts, so that artifacts of an interlayer super resolution result are reduced. To further improve the MRI image quality, the generator and the discriminator are trained cooperatively through a super-resolution countermeasure generation network, and intra-layer super-resolution of MRI slices of the MRI images is achieved through the generator, thereby improving the resolution of each MRI slice in the MRI images. Finally, the effect of the invention on MRI image optimization is quantitatively assessed by comparing Peak signal-to-noise ratio (PSNR) and structural similarity (Structural Similarity, SSIM) of MRI images before and after optimization.

The invention provides a method for simultaneously realizing super-resolution and artifact removal of a magnetic resonance image, which is shown in figure 1 and comprises the following steps:

s11, acquiring a plurality of first-direction magnetic resonance image slices obtained by scanning the magnetic resonance image along a first direction;

s12, inputting a plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, and outputting a plurality of reconstructed first-direction magnetic resonance image slices;

s13, inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and outputting the reconstructed first-direction magnetic resonance image slices;

s14, inputting the plurality of artifact-removed first-direction magnetic resonance image slices into a super-resolution module in the layer, and outputting a plurality of first-direction target magnetic resonance image slices;

s15, reconstructing a target magnetic resonance image according to a plurality of target magnetic resonance image slices in the first direction;

Because the magnetic resonance image is three-dimensional, the magnetic resonance image can be scanned along one direction to obtain a plurality of slices for realizing integral optimization, and after the slices are optimized, the target magnetic resonance image is reconstructed according to the optimized slices to finish integral optimization.

It should be noted that the first direction may be set according to actual needs, and preferably one direction may be selected from the sagittal direction, the coronal direction and the axial direction as the first direction.

In the embodiment of the invention, a plurality of first-direction magnetic resonance image slices are obtained by scanning the magnetic resonance image, the interlayer spacing between the plurality of magnetic resonance image slices is reduced by the interlayer super-resolution module, the artifact fringes in the reconstructed first-direction magnetic resonance image slices are removed by the artifact removal module, the resolution of the artifact removed first-direction magnetic resonance image slices is improved by the interlayer super-resolution module, and the target magnetic resonance image is obtained by reconstructing the plurality of first-direction target magnetic resonance image slices. The method has the advantages that the optimization thought of the magnetic resonance image from whole to slice to whole is realized, the interlayer spacing between the magnetic resonance image slices is reduced in the optimization process, the artifact stripes of the magnetic resonance image slices are reduced, the resolution in the magnetic resonance image slices is improved in various modes, and the image quality of the magnetic resonance image is improved in all directions.

step S12 includes:

s121, carrying out scanning reconstruction on a plurality of first-direction magnetic resonance image slices along a second direction to obtain a plurality of second-direction magnetic resonance image slices;

s122, inputting a plurality of second-direction magnetic resonance image slices into the residual dense network, and improving the in-layer resolution of each second-direction magnetic resonance image slice to obtain a plurality of optimized second-direction magnetic resonance image slices;

s123, carrying out scanning reconstruction on the plurality of first-direction magnetic resonance image slices along a third direction to obtain a plurality of third-direction magnetic resonance image slices;

s124, inputting a plurality of third-direction magnetic resonance image slices into the residual dense network, and improving the in-layer resolution of each third-direction magnetic resonance image slice to obtain a plurality of optimized third-direction magnetic resonance image slices;

s125, projecting a plurality of optimized second-direction magnetic resonance image slices and a plurality of optimized third-direction magnetic resonance image slices to the first direction to obtain a plurality of reconstructed first-direction magnetic resonance image slices;

It should be noted that steps S123 to S124 may be performed after steps S121 to S122, may be performed before steps S121 to S122, or may be performed simultaneously with steps S121 to S122.

Specifically, taking the first direction as a sagittal direction as an example, the second direction may be a coronal direction, the third direction may be an axial direction, and the second direction and the third direction may be reversed.

Taking a plurality of magnetic resonance image slices scanned in a sagittal position as an example, the interlayer super-resolution module slices the stereoscopic magnetic resonance image slice along the direction of the coronal position (namely, the second direction), reconstructs the stereoscopic magnetic resonance image slice into the image slice of the coronal position (namely, the magnetic resonance image slice of the second direction), uses an RRDB (Residual-in-Residual Dense Block) Residual dense network as a backbone network for the intra-layer super-resolution of the image slice of the coronal position, improves the intra-layer resolution of the magnetic resonance image slice of the coronal position, namely, reduces the pixel size of the coronal position, and is convenient for reducing the inter-layer distance of the reconstructed magnetic resonance image slice of the sagittal position in the coronal position direction when the magnetic resonance image slice of the sagittal position is reconstructed by projection in the subsequent sagittal position direction.

In the same way, the plurality of magnetic resonance image slices scanned in the sagittal direction are sliced along the axial direction (namely the third direction) to reconstruct into an axial image slice (namely the third direction magnetic resonance image slice), then the RRDB residual dense network is used for carrying out in-layer super resolution on the axial image slice, the in-layer resolution of the axial direction magnetic resonance image slice is improved, namely the pixel size in the axial direction is reduced, and when the subsequent projection reconstruction of the sagittal direction magnetic resonance image slice is carried out in the sagittal direction, the interlayer spacing of the reconstructed sagittal direction magnetic resonance image slice in the axial direction is reduced.

The magnetic resonance image slice with the coronal position and the axial position is projected back to the sagittal position, namely, the interlayer super resolution of the magnetic resonance image slice scanned in the sagittal position is realized, and the interlayer spacing of the magnetic resonance image slice in the sagittal position is reduced in the sagittal position direction through projection reconstruction due to the reduction of the pixel size in the coronal position direction and the pixel size in the axial position direction. For the first direction magnetic resonance image slice of other scanning directions, the same flow can be adopted, namely, the intra-layer super-resolution is carried out on the other two projection directions to realize the inter-layer super-resolution in the projection directions. This improves the magnetic resonance imaging resolution for fine scanning to achieve MRI with unchanged scan time.

Further, the residual dense network is obtained by:

s21, inputting a first image sample obtained in advance into a preset initial residual dense network, and outputting a second image sample, wherein the resolution of the first image sample is lower than that of the second image sample;

s22, calculating regularization function loss according to the first image sample and the second image sample;

s23, optimizing parameters of the initial ragged dense network according to the regularization function loss, returning to the step of re-executing the output second image sample until the regularization function loss meets a preset threshold, and determining the initial residual dense network as the residual dense network.

Specifically, the loss function used to train the initial RRDB residual dense network is the L1 regularization function loss, see equation 1:

L ₁ ＝|HR-SR| (1)

wherein L is ₁ Representing the regularization function penalty, HR represents a first image sample, and SR represents a second image sample.

In the embodiment of the invention, for the magnetic resonance image slice in the first direction, the interlayer super-resolution in the first direction is realized by carrying out the interlayer super-resolution on the magnetic resonance slices in the second direction and the third direction. The fine scanning of the magnetic resonance image is realized under the condition that the scanning time is unchanged, and the interlayer resolution of the magnetic resonance image is improved.

According to the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image, provided by the invention, the artifact removal module comprises a preset depth residual error learning network; the step S13 specifically includes:

s131, inputting each reconstructed first-direction magnetic resonance image slice into the depth residual error learning network, removing artifacts from the reconstructed first-direction magnetic resonance image slices, and outputting corresponding first-direction magnetic resonance image slices after removing the artifacts.

Specifically, in the image output by the interlayer super-resolution module (i.e. the reconstructed first-direction magnetic resonance image slice), undesirable artifact stripes can appear, so that the interlayer super-resolution reconstructed magnetic resonance image slice has a certain difference from the actual magnetic resonance image slice of the clinical scan. Aiming at the problem, a solution based on a deep learning method is provided, because the generated artifact images are mostly horizontal or vertical stripes, the images in the other two projection directions cannot be completely aligned after interlayer super resolution basically due to the limitation of hardware conditions, and therefore certain dislocation can occur in the magnetic resonance image slices reconstructed in the scanning direction, and certain difference exists between the magnetic resonance image slices actually scanned clinically.

Since there is no one-to-one pairing of artifact images and artifact-free image pairs in reality, it is difficult to directly realize learning from artifact-free images to artifact removal. The artifact-free magnetic resonance image slice is thus restored from the simulated artifact-free magnetic resonance image slice by simulating the composite quality-degraded artifact slice on the basis of the artifact-free magnetic resonance image slice and then learning the distinction between the artifact-free magnetic resonance image slice and the degraded artifact-free magnetic resonance image slice using the depth residual learning network. And finally, the depth residual error learning network is transferred to a real super-resolution reconstruction image with artifacts (namely, a reconstructed first-direction magnetic resonance image slice), so that artifact fringes of the super-resolution reconstructed first-direction magnetic resonance image slice are removed.

The preset deep residual error learning network is obtained by the following mode:

s31, performing artifact simulation based on a third image sample acquired in advance to obtain an artifact image sample;

s32, inputting the artifact image sample into a preset initial depth residual error learning network, and outputting a fourth image sample;

s33, calculating a mean square error loss according to the pixel value of the third image sample and the pixel value of the fourth sample;

And S34, optimizing parameters of the initial depth residual error learning network according to the mean square error loss, returning to the step of re-executing the output fourth image sample until the mean square error loss meets a preset threshold value, and determining the initial depth residual error learning as the depth residual error learning network.

In one example, as shown in fig. 2, the left side of fig. 2 is an artifact image sample obtained by artifact simulation, and the right side of fig. 2 is a third image sample before artifact simulation.

The loss function of the depth residual learning network is the mean square error (MSE, mean Squared Error) loss, defined as:

wherein y is _i Representing the pixel values of the third image sample,the pixel value of the fourth image sample is represented, n represents the total number of pixels, and i represents the pixels.

In the embodiment of the invention, an artifact image is simulated through a third image sample, the artifact image is input into an initial depth residual error learning network to remove the artifact to obtain a fourth image sample, the depth residual error learning network is trained according to the mean square error loss obtained by calculating the pixel values of the fourth image sample and the third image sample, and the removal of artifact fringes of the super-resolution reconstructed first-direction magnetic resonance image slice is realized through the depth residual error learning network.

the step S14 specifically includes:

s141, inputting each first-direction magnetic resonance image slice after artifact removal into a generator in a three-dimensional super-resolution generation countermeasure network, improving the resolution of the first-direction magnetic resonance image slice after artifact removal, and outputting a corresponding first-direction target magnetic resonance image slice;

Specifically, in order to preserve the three-dimensional information of the magnetic resonance image in the process of realizing the intra-layer super resolution, a three-dimensional super resolution generation countermeasure network (SRGAN) is utilized as a backbone network of the intra-layer super resolution module. The method mainly comprises two parts, namely a generator for generating intra-layer super-resolved first-direction target magnetic resonance image slices and a discriminator for judging whether the generated first-direction target magnetic resonance image slices are close to real first-direction magnetic resonance image slices.

Because of the lack of a true pair of images for high-resolution and low-resolution MRI of the same patient, the high-resolution MRI images are downsampled prior to training to obtain paired low-resolution and high-resolution MRI datasets, and the datasets are applied to SRGAN for training.

In one example, the three-dimensional super-resolution generation countermeasure network is obtained by:

s401, acquiring training data, wherein the training data comprises a fifth image sample, a fifth image sample noise distribution function, a sixth image sample and a sixth image sample distribution function, the fifth image sample is obtained by scanning a preset first magnetic resonance image sample along the first direction, and the sixth image sample is obtained by scanning a preset second magnetic resonance image sample along the first direction;

s402, inputting the fifth image sample into a preset initial generator to generate a seventh image sample;

s403, inputting the seventh image sample into the initial discriminator to obtain a seventh image sample discrimination result;

s404, inputting the sixth image sample into the initial discriminator to obtain a sixth image sample discrimination result;

S405, calculating and generating a countermeasures loss according to the sixth image sample distinguishing result, the seventh image sample distinguishing result, the fifth image sample noise distribution function and the sixth image sample distribution function;

s407, extracting fifth image sample characteristics corresponding to the fifth image sample and seventh image sample characteristics corresponding to the seventh image sample;

s408, calculating visual perception loss according to the fifth image sample characteristic and the seventh image sample characteristic;

s409, calculating a projection loss according to the projection of the fifth image sample in the second direction, the projection of the fifth image sample in the third direction, the projection of the seventh image sample in the second direction and the projection of the seventh image sample in the third direction;

s410, determining comprehensive loss according to the generated countermeasures loss, the visual perception loss and the projection loss;

s411 optimizes the parameters of the initial generator and the parameters of the initial discriminator according to the comprehensive loss, and returns to re-execute the step of acquiring training data until the comprehensive loss meets a preset threshold, determines the initial generator as the generator, and determines the initial discriminator as the discriminator.

Specifically, the comprehensive loss function for training the three-dimensional super-resolution generation countermeasure network comprises three parts, namely generation countermeasure loss (GAN loss) for improving the quality of the generated result of the generator and the discrimination capability of the discriminator; secondly, visual perception loss (visual loss) is used for further constraining the output of the model so that the result output by the model is more similar to the effect seen by the real human eyes; and thirdly, projection loss (projection loss) designed for the task aims at restricting the output of the model in the other two projection/scanning directions, so that the output result of the model can more keep the three-dimensional information of the magnetic resonance image slice.

Wherein, a countering loss is generated, see formula 3:

wherein D represents an initial discriminator and G represents an initial generator; z represents a fifth image sample, and x represents a sixth image sample; p is p _data (x) Representing a distribution of sixth image samples;representing the expected value, p, of the sixth image sample distribution function _noise (z) represents a noise distribution of the fifth image sample; />Representing the expected value of the fifth image sample noise distribution function.

Loss of visual perception, see equation 4:

wherein L is _perceptual Representing visual perception loss, F represents feature extraction, preferably using VGG16 network _i (x) Representing the result of feature extraction on the seventh image sample, F _i (y) represents a result of feature extraction of the fifth image sample; n represents the total number of samples; i represents the i-th sample.

Projection loss, see equation 5:

L _projection ＝0.5MSE(HR ₂ ，SR ₂ )+0.5MSE(HR ₃ ，SR ₃ )

wherein L is _projection Representing projection loss, HR ₂ Representing the projection of the fifth image sample in the second direction, SR ₂ Representing a projection of the seventh image sample in the second direction, HR ₃ Representing the projection of the fifth image sample in the third direction, SR ₃ Representing the projection of the seventh image sample in the third direction, MSE represents the mean square error calculation.

Comprehensive loss, see formula 5:

L _total ＝λ ₁ L _GAN +λ ₂ L _perceptual +λ ₃ L _projection (5)

wherein lambda is ₁ ，λ ₂ ，λ ₃ Representing the weight coefficient.

Preferably, the weight coefficient may be set to λ ₁ 0.4 lambda ₂ 0.4 lambda ₃ 0.2.

In the embodiment of the invention, the resolution of the first-direction magnetic resonance image slice after artifact removal can be improved by generating the three-dimensional super-resolution generation countermeasure network generator obtained by the countermeasure loss, the visual perception loss and the projection loss training, so that the intra-layer super-resolution is realized, and the first-direction target magnetic resonance image slice with higher image quality is generated.

Based on one example of the embodiments described above, to evaluate the effectiveness of the proposed method of simultaneously achieving magnetic resonance imaging super resolution and artifact removal, the labeled MRI image is divided into three parts for model training, validation and testing. Wherein 50 patients, total 900 images are used for training the three partial models, total 432 images of 24 patients are used for verifying the three partial models, and total 360 images of 20 patients are used for verifying the three partial models and the whole flow.

In the training process, adam optimizers were used to train on NVIDIATITAN RTX graphics cards with 24GB computing memory. All models were trained for 800 rounds with a learning rate of 0.0002.

The performance of the model was evaluated using Peak signal-to-noise ratio (PSNR) and structural similarity (Structural Similarity, SSIM). Particularly, for the interlayer super-resolution effect, the output super-resolution model and the original image are loaded into a medical image processing software 3D slice, and the difference of the interlayer spacing of the super-resolution model and the original image is visually compared.

The flow for simultaneously realizing the super-resolution and artifact removal of the magnetic resonance image is shown in fig. 3:

and scanning the magnetic resonance image along the sagittal direction to obtain a sagittal magnetic resonance image slice, and inputting the sagittal magnetic resonance image slice into the interlayer super-resolution module.

In the interlayer super-resolution module, the sagittal magnetic resonance image slice is scanned and reconstructed along the axial position direction and the coronal position direction respectively to obtain the axial position direction magnetic resonance image slice and the coronal position direction magnetic resonance image slice, a residual dense network is input respectively to obtain an optimized axial position direction magnetic resonance image slice with reduced axial position direction spacing and a coronal position direction magnetic resonance image slice with reduced coronal position direction spacing, the axial position direction magnetic resonance image slice and the coronal position direction magnetic resonance image slice are projected to the sagittal position direction simultaneously to obtain the reconstructed sagittal magnetic resonance image slice, and the reduction of the sagittal position direction magnetic resonance image slice spacing in the sagittal position direction is realized through projection reconstruction due to the reduction of the coronal position direction pixel size and the axial position direction pixel size.

And inputting the reconstructed sagittal magnetic resonance image slice into a depth residual error learning network in an artifact removal module, and removing artifact strips from the reconstructed sagittal magnetic resonance image slice to obtain the artifact-removed sagittal magnetic resonance image slice.

The method comprises the steps of inputting a magnetic resonance image slice in the sagittal direction after artifact removal into a generator in a three-dimensional super-resolution generation countermeasure network in an intra-layer super-resolution module, carrying out intra-layer resolution improvement on the magnetic resonance image slice in the sagittal direction after artifact removal to obtain a target magnetic resonance image slice in the sagittal direction, and outputting the target magnetic resonance image slice.

And reconstructing a target magnetic resonance image according to the sagittal target magnetic resonance image slice, so as to realize the optimization of the magnetic resonance image.

In the above procedure, the structure of the residual dense network in the interlayer super-resolution module is shown in fig. 4, and the effect pairs of the multiple image slices passing through the interlayer super-resolution module are shown in fig. 5, where the upper part of fig. 5 is a magnetic resonance image slice that does not pass through the interlayer super-resolution module (i.e. before super-resolution), and the lower part of fig. 5 is a magnetic resonance image slice that passes through the interlayer super-resolution module (i.e. after super-resolution). The interlayer spacing of the magnetic resonance image after passing through the interlayer super-resolution module is obviously reduced from original 5mm to 0.78mm, and the effectiveness of the interlayer super-resolution model is verified.

In the above-mentioned flow, the structure of the depth residual learning network in the artifact removal module is shown in fig. 6, and the image slice effect after passing through the artifact removal module is shown in fig. 7, for example, the magnetic resonance image slice before passing through the artifact removal module is shown on the left side of fig. 7, and the magnetic resonance image slice after passing through the artifact removal module is shown on the right side of fig. 7. Some horizontal and vertical stripes exist on the magnetic resonance image slice before artifact removal, and after the artifact removal module, the artifact stripes are obviously inhibited. From a numerical point of view, PSNR in the test set image is improved by 15.68db on average and ssim is improved by 0.097 on average through this module.

In the above-described flow, the generator structure and the discriminator structure in the intra-layer super-resolution module are as shown in fig. 8, the upper part of fig. 8 is the generator structure, and the lower part of fig. 8 is the discriminator structure. The effect of the magnetic resonance image slice before and after passing through the intra-layer super-resolution module is as shown in fig. 9, the left side of fig. 9 is the magnetic resonance image slice before passing through the intra-layer super-resolution module, and the right side of fig. 9 is the magnetic resonance image slice after passing through the intra-layer super-resolution module. For the enhancement of the in-layer resolution, the in-layer resolution of 1mm can be enhanced to 0.5mm from the original. The detail part in the magnetic resonance image is obviously recovered and improved after passing through the in-layer super-resolution module. Numerically, PSNR in the test set images increased by 2.88db on average and ssim increased by 0.046 on average.

The average improving effects of PSNR and SSIM on the test set in the method for simultaneously realizing the super-resolution and artifact removal of the magnetic resonance image are shown in Table 1:

table 1, magnetic resonance image optimizing effect contrast table

In summary, the magnetic resonance imaging method provided by the invention can improve the image quality of MRI from three aspects, can realize the improvement of the inter-layer resolution and the intra-layer resolution of the magnetic resonance image, and can also realize a remarkable inhibition effect on the artifact stripes generated after super resolution.

The system for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image provided by the invention is described below, and the system for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image described below and the method for simultaneously realizing the super-resolution and the artifact removal of the magnetic resonance image described above can be correspondingly referred to each other.

The invention also provides a system for simultaneously realizing super-resolution and artifact removal of the magnetic resonance image, as shown in fig. 10, comprising:

an acquisition unit 101, configured to acquire a plurality of first-direction magnetic resonance image slices obtained by scanning a magnetic resonance image along a first direction;

an interlayer super-resolution unit 102, configured to input a plurality of the first-direction magnetic resonance image slices into an interlayer super-resolution module, and output a plurality of reconstructed first-direction magnetic resonance image slices;

An artifact removal unit 103, configured to input the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and output the reconstructed first-direction magnetic resonance image slices;

an intra-layer super-resolution unit 104, configured to input the plurality of artifact-removed first-direction magnetic resonance image slices into an intra-layer super-resolution module, and output a plurality of first-direction target magnetic resonance image slices;

a reconstruction unit 105 for reconstructing a target magnetic resonance image from a plurality of said first direction target magnetic resonance image slices;

the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the intra-layer super-resolution module is used for improving the resolution of the magnetic resonance image slice in the first direction after the artifact is removed.

In the embodiment of the invention, a plurality of first-direction magnetic resonance image slices are obtained by scanning a magnetic resonance image, interlayer distances among the plurality of magnetic resonance image slices are reduced through an interlayer super-resolution module, artifact fringes in the reconstructed first-direction magnetic resonance image slices are removed through an artifact removal module, the in-layer resolution of the artifact removed first-direction magnetic resonance image slices is improved through an in-layer super-resolution module, and a target magnetic resonance image is obtained according to the reconstruction of the plurality of first-direction target magnetic resonance image slices. The method has the advantages that the optimization thought of the magnetic resonance image from whole to slice to whole is realized, the interlayer spacing between the magnetic resonance image slices is reduced in the optimization process, the artifact stripes of the magnetic resonance image slices are reduced, the resolution in the magnetic resonance image slices is improved in various modes, and the image quality of the magnetic resonance image is improved in all directions.

Fig. 11 illustrates a physical structure diagram of an electronic device, as shown in fig. 11, which may include: processor 1110, communication interface Communications Interface 1120, memory 1130 and communication bus 1140, wherein processor 1110, communication interface 1120 and memory 1130 communicate with each other via communication bus 1140. The processor 1110 may invoke logic instructions in the memory 1130 to perform a method for simultaneously achieving magnetic resonance imaging super resolution and artifact removal, the method comprising: acquiring a plurality of first-direction magnetic resonance image slices obtained by scanning the magnetic resonance image along a first direction; inputting a plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, and outputting a plurality of reconstructed first-direction magnetic resonance image slices; inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and outputting the reconstructed first-direction magnetic resonance image slices from which the artifacts are removed; inputting the plurality of artifact-removed first-direction magnetic resonance image slices into a super-resolution module in the layer, and outputting a plurality of first-direction target magnetic resonance image slices; reconstructing a target magnetic resonance image according to a plurality of the first direction target magnetic resonance image slices; the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the in-layer super-resolution module is used for improving the in-layer resolution of the first-direction magnetic resonance image slice after the artifact is removed.

Further, the logic instructions in the memory 1130 described above may be implemented in the form of software functional units and sold or used as a stand-alone product, stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the method for simultaneously achieving super resolution and artifact removal of a magnetic resonance image provided by the methods described above, the method comprising: acquiring a plurality of first-direction magnetic resonance image slices obtained by scanning the magnetic resonance image along a first direction; inputting a plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, and outputting a plurality of reconstructed first-direction magnetic resonance image slices; inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and outputting the reconstructed first-direction magnetic resonance image slices from which the artifacts are removed; inputting the plurality of artifact-removed first-direction magnetic resonance image slices into a super-resolution module in the layer, and outputting a plurality of first-direction target magnetic resonance image slices; reconstructing a target magnetic resonance image according to a plurality of the first direction target magnetic resonance image slices; the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the in-layer super-resolution module is used for improving the in-layer resolution of the first-direction magnetic resonance image slice after the artifact is removed.

In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is configured to perform the methods provided above to acquire a plurality of first-direction magnetic resonance image slices scanned along a first direction; inputting a plurality of first-direction magnetic resonance image slices into an interlayer super-resolution module, and outputting a plurality of reconstructed first-direction magnetic resonance image slices; inputting the reconstructed first-direction magnetic resonance image slices into an artifact removal module, and outputting the reconstructed first-direction magnetic resonance image slices from which the artifacts are removed; inputting the plurality of artifact-removed first-direction magnetic resonance image slices into a super-resolution module in the layer, and outputting a plurality of first-direction target magnetic resonance image slices; reconstructing a target magnetic resonance image according to a plurality of the first direction target magnetic resonance image slices; the interlayer super-resolution module is used for reducing interlayer spacing among a plurality of first-direction magnetic resonance image slices; the artifact removal module is used for removing artifact stripes in the reconstructed first-direction magnetic resonance image slice; the in-layer super-resolution module is used for improving the in-layer resolution of the first-direction magnetic resonance image slice after the artifact is removed.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for simultaneously achieving super-resolution and artifact removal of magnetic resonance images, comprising:

2. The method for simultaneously achieving magnetic resonance image super-resolution and artifact removal according to claim 1, wherein the interlayer super-resolution module comprises a preset residual dense network;

3. The method for simultaneously achieving super-resolution and artifact removal of magnetic resonance images according to claim 2, wherein the residual dense network is obtained by:

4. The method for simultaneously achieving super-resolution and artifact removal of a magnetic resonance image according to claim 1, wherein the artifact removal module comprises a preset depth residual learning network;

5. The method for simultaneously achieving super-resolution and artifact removal of magnetic resonance images according to claim 4, wherein the preset depth residual learning network is obtained by the following manner:

6. The method of claim 1, wherein the intra-layer super-resolution module comprises a three-dimensional super-resolution generation countermeasure network, the three-dimensional super-resolution generation countermeasure network comprising a generator and a discriminator;

7. The method for simultaneously achieving magnetic resonance image super-resolution and artifact removal according to claim 6, wherein the three-dimensional super-resolution generation countermeasure network is obtained by:

8. A system for simultaneously achieving magnetic resonance image super-resolution and artifact removal, comprising:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor performs the steps of the method of any one of claims 1 to 7 for simultaneously achieving super resolution of magnetic resonance images and artifact removal.

10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor performs the steps of the method of simultaneously performing magnetic resonance image super resolution and artifact removal according to any of claims 1 to 7.