CN115880440B - Magnetic particle three-dimensional reconstruction imaging method based on generation countermeasure network - Google Patents

Abstract

The invention belongs to the technical field of medical imaging, and specifically relates to a method, system, and device for three-dimensional reconstruction and imaging of magnetic particles based on generative confrontation networks, aiming to solve the problems of long reconstruction time and low imaging resolution of existing MPI high-resolution images. The method comprises: collecting a multi-view sparse two-dimensional MPI image of a phantom of an object to be imaged and reconstructed, and performing down-sampling to obtain a down-sampled sparse two-dimensional MPI image; inputting the down-sampled sparse two-dimensional MPI image into training A good neighborhood point average diffusion search self-attention generative confrontation network generation model, to obtain a dense two-dimensional MPI image; through the filter back projection reconstruction algorithm to reconstruct each dense two-dimensional MPI image, and finally obtain the three-dimensional object to be imaged and reconstructed MPI image; the invention speeds up the MPI image reconstruction speed, improves the imaging resolution, and makes the magnetic particle imaging equipment have greater application prospects in the medical field.

Description

A 3D reconstruction and imaging method of magnetic particles based on generative adversarial network

technical field

The invention belongs to the technical field of medical imaging, and in particular relates to a method, system and device for three-dimensional reconstruction and imaging of magnetic particles based on generative confrontation networks.

Background technique

Magnetic particle imaging is a new type of imaging technology, which has the advantages of no radiation and high sensitivity. At present, magnetic particle imaging equipment has shown broad application prospects in clinical research, including frontier directions such as early detection of lesions and targeted drug therapy. In the current magnetic particle imaging equipment, how to quickly realize the scanning of objects and high-resolution three-dimensional image reconstruction is a challenging problem at present. At present, some devices implement tomographic scanning of objects by rotating the device coil, and scan two-dimensional images of objects at different angles by physically rotating the imaging coil, and then perform filtered back projection reconstruction on the scanned images. However, this imaging technology needs to rotate and move the coil, and if the rotation speed is too fast, the service life of the equipment will be reduced. Secondly, if you want to improve the time resolution of scanning, you need to scan multiple times, and the process is time-consuming and laborious. Therefore, how to improve the reconstruction speed and imaging resolution of imaging while taking into account the construction cost of hardware equipment is an urgent problem to be faced at present. Based on this, the present invention proposes a three-dimensional reconstruction and imaging method of magnetic particles based on generative confrontation network.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, in order to solve the problems of long reconstruction time and low imaging resolution of existing MPI three-dimensional high-resolution images, the present invention provides a three-dimensional reconstruction and imaging method of magnetic particles based on generative confrontation network, The method includes the following steps:

Step S100, collect the multi-view sparse two-dimensional MPI image of the phantom of the object to be imaged and reconstructed, and perform down-sampling to obtain the down-sampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI image is followed by Set the angle to rotate multiple MPI projection images collected;

Step S200, inputting the downsampled sparse two-dimensional MPI image into the trained average diffusion search self-attention generating adversarial network generation model of the neighborhood points to obtain a dense two-dimensional MPI image;

Step S300, reconstructing each dense two-dimensional MPI image through a filtered back-projection reconstruction algorithm, and finally obtaining a three-dimensional MPI image of the object to be imaged and reconstructed;

The self-attention generation confrontation network of the neighborhood point average diffusion search includes a generation model and a discrimination model;

The generation model includes five diffusion search attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on sequentially connected neighborhood point average diffusion convolution subnetwork, self-attention fusion network and activation function layer ; The activation function used in the activation function layer of the first four diffusion search attention mechanism convolution modules is Leaky ReLU, and the activation function used in the activation function layer of the fifth diffusion search attention mechanism convolution module is a hyperbolic tangent function;

The neighborhood point average diffusion convolution sub-network is configured to sequentially encode the down-sampled sparse two-dimensional MPI image through a coding function to obtain sequentially coded images; to perform regional convolution operations on each sequentially coded image to obtain each sequence Encode the particle concentration information, angle information and correlation information between images of the image;

The self-attention fusion network is configured to perform linear transformation by multiplying the particle concentration information, angle information, and correlation information between images of each sequentially encoded image with the corresponding weight matrix, and perform fusion through a multi-head attention mechanism, and fuse Afterwards, upsampling is performed to obtain a dense two-dimensional MPI image;

The discriminant model includes five convolutional neural network modules; the convolutional neural network module is constructed based on sequentially connected convolutional layers, regularization layers and activation function layers; the activation function layers of the first four convolutional neural network modules adopt The activation function of the network is Leaky ReLU, and the activation function used in the activation function layer of the fifth convolutional neural network module is softmax.

In some preferred embodiments, the training process of the Neighborhood Point Average Diffusion Search Self-Attention Generation Adversarial Network is:

Step A100, using the MPI imaging device to collect a multi-view sparse two-dimensional MPI image of the phantom of the object to be imaged and reconstructed, and perform down-sampling to obtain a down-sampled sparse two-dimensional MPI image; based on the down-sampled sparse two-dimensional Dimensional MPI image and its corresponding ground truth label to obtain the training data set;

Step A200, inputting the down-sampled sparse two-dimensional MPI image into the pre-built neighborhood point average diffusion search self-attention generation confrontation network generation model to obtain a dense two-dimensional MPI image;

Step A300, input the dense two-dimensional MPI image and its corresponding ground-truth label into the discriminant model of the neighborhood point average diffusion search self-attention generation confrontation network, and obtain the discriminant result of the dense two-dimensional MPI image;

Step A400, based on the discriminant result, combine each dense two-dimensional MPI image, each sparse two-dimensional MPI image and its corresponding ground truth label, calculate the total loss value through the pre-built loss function, and update the neighborhood points Average Diffusion Search Self-Attention Generative Adversarial Network Generative Model, Network Parameters of Discriminative Model;

Step A500, cyclically train the generator network and the discriminator network of the Neighborhood Point Average Diffusion Search Self-Attention Generative Adversarial Network until a trained Neighborhood Point Average Diffusion Search Self-Attention Generative Adversarial Network is obtained.

In some preferred embodiments, let the number of generated angles be , at an angle of when, the The encoding function of a sparse two-dimensional MPI image is :

.

In some preferred embodiments, the area convolution operation is performed on the sequentially encoded image, and the method is: process the concentration value of the pixel in the sequentially encoded image, and use a 3×3 convolution kernel to perform operation;

The method for processing the density values of the pixels in the sequentially coded image is:

in, Indicates that the first in the sequence coded picture row number the pixels of the column, Represents a point neighborhood The average of all values in the express the minimum of all values within the neighborhood of the point, express the maximum value of all values within the neighborhood of a point, express Variable values within a point neighborhood, consisting of horizontally, vertically, and diagonally adjacent points.

In some preferred embodiments, the particle concentration information, angle information, and correlation degree information between images of each sequentially encoded image are multiplied by the corresponding weight matrix for linear transformation, and the method is as follows:

;

;

in, Represents the self-attention mechanism, a vector Indicates the particle concentration information, ;vector represents angle information, ;vector Represents the correlation information between images, , Indicates that the matrix is transformed into a one-dimensional column vector, and the vector Represents the sequentially encoded image of the input neighborhood point-averaged diffusion convolutional sub-network; , , Represent the weight matrix corresponding to the particle concentration information, angle information and correlation degree information between images of sequentially encoded images, Represents each sequentially coded image neurons in all output layers The sum of output results from 1 to N, Represents each sequentially coded image After the first The output value of a neuron, Represents each sequentially coded image After the first The output value of a neuron, Represents each sequentially coded image After the first The output value of neurons accounts for the total output value proportion.

In some preferred embodiments, the particle concentration information, angle information and correlation degree information between images of each sequentially encoded image are fused through a multi-head attention mechanism, and the method is as follows:

in, is the stitching function, Indicates the first The output of the attention mechanism sub-module, represents the total weight matrix output by the multi-head attention mechanism, Indicates the information obtained by fusing the particle concentration information, angle information, and correlation degree information between images of sequentially encoded images.

In some preferred implementations, the total loss value is calculated through a pre-built loss function, and the method is as follows:

in, is the total loss function, Represents the pixel loss function and the adversarial loss function, Represents the angular order sorting optimization function, Indicates the pixel information error, represents the loss of the generative model, Represents a sparse 2D MPI image, Indicates the generated A dense 2D MPI image, Indicates the generated A dense 2D MPI image, Indicates the first The ground truth label corresponding to a sparse 2D MPI image, represents the coefficient of the pixel loss, represents the coefficient of the adversarial loss, Indicates the probability of whether it is true or not judged by the discriminant network, represents the output of the generative model, represents the angle difference function, is the hyperbolic tangent function, Indicates the predicted value of the angle under the corresponding input, , Represents any two dense two-dimensional MPI images Angle Judgment Function, and Respectively represent the first The angle labels corresponding to the generated dense 2D MPI images, Indicates the parameters set for the richness of information contained in different phantoms.

In the second aspect of the present invention, a magnetic particle three-dimensional reconstruction imaging system based on generative confrontation network is proposed, and the system includes: an image acquisition module 100, an image reconstruction module 200, and a three-dimensional reconstruction module 300;

The image acquisition module 100 is configured to acquire a multi-view sparse two-dimensional MPI image of a phantom of an object to be imaged and reconstructed, and perform down-sampling to obtain a down-sampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI image The dimensional MPI image is a plurality of MPI projection images that are sequentially rotated and collected according to the set angle;

The image reconstruction module 200 is configured to input the sparse two-dimensional MPI image after down-sampling into the trained neighborhood point average diffusion search self-attention generation confrontation network generation model to obtain a dense two-dimensional MPI image;

The three-dimensional reconstruction module 300 is configured to reconstruct each dense two-dimensional MPI image through a filtered back-projection reconstruction algorithm, and finally obtain a three-dimensional MPI image of an object to be imaged and reconstructed;

The self-attention generation confrontation network of the neighborhood point average diffusion search includes a generation model and a discrimination model;

The generation model includes five diffusion search attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on sequentially connected neighborhood point average diffusion convolution subnetwork, self-attention fusion network and activation function layer ; The activation function used in the activation function layer of the first four diffusion search attention mechanism convolution modules is Leaky ReLU, and the activation function used in the activation function layer of the fifth diffusion search attention mechanism convolution module is a hyperbolic tangent function;

The neighborhood point average diffusion convolution sub-network is configured to sequentially encode the down-sampled sparse two-dimensional MPI image through a coding function to obtain sequentially coded images; to perform regional convolution operations on each sequentially coded image to obtain each sequence Encode the particle concentration information, angle information and correlation information between images of the image;

The self-attention fusion network is configured to perform linear transformation by multiplying the particle concentration information, angle information, and correlation information between images of each sequentially encoded image with the corresponding weight matrix, and perform fusion through a multi-head attention mechanism, and fuse Afterwards, upsampling is performed to obtain a dense two-dimensional MPI image;

The discriminant model includes five convolutional neural network modules; the convolutional neural network module is constructed based on sequentially connected convolutional layers, regularization layers and activation function layers; the activation function layers of the first four convolutional neural network modules adopt The activation function of the network is Leaky ReLU, and the activation function used in the activation function layer of the fifth convolutional neural network module is softmax.

The third aspect of the present invention proposes a storage device in which a plurality of programs are stored, and the programs are suitable for being loaded and executed by a processor to realize the above-mentioned three-dimensional reconstruction and imaging method of magnetic particles based on a generative adversarial network.

In a fourth aspect of the present invention, a processing arrangement is proposed, including a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing multiple programs; the program is suitable for being loaded by the processor And execute to realize the above-mentioned three-dimensional reconstruction and imaging method of magnetic particles based on generative confrontation network.

Beneficial effects of the present invention:

The invention improves the reconstruction speed and imaging resolution of MPI imaging;

The invention integrates the self-attention mechanism of neighborhood point average diffusion search into the generative confrontation network, uses the angle order sorting optimization function to optimize, accelerates the speed of MPI image reconstruction, improves the quality of MPI image reconstruction, and makes up for the hardware defects existing in current equipment. The magnetic particle imaging equipment has a greater application prospect in the medical field.

Description of drawings

Other characteristics, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

Fig. 1 is a schematic flow chart of a three-dimensional reconstruction and imaging method of magnetic particles based on generative confrontation network according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of the training process of the magnetic particle three-dimensional reconstruction and imaging method based on the generative confrontation network according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of the network module of the attention mechanism of an embodiment of the present invention;

Fig. 4 is a schematic diagram of the network structure of the magnetic particle three-dimensional reconstruction and imaging method based on the generative confrontation network according to an embodiment of the present invention;

Fig. 5 is a schematic framework diagram of a three-dimensional reconstruction and imaging system of magnetic particles based on a generative adversarial network according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, not to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

The present invention provides a method for three-dimensional reconstruction and imaging of magnetic particles based on generative confrontation network, as shown in Figure 1, the method includes the following steps:

Step S100, collect the multi-view sparse two-dimensional MPI image of the phantom of the object to be imaged and reconstructed, and perform down-sampling to obtain the down-sampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI image is followed by Set the angle to rotate multiple MPI projection images collected;

Step S200, inputting the downsampled sparse two-dimensional MPI image into the trained average diffusion search self-attention generating adversarial network generation model of the neighborhood points to obtain a dense two-dimensional MPI image;

Step S300, reconstructing each dense two-dimensional MPI image through a filtered back-projection reconstruction algorithm, and finally obtaining a three-dimensional MPI image of the object to be imaged and reconstructed;

The self-attention generation confrontation network of the neighborhood point average diffusion search includes a generation model and a discrimination model;

The generation model includes five diffusion search attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on sequentially connected neighborhood point average diffusion convolution subnetwork, self-attention fusion network and activation function layer ; The activation function used in the activation function layer of the first four diffusion search attention mechanism convolution modules is Leaky ReLU, and the activation function used in the activation function layer of the fifth diffusion search attention mechanism convolution module is a hyperbolic tangent function;

The neighborhood point average diffusion convolution sub-network is configured to sequentially encode the down-sampled sparse two-dimensional MPI image through a coding function to obtain sequentially coded images; to perform regional convolution operations on each sequentially coded image to obtain each sequence Encode the particle concentration information, angle information and correlation information between images of the image;

The self-attention fusion network is configured to perform linear transformation by multiplying the particle concentration information, angle information, and correlation information between images of each sequentially encoded image with the corresponding weight matrix, and perform fusion through a multi-head attention mechanism, and fuse Afterwards, upsampling is performed to obtain a dense two-dimensional MPI image;

The discriminant model includes five convolutional neural network modules; the convolutional neural network module is constructed based on sequentially connected convolutional layers, regularization layers and activation function layers; the activation function layers of the first four convolutional neural network modules adopt The activation function of the network is Leaky ReLU, and the activation function used in the activation function layer of the fifth convolutional neural network module is softmax.

In order to more clearly describe a method for three-dimensional reconstruction and imaging of magnetic particles based on a generative adversarial network of the present invention, each step in the method embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

In the process of magnetic particle imaging (MPI) imaging with existing commercial equipment, in order to image the distribution of particles inside the object, the object needs to be scanned, but the current physical scanning method based on coil rotation takes a relatively long time to reconstruct the image. Long, multi-angle tomography is limited by the rotation speed of the coil, which greatly increases the hardware cost of the device and shortens the service life of the device. In order to effectively reduce reconstruction time and save hardware cost, the present invention designs a multi-view three-dimensional magnetic particle imaging (Multi-view Three Dimensional Magnetic Particle Imaging, MV-3D-MPI) method and system based on a generative discriminant model. The invention proposes a neighborhood point average diffusion search self-attention generation confrontation network model, aiming at realizing fast reconstruction of MPI high-resolution images and improving imaging resolution.

In the following embodiments, the training process of the Neighborhood Point Average Diffusion Search Self-Attention Generation Adversarial Network will be described first, and then the testing process of the Neighborhood Point Average Diffusion Search Self-Attention Generation Adversarial Network will be described in detail.

1. Neighborhood point average diffusion search self-attention generation confrontation network training process, the specific steps are as follows:

Step A100, using the MPI imaging device to collect a multi-view sparse two-dimensional MPI image of the phantom of the object to be imaged and reconstructed, and perform down-sampling to obtain a down-sampled sparse two-dimensional MPI image; based on the down-sampled sparse two-dimensional Dimensional MPI image and its corresponding ground truth label to obtain the training data set;

In this embodiment, the MPI projection images with dense multi-view angles are collected by self-owned MPI equipment, and 15,000 examples of images are used as training sets; for a phantom, it is preferable to collect an image every 5° (in other embodiments, you can According to actual needs, set the angle interval to collect images), and collect a total of 36 angles of projected MPI images; through any conventional down-sampling method, the pixel size is changed to 12×12, and the down-sampled image is used as the input object of the network .

Step A200, inputting the down-sampled sparse two-dimensional MPI image into the pre-built neighborhood point average diffusion search self-attention generation confrontation network generation model to obtain a dense two-dimensional MPI image;

In this embodiment, the Neighborhood Point Average Diffusion Search Self-Attention Generation Adversarial Network includes a generation model and a discrimination model;

The generation model includes five diffusion search attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on sequentially connected neighborhood point average diffusion convolution subnetwork, self-attention fusion network and activation function layer ; The activation function used in the activation function layer of the first four diffusion search attention mechanism convolution modules is Leaky ReLU, and the activation function used in the activation function layer of the fifth diffusion search attention mechanism convolution module is a hyperbolic tangent function;

The neighborhood point average diffusion convolution sub-network is configured to sequentially encode the down-sampled sparse two-dimensional MPI image through a coding function to obtain sequentially coded images; to perform regional convolution operations on each sequentially coded image to obtain each sequence Encode the particle concentration information, angle information and correlation information between images of the image;

The self-attention fusion network is configured to perform linear transformation by multiplying the particle concentration information, angle information, and correlation information between images of each sequentially encoded image with the corresponding weight matrix, and perform fusion through a multi-head attention mechanism, and fuse Afterwards, upsampling is performed to obtain a dense two-dimensional MPI image;

In the present invention, since the input has a sequence relationship of angles, in order to allow the model to learn the relevance of adjacent angles and the global correlation of different angles, the input layer of the network adopts sequential encoding, and the input of the set model includes 6 different The image of the angle is taken every 30° in [0°,180°), and the angle is when, for the Coding function of a sparse two-dimensional MPI image for:

in, Indicates the number of generated angles, and the sine and cosine functions in the formula can ensure that the constructed model learns the relative positional relationship between different images.

According to the characteristics of the MPI image, the value of neighborhood point diffusion search is mainly used to more fully model the relationship between a certain pixel value of the MPI image and its neighbors, because the magnetic particle response in the magnetic field free area will be affected by the particles next to it, resulting in The imaged density values at some points are related to the density values of other surrounding pixels. Carrying out area convolution operation on the sequentially coded image, the method is: processing the concentration value of the pixels in the sequentially coded image, preferably using a 3×3 convolution kernel to perform the operation after processing;

The method for processing the density values of the pixels in the sequentially coded image is:

by Take the point as an example to process the concentration value as follows:

in, Indicates that the first in the sequence coded picture row number the pixels of the column, Represents a point neighborhood The average of all values in the expressthe minimum of all values within the neighborhood of the point, express the maximum value of all values within the neighborhood of a point, express Variable values within a point neighborhood, consisting of horizontally, vertically, and diagonally adjacent points.

The present invention proposes a self-attention fusion network module in order to fuse key information extracted from images by different sub-networks in the part of generating the network model. Let the input of each sub-network be the preprocessed image, three sub-networks are used to process the input to obtain different types of information, and the obtained three types of information are multiplied by the corresponding weight matrix for linear transformation. The corresponding weight matrices are respectively,,, and then through the self-attention fusion network module, different types of information are fused, and the particle concentration information, angle information and correlation information between images of each sequentially encoded image are multiplied by the corresponding weight matrix for linear transformation. The method is :

in,Represents the self-attention mechanism, a vectorIndicates the particle concentration information,;vectorrepresents angle information,;vectorRepresents the correlation information between images,,Indicates that the matrix is transformed into a one-dimensional column vector, and the vectorRepresents the sequentially encoded image of the input neighborhood point-averaged diffusion convolutional sub-network;,,Represent the weight matrix corresponding to the particle concentration information, angle information and correlation degree information between images of sequentially encoded images,Represents each sequentially coded imageneurons in all output layersThe sum of output results from 1 to N,Represents each sequentially coded imageAfter the firstThe output value of a neuron,Represents each sequentially coded imageAfter the firstThe output value of a neuron,Represents each sequentially coded imageAfter the firstThe output value of neurons accounts for the total output valueproportion.

Through the multi-head attention mechanism, the particle concentration information, angle information and correlation information between images of each sequentially encoded image are fused, and the method is as follows:

{X}_{O}=Upsample\left [ {MF\left ( {{H}_{i}\left ( {{X}_{1},{X}_{2},{X}_{ 3}} \right )} \right )} \right ]

in,is the stitching function,Indicates the firstThe output of the attention mechanism sub-module,represents the total weight matrix output by the multi-head attention mechanism,Represents the fused information of the particle concentration information, angle information and correlation information between the images of the sequentially encoded images, and splicing the output of each attention sub-module by combining withMultiply for linear transformation, and finally get the output of the generated model by upsampling,Represents the upsampling function.

Step A300, input the dense two-dimensional MPI image and its corresponding ground-truth label into the discriminant model of the neighborhood point average diffusion search self-attention generation confrontation network, and obtain the discriminant result of the dense two-dimensional MPI image;

The discriminant model includes five convolutional neural network modules; the convolutional neural network module is constructed based on sequentially connected convolutional layers, regularization layers and activation function layers; the activation function layers of the first four convolutional neural network modules adopt The activation function of the network is Leaky ReLU, and the activation function used in the activation function layer of the fifth convolutional neural network module is softmax.

The convolution layer performs convolution operation on the input data of the discriminant model, and then performs regularization through the regularization layer, and finally inputs the activation function layer to perform function operation.

Step A400, based on the discriminant result, combine each dense two-dimensional MPI image, each sparse two-dimensional MPI image and its corresponding ground truth label, calculate the total loss value through the pre-built loss function, and update the neighborhood points Average Diffusion Search Self-Attention Generative Adversarial Network Generative Model, Network Parameters of Discriminative Model;

In this embodiment, the total loss value is calculated through the pre-built loss function, and the method is as follows:

in,is the total loss function,Represents the pixel loss function and the adversarial loss function,Represents the angular order sorting optimization function,Indicates the pixel information error,represents the loss of the generative model,Represents a sparse 2D MPI image,Indicates the generatedA dense 2D MPI image,Indicates the generatedA dense 2D MPI image,Indicates the firstThe ground truth label corresponding to a sparse 2D MPI image,represents the coefficient of the pixel loss,Represents the coefficient of the adversarial loss, in this embodimentandPreferably set to 1,Indicates the probability of whether it is true or not judged by the discriminant network,represents the output of the generative model,represents the angle difference function,is the hyperbolic tangent function,Indicates the predicted value of the angle under the corresponding input,,Represents any two dense two-dimensional MPI imagesAngle judgment function, if its judgment value is true, then the value is 1, otherwise it is 0,andRespectively represent the firstThe angle label corresponding to the generated dense 2D MPI image;Indicates the parameters set for the richness of information contained in different imitation bodies. In the present invention, if the imitation body collected is a letter imitation body, for the letter imitation bodyPreferably set to 1, for more complex Chinese character imitationThe preferred setting is 2. For phantoms specially used to evaluate the resolution, due to the rich details,The preferred setting is 3. When the image pair is taken, the phantom is used as the unit, and the image pair taken by each phantom image is supervised, and the wrong order is punished. The three sets of phantoms are preferably filled with an inner diameter of 0.5 mm containing 10 mg/ml Perimag solution. The field of view (FOV) of the three phantoms is preferably 6 cm×6 cm×6 cm (letter), 6 cm×6 cm×10 cm (spiral, the acquisition time of a sparse projection is 3 minutes), 6cm×6cm × cm (container, acquisition time 2 minutes for one sparse projection).

Step A500, cyclically train the generator network and the discriminator network of the Neighborhood Point Average Diffusion Search Self-Attention Generative Adversarial Network until the trained Neighborhood Point Average Diffusion Search Self-Attention Generative Adversarial Network is obtained;

2. Neighborhood Point Average Diffusion Search Self-Attention Generative Adversarial Network Testing Process

Step S100, collect the multi-view sparse two-dimensional MPI image of the phantom of the object to be imaged and reconstructed, and perform down-sampling to obtain the down-sampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI image is followed by Set the angle to rotate multiple MPI projection images collected;

In this embodiment, 200 cases of images collected by the MPI device are used as a test set.

Step S200, inputting the downsampled sparse two-dimensional MPI image into the trained average diffusion search self-attention generating adversarial network generation model of the neighborhood points to obtain a dense two-dimensional MPI image;

In this embodiment, the self-attention generation adversarial network is searched based on the average diffusion of trained neighborhood points. °, 30°,35°,40°,45°,50°,55°,60°,65°,...,170°,175°] at any angle, the pixel size of the generated dense two-dimensional MPI image is 48×48.

Step S300, reconstructing each dense two-dimensional MPI image through a filtered back-projection reconstruction algorithm, and finally obtaining a three-dimensional MPI image of the object to be imaged and reconstructed;

A magnetic particle three-dimensional reconstruction and imaging system based on a generative confrontation network according to the second embodiment of the present invention, as shown in FIG. 4 , includes: an image acquisition module 100, an image reconstruction module 200, and a three-dimensional reconstruction module 300;

The image acquisition module 100 is configured to acquire a multi-view sparse two-dimensional MPI image of a phantom of an object to be imaged and reconstructed, and perform down-sampling to obtain a down-sampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI image The dimensional MPI image is a plurality of MPI projection images that are sequentially rotated and collected according to the set angle;

The image reconstruction module 200 is configured to input the sparse two-dimensional MPI image after down-sampling into the trained neighborhood point average diffusion search self-attention generation confrontation network generation model to obtain a dense two-dimensional MPI image;

The three-dimensional reconstruction module 300 is configured to reconstruct each dense two-dimensional MPI image through a filtered back-projection reconstruction algorithm, and finally obtain a three-dimensional MPI image of an object to be imaged and reconstructed;

The self-attention generation confrontation network of the neighborhood point average diffusion search includes a generation model and a discrimination model;

The generation model includes five diffusion search attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on sequentially connected neighborhood point average diffusion convolution subnetwork, self-attention fusion network and activation function layer ; The activation function used in the activation function layer of the first four diffusion search attention mechanism convolution modules is Leaky ReLU, and the activation function used in the activation function layer of the fifth diffusion search attention mechanism convolution module is a hyperbolic tangent function;

The neighborhood point average diffusion convolution sub-network is configured to sequentially encode the down-sampled sparse two-dimensional MPI image through a coding function to obtain sequentially coded images; to perform regional convolution operations on each sequentially coded image to obtain each sequence Encode the particle concentration information, angle information and correlation information between images of the image;

The self-attention fusion network is configured to perform linear transformation by multiplying the particle concentration information, angle information, and correlation information between images of each sequentially encoded image with the corresponding weight matrix, and perform fusion through a multi-head attention mechanism, and fuse Afterwards, upsampling is performed to obtain a dense two-dimensional MPI image;

The discriminant model includes five convolutional neural network modules; the convolutional neural network module is constructed based on sequentially connected convolutional layers, regularization layers and activation function layers; the activation function layers of the first four convolutional neural network modules adopt The activation function of is LeakyReLU, and the activation function used in the activation function layer of the fifth convolutional neural network module is softmax.

Those skilled in the technical field can clearly understand that for the convenience and brevity of the description, the specific working process and relevant descriptions of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

It should be noted that the GAN-based magnetic particle three-dimensional reconstruction and imaging system provided by the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned by different functional modules, that is, to decompose or combine the modules or steps in the embodiments of the present invention. For example, the modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules to complete the above-described full or partial functionality. The names of the modules and steps involved in the embodiments of the present invention are only used to distinguish each module or step, and are not regarded as improperly limiting the present invention.

A storage device according to the third embodiment of the present invention stores a plurality of programs, and the programs are suitable for being loaded by a processor to implement the above-mentioned three-dimensional reconstruction and imaging method of magnetic particles based on a generative adversarial network.

A processing device according to the fourth embodiment of the present invention includes a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing multiple programs; the program is suitable for being loaded and executed by the processor In order to realize the above-mentioned three-dimensional reconstruction and imaging method of magnetic particles based on generative confrontation network.

Those skilled in the technical field can clearly understand that for the convenience and brevity of the description, the specific working process and related instructions of the storage device and the processing device described above can refer to the corresponding process in the aforementioned method examples, and will not be repeated here. repeat.

Those skilled in the art should be able to realize that the modules and method steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two, and that the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or known in the technical field any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of functions in the above description. Whether these functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functionality using different methods for each particular application, but such implementation should not be considered as exceeding the scope of the present invention.

The terms "first", "second", "third", etc. are used to distinguish similar items, and are not used to describe or represent a specific order or sequence.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to related technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (10)

1. A magnetic particle three-dimensional reconstruction imaging method based on generation of a countermeasure network, the method comprising the steps of:

step S100, acquiring a multi-view sparse two-dimensional MPI image of an imitation body of a reconstructed object to be imaged, and performing downsampling to obtain a downsampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI images are a plurality of MPI projection images which are collected in sequence according to set angles in a rotating mode;

step S200, inputting the downsampled sparse two-dimensional MPI image into a trained generation model of a self-attention generation countermeasure network by average diffusion search of neighborhood points, and obtaining a dense two-dimensional MPI image;

Step S300, reconstructing each dense two-dimensional MPI image by a filtering back projection reconstruction algorithm to finally obtain a three-dimensional MPI image of the object to be imaged;

the neighborhood point average diffusion search self-attention generation countermeasure network comprises a generation model and a discrimination model;

the generating model comprises five diffusion searching attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on a neighborhood point average diffusion convolution sub-network, a self-attention fusion network and an activation function layer which are connected in sequence; the activation function adopted by the activation function layer in the first four diffusion searching attention mechanism convolution modules is a leakage ReLU, and the activation function adopted by the activation function layer in the fifth diffusion searching attention mechanism convolution module is a hyperbolic tangent function;

the neighborhood point average diffusion convolution sub-network is configured to sequentially encode the downsampled sparse two-dimensional MPI image through an encoding function to obtain a sequentially encoded image; performing regional convolution operation on each sequence coding image to obtain particle concentration information, angle information and association degree information among the images of each sequence coding image;

the self-attention fusion network is configured to multiply the particle concentration information, the angle information and the association degree information among the images of each sequence coding image with the corresponding weight matrix for linear transformation, fuse the images through a multi-head attention mechanism, and up-sample the fused images to obtain a dense two-dimensional MPI image;

The discrimination model comprises five convolutional neural network modules; the convolutional neural network module is constructed based on a convolutional layer, a regularization layer and an activation function layer which are sequentially connected; the activation function adopted by the activation function layer of the first four convolutional neural network modules is a leak ReLU, and the activation function adopted by the activation function layer of the fifth convolutional neural network module is a softmax.

2. The method for three-dimensional reconstruction imaging of magnetic particles based on generation of an countermeasure network according to claim 1, wherein the training process of generating the countermeasure network by average spread search self-attention of the neighborhood points is as follows:

step A100, acquiring a multi-view sparse two-dimensional MPI image of an imitation body of a reconstructed object to be imaged through MPI imaging equipment, and performing downsampling to obtain a downsampled sparse two-dimensional MPI image; obtaining a training data set based on the downsampled sparse two-dimensional MPI image and a corresponding truth value label thereof;

step A200, inputting the downsampled sparse two-dimensional MPI image into a pre-constructed neighborhood point average diffusion search self-attention generation countermeasure network generation model to obtain a dense two-dimensional MPI image;

step A300, inputting the dense two-dimensional MPI image and the truth value label corresponding to the dense two-dimensional MPI image into the neighborhood point average diffusion search self-attention generation countermeasure network discrimination model, and acquiring discrimination results of the dense two-dimensional MPI image;

Step A400, based on the discrimination result, combining each dense two-dimensional MPI image, each sparse two-dimensional MPI image and the corresponding truth value labels thereof, calculating a total loss value through a pre-constructed loss function, and updating the network parameters of the neighborhood point average diffusion search self-attention generation countermeasure network generation model and discrimination model;

and step A500, training the generator network and the discriminator network of the neighborhood point average spread search self-attention generation countermeasure network in a circulating way until the trained neighborhood point average spread search self-attention generation countermeasure network is obtained.

3. The method for three-dimensional reconstruction imaging of magnetic particles based on generation of countermeasure network according to claim 1, wherein the number of generation angles is set to be

At an angle of +>

At the time->

Zhang Xishu two-dimensional MPI imageIs a function of the code of (a)

The method comprises the following steps: />

。

4. The method for three-dimensional reconstruction imaging of magnetic particles based on generation of countermeasure network according to claim 1, wherein the sequentially encoded images are subjected to a region convolution operation, which comprises the following steps: processing the concentration values of the pixel points in the sequential coding image, and performing operation by adopting a 3 multiplied by 3 convolution kernel after processing;

The method for processing the concentration value of the pixel point in the sequential coding image comprises the following steps:

；

wherein ,

representing the +.o in the sequentially encoded pictures>

Line->

Pixel points of column->

Representing the neighborhood of points->

Mean value of all values in>

Representation->

Minimum of all values in the point neighborhood, +.>

Representation->

Maximum value of all values in the point neighborhood, +.>

Representation->

Variable values in the point neighborhood.

5. The method for three-dimensional reconstruction imaging of magnetic particles based on generation of countermeasure network according to claim 1, wherein the particle concentration information, angle information, and correlation degree information between images of each sequentially encoded image are multiplied by corresponding weight matrices to perform linear transformation, the method is as follows:

；

；

wherein ,

representing the self-attention mechanism, vector->

Information indicating particle concentration>

The method comprises the steps of carrying out a first treatment on the surface of the Vector->

The information of the angle is represented by a set of angles,/>

the method comprises the steps of carrying out a first treatment on the surface of the Vector->

Information representing the degree of association between images, +.>

，/>

Representing the deformation of the matrix into a one-dimensional column vector, vector +.>

Representing sequentially encoded images of the input neighborhood point average diffusion convolution sub-network; />

、/>

、/>

Respectively representing weight matrices corresponding to particle concentration information, angle information and correlation degree information between images of sequentially encoded images, ++>

Representing each sequentially encoded image +. >

Neurons in all output layers +.>

Sum of output results from 1 to N, < ->

Representing each sequentially encoded image +.>

Through->

Output values of individual neurons, < >>

Representing each sequentially encoded image +.>

Through->

Output values of individual neurons, < >>

Representing each sequentially encoded image +.>

Through->

The output value of the individual neurons is the total output value +.>

Is a specific gravity of (c).

6. The method for three-dimensional reconstruction imaging of magnetic particles based on generation of countermeasure network according to claim 5, wherein the particle concentration information, the angle information and the association degree information between images of each sequentially encoded image are fused by a multi-head attention mechanism, the method comprising:

；

；

wherein ,

is a splicing function->

Indicate->

Output of the attention machine submodule, +.>

Total weight matrix representing the output of the multi-head attention mechanism,/->

And information obtained by fusing the particle concentration information, the angle information and the association degree information between the images of the sequential encoded images.

7. A method of three-dimensional reconstruction imaging of magnetic particles based on generation of a countermeasure network according to claim 2, characterized in that the total loss value is calculated by a pre-constructed loss function by:

；

；

；

；

wherein ,

for the total loss function- >

Representing pixel loss function and contrast loss function, < ->

Representing an angular sequential ordering optimization function,/->

Representing pixel information error,/>

Representing the loss of the generative model->

A sparse two-dimensional MPI image is represented,

representing the generated->

Zhang Miji two-dimensional MPI image,>

representing the generated->

Zhang Miji two-dimensional MPI image,>

indicate->

Zhang Xishu truth label corresponding to two-dimensional MPI image, < ->

Coefficients representing pixel loss, < >>

Coefficient representing countermeasures against losses->

Represents the probability of judging whether or not it is true by the judging network, < >>

Representing the output of the generative model,/>

Representing the angle difference function, ++>

As hyperbolic tangent function, +.>

Representing the predicted value of the angle at the corresponding input,

，/>

representing any two dense two-dimensional MPI images +.>

Angle judging function (F)>

and />

Respectively represent +.>

Angle label corresponding to dense two-dimensional MPI image generated by sheet,/->

Parameters indicating the richness settings for the information contained in the different mimics.

8. A magnetic particle three-dimensional reconstruction imaging system based on a generation countermeasure network, the system comprising: an image acquisition module 100, an image reconstruction module 200 and a three-dimensional reconstruction module 300;

the image acquisition module 100 is configured to acquire a multi-view sparse two-dimensional MPI image of an imitation body of a reconstructed object to be imaged, and perform downsampling to obtain a downsampled sparse two-dimensional MPI image; the multi-view sparse two-dimensional MPI images are a plurality of MPI projection images which are collected in sequence according to set angles in a rotating mode;

The image reconstruction module 200 is configured to input the downsampled sparse two-dimensional MPI image into a trained generation model of a neighborhood point average diffusion search self-attention generation countermeasure network to obtain a dense two-dimensional MPI image;

the three-dimensional reconstruction module 300 is configured to reconstruct each dense two-dimensional MPI image through a filtered back projection reconstruction algorithm, and finally obtain a three-dimensional MPI image of the reconstructed object to be imaged;

the neighborhood point average diffusion search self-attention generation countermeasure network comprises a generation model and a discrimination model;

the generating model comprises five diffusion searching attention mechanism convolution modules; the diffusion search attention mechanism convolution module is constructed based on a neighborhood point average diffusion convolution sub-network, a self-attention fusion network and an activation function layer which are connected in sequence; the activation function adopted by the activation function layer in the first four diffusion searching attention mechanism convolution modules is a leakage ReLU, and the activation function adopted by the activation function layer in the fifth diffusion searching attention mechanism convolution module is a hyperbolic tangent function;

the neighborhood point average diffusion convolution sub-network is configured to sequentially encode the downsampled sparse two-dimensional MPI image through an encoding function to obtain a sequentially encoded image; performing regional convolution operation on each sequence coding image to obtain particle concentration information, angle information and association degree information among the images of each sequence coding image;

The self-attention fusion network is configured to multiply the particle concentration information, the angle information and the association degree information among the images of each sequence coding image with the corresponding weight matrix for linear transformation, fuse the images through a multi-head attention mechanism, and up-sample the fused images to obtain a dense two-dimensional MPI image;

the discrimination model comprises five convolutional neural network modules; the convolutional neural network module is constructed based on a convolutional layer, a regularization layer and an activation function layer which are sequentially connected; the activation function adopted by the activation function layer of the first four convolutional neural network modules is a leak ReLU, and the activation function adopted by the activation function layer of the fifth convolutional neural network module is a softmax.

9. A storage device in which a plurality of programs are stored, characterized in that the programs are adapted to be loaded and executed by a processor to implement a method of generating a countermeasure network-based magnetic particle three-dimensional reconstruction imaging as claimed in any of claims 1 to 7.

10. A processing device, comprising a processor and a storage device; a processor adapted to execute each program; a storage device adapted to store a plurality of programs; characterized in that the program is adapted to be loaded and executed by a processor for implementing a method for three-dimensional reconstruction imaging of magnetic particles based on generation of a countermeasure network as claimed in any of claims 1 to 7.

Images