CN118115677A - Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning - Google Patents

Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning Download PDF

Info

Publication number
CN118115677A
CN118115677A CN202410524431.1A CN202410524431A CN118115677A CN 118115677 A CN118115677 A CN 118115677A CN 202410524431 A CN202410524431 A CN 202410524431A CN 118115677 A CN118115677 A CN 118115677A
Authority
CN
China
Prior art keywords
dimensional
resolution
image
photoacoustic
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410524431.1A
Other languages
Chinese (zh)
Other versions
CN118115677B (en
Inventor
杨思华
陈昕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China Normal University
Original Assignee
South China Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China Normal University filed Critical South China Normal University
Priority to CN202410524431.1A priority Critical patent/CN118115677B/en
Publication of CN118115677A publication Critical patent/CN118115677A/en
Application granted granted Critical
Publication of CN118115677B publication Critical patent/CN118115677B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/1702—Systems in which incident light is modified in accordance with the properties of the material investigated with opto-acoustic detection, e.g. for gases or analysing solids
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/1702—Systems in which incident light is modified in accordance with the properties of the material investigated with opto-acoustic detection, e.g. for gases or analysing solids
    • G01N2021/1706—Systems in which incident light is modified in accordance with the properties of the material investigated with opto-acoustic detection, e.g. for gases or analysing solids in solids
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Pathology (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Analytical Chemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Ultra Sonic Daignosis Equipment (AREA)

Abstract

The invention discloses a three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning, which comprises the following steps of: manufacturing a simulation data set according to the actual imaging sample and system parameters; constructing a signal correction reconstruction sub-network and training; establishing CycleGAN-Pixel2Pixel resolution balance reconstruction sub-network and training, wherein CycleGAN and Pixel2 pixels need to be trained independently and then subjected to cascade joint optimization; constructing a three-dimensional fidelity reconstruction sub-network and training; and carrying out network reconstruction on the three-dimensional photoacoustic signals acquired by the photoacoustic tomography system. The invention is used for solving the technical problem of unbalanced resolution in all directions of the traditional photoacoustic tomography, improves the quality of three-dimensional photoacoustic imaging, and is beneficial to better analyzing and identifying tissue structures and vascular networks.

Description

Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning
Technical Field
The invention relates to the field of photoacoustic imaging and deep learning image processing, in particular to a three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning.
Background
Photoacoustic imaging is an imaging mode based on the photoacoustic effect, and light contrast and ultrasonic spatial resolution are achieved cooperatively by utilizing light excitation and ultrasonic detection. Among them, photoacoustic computed tomography is a major implementation of photoacoustic imaging, which can image large biological samples such as tissues, organs, etc. at depths of millimeter to centimeter. However, the tomographic characteristic means that accurate imaging can be presented only when a photoacoustic signal is received around a sample with sufficient angular coverage. In practical applications, the system cannot meet the condition of full-angle coverage due to the limitation of the shape of a sample, especially in clinical applications, a linear array is often only used to scan a region of interest, such as a limb, from one direction, resulting in serious imaging artifacts.
A more prominent disadvantage of linear arrays compared to other detectors is the imbalance in the performance of each dimension in three-dimensional imaging. In tomographic imaging using a linear array, the reconstruction of a three-dimensional object is formed by overlapping a plurality of 2D tangential planes after reconstruction, i.e. the dimension of information reconstruction is reduced from a three-dimensional space to a 2D plane. Because of the scattering of light in an object, even a linear spot, its excitation area is not an ideal tangential plane, but a three-dimensional "tangential plane" with a certain thickness. In order to make the reconstructed 2D section information more accurately correspond to the section information of the actual three-dimensional object, the signals received by the array elements are limited in the corresponding 2D section as much as possible through focusing means such as an acoustic lens, so that the signal intensity outside the section is reduced. The resolution of a linear array has three dimensions: axial, transverse, tangential direction. Generally, because the array element spacing is generally far smaller than the scanning step distance, the focusing capability of a single array element is far smaller than the virtual focusing capability of array element combination, and therefore the transverse resolution is higher than the tangential direction, the traditional reconstruction method only focuses on 2D reconstruction of axial-transverse tangential planes, the reconstructed 2D tangential planes are directly stacked into a three-dimensional body, tangential direction information is ignored, and the visual difference of the reconstructed three-dimensional body in each direction is large, so that the analysis and the utilization of subsequent reconstruction bodies are influenced.
Disclosure of Invention
The invention mainly aims to overcome the defects and shortcomings of the prior art, provides a three-dimensional isotropic high-resolution photoacoustic tomography method based on deep learning, and solves the technical problems of poor quality, unbalanced resolution in all directions and the like of the traditional photoacoustic tomography method.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
The three-dimensional isotropy high-resolution photoacoustic tomography method based on the deep learning comprises the following steps of:
step S01, manufacturing a simulation data set: performing photoacoustic simulation by using the three-dimensional simulated vessel set to obtain a photoacoustic signal, and forming a photoacoustic data set by using a 2.5D photoacoustic signal sinogram, a 2D traditional reconstructed image and a 2D original image as a group of paired data;
step S02, a signal correction reconstruction sub-network is constructed and trained, the data set formed in the step S01 is used for monitoring and training, a 2.5D photoacoustic signal sinogram and a traditional 2D reconstruction image are input, and the corrected 2D reconstruction image is output;
Step S03, constructing a resolution balance reconstruction sub-network and training: adopting a CycleGAN-Pixel2Pixel network structure, stacking 2D reconstructed images output in the step S02 to form a three-dimensional body, taking two-direction slices for independent training CycleGAN, independently training the Pixel2Pixel by using an image output by CycleGAN, performing cascade connection after independent training for joint optimization, inputting low-resolution slices in the simulated three-dimensional body after joint training is finished to restore the low-resolution slices to high-resolution slices, and stacking the slices to obtain an isotropic-resolution reconstructed three-dimensional body;
Step S04, constructing a three-dimensional fidelity reconstruction sub-network and training: performing supervision training by taking the three-dimensional photoacoustic signal sinogram output in the step S01 and the reconstructed three-dimensional volume output in the step S03 as a data set, inputting the data set into the reconstructed three-dimensional volume output in the step S03, outputting the data set into a fidelity reconstructed three-dimensional volume, performing operation on the output by using a photoacoustic forward operator to obtain a three-dimensional photoacoustic signal sinogram generated by the fidelity reconstructed three-dimensional volume, and performing loss calculation by taking the sinogram in the step S01 as a label;
And S05, inputting the photoacoustic three-dimensional photoacoustic signal matrix acquired by the photoacoustic tomography system and the traditional reconstructed three-dimensional image matrix into the network to obtain a high-fidelity three-dimensional photoacoustic reconstructed image with balanced final resolution.
Furthermore, the three-dimensional simulation blood vessel set is generated by setting the branch position, the branch radius and the branch hierarchical structure parameters of the generated blood vessel according to the actual imaging tissue region.
Furthermore, the photoacoustic simulation sets an ultrasonic transducer array, an imaging field of view and an imaging grid according to an actual imaging system, an imaging sample is set to generate blood vessels, and an actual system scanning step is converted from an actual distance to a grid distance to set a simulation scanning step.
Further, in the simulation, a 2D photoacoustic signal sinogram is obtained through single scanning, and after the sample is scanned layer by layer, the signal sinogram is stacked to obtain a 3D photoacoustic signal sinogram; the 2.5D photoacoustic signal sinogram is segmented from the 3D photoacoustic signal sinogram, and the segmentation thickness depends on the thickness of the linear light spot according to an actual system.
Furthermore, the signal correction reconstruction sub-network adopts a full convolution neural network architecture and is divided into a signal channel, an image channel and a merging channel;
The signal channel is used for correcting a signal sinogram, and specifically comprises 3D full convolution layers, a convolution kernel is designed according to the system sampling rate, the number of ultrasonic transducer array elements and the proportion of scanning steps, and the convolution kernel sizes are 3×35×7, 3×21×5 and 3×13×3 respectively;
The image channel is used for accelerating the network training speed, adopts ResNet network architecture, is formed by stacking 3 images ResBlock, and extracts sample characteristics from the traditional reconstructed image;
The combining channel is used for combining the output of the signal and the image channel, and a 2D-UNet network architecture is adopted to combine the feature images from the signal channel and the image channel in the channel dimension and then input the feature images into the 2D-UNet.
Further, cycleGAN and Pixel2 pixels in the resolution equalization reconstruction sub-network are adopted, the generator adopts a U-Net architecture, and the arbiter adopts a PatchGAN architecture.
Further, the data set adopted by the resolution equalization reconstruction sub-network is as follows:
the CycleGAN data set is composed of an xy slice and an xz slice which are obtained by dividing a three-dimensional body formed by stacking 2D reconstructed images corrected in the step S02 along z and y, wherein due to imaging characteristics of a system, the xy slice is a high-resolution image, and the xz slice is a low-resolution image; learning two processes during training, wherein the generator G A learns to retire the high-resolution image into a low-resolution image, and the generator G B learns to restore the low-resolution image into the high-resolution image; only calling a generator G A during testing, and degenerating the xy slice into a low-resolution image, so as to obtain a paired high-definition and low-definition image data pair;
The Pixel2Pixel data set consists of high-definition image pairs generated by CycleGAN, two processes are learned during training, a generator G C learns to degenerate a high-resolution image into a low-resolution image, and a generator G D learns to restore the low-resolution image into a high-resolution image; only generator G D is called during testing to restore the xz slice to a high resolution image.
Furthermore, the CycleGAN-Pixel2Pixel joint optimization step is that the parameter value during independent training is used as an initial value after cascade connection, the loss weighted sum of the CycleGAN, pixel Pixel sub-network is used as a loss function after cascade connection, the weight is used as a super parameter, and the adjustment is carried out according to the magnitude order obtained by actual calculation; parameters CycleGAN are updated firstly, parameters of the Pixel2Pixel are locked at the moment, then the parameters of the Pixel2Pixel are updated, at the moment, the parameters of CycleGAN are locked, the parameters are updated once by CycleGAN, the parameters of the Pixel2Pixel are updated once for one cycle, and the process is repeated until the process is repeated, and a high-resolution image which can keep the original low-resolution slice content can be output.
Further, the three-dimensional fidelity reconstruction sub-network adopts a 3D-UNet network architecture; the photoacoustic forward operator adopts a pre-trained network architecture of a 3D-ResNet + full-connection layer, after training is completed, parameters are fixed and used for loss calculation of a three-dimensional fidelity reconstruction sub-network, so that gradient return can be carried out in the loss calculation process.
Further, the processing method of the three-dimensional reconstruction volume data set comprises the following steps: and according to the proportion of the signal and the image in the xyz three directions, the original three-dimensional body is cut off in an equal proportion to obtain corresponding signal-image three-dimensional voxel pairs with smaller sizes, so that the number of network parameters is reduced.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. The invention reconstructs the isotropic resolution high-fidelity three-dimensional photoacoustic image based on the deep learning method, solves the problem that the resolution anisotropy exists in the reconstructed image caused by the physical characteristics of the system in the traditional photoacoustic imaging reconstruction method, ensures that the imaging result is clearer, has more abundant details, and is beneficial to better analyzing and identifying the tissue structure and the vascular network.
2. The self-supervision training resolution balance reconstruction sub-network provided by the invention does not need to rely on an additional marked data set, can directly use the reconstructed image as a training set, reduces the difficulty and cost of acquiring high-quality labeling data, and improves the training efficiency and flexibility.
3. The three-dimensional fidelity reconstruction sub-network provided by the invention utilizes the photoacoustic signal obtained by the pre-trained photoacoustic forward operator to perform loss calculation gradient feedback, not only reconstructs a high-resolution image, but also ensures the fidelity of the image, namely the reconstructed image is highly consistent with the physical and biological characteristics of the actual tissue, and provides accurate reference information for clinical diagnosis and biological research.
4. According to the method, the three sub-networks are cascaded and combined and optimized, so that the whole process from signal processing to final image reconstruction is optimized, each step can be mutually coordinated, and the overall imaging performance is further improved.
Proper nouns remark:
ResNet: the residual network is a deep neural network architecture, and is characterized in that the concept of residual learning is introduced, and data is allowed to directly jump from one layer to the following layers by adding jump connection between different layers of the network.
3D-ResNet: resNet is specifically designed to process three-dimensional data, and in contrast to conventional ResNet, 3D-ResNet considers the depth (third dimension) of the data in processing the convolution and pooling operations, enabling it to process three-dimensional image or video data.
U-Net: is a convolutional neural network architecture specifically designed for medical image segmentation, named from its U-shaped structure, which includes a contracted path (for capturing context information) and a symmetrical expanded path (for accurate localization), which are interconnected by a jump connection.
2D-UNet, 3D-UNet: is two variants of the U-Net architecture, processing two-dimensional and three-dimensional image data, respectively.
CycleGAN: is a network of image-to-image conversion that allows one style of image to be converted to another without paired training data. CycleGAN by learning the mapping between the two image domains and introducing the cyclic consistency loss, ensures that the content of the input image can be kept unchanged when the input image is converted back to the original image after conversion, thereby realizing style migration.
Pixel2Pixel: is a conditional generation countermeasure network for image-to-image conversion tasks. It learns the mapping between the input image to the target image by paired image training data. Pixel2Pixel uses a generator network to generate the target image and a discriminator network to distinguish the generated image from the actual target image, thereby achieving high quality image conversion.
CycleGAN-Pixel2Pixel: network architecture that concatenates CycleGAN and Pixel2Pixel networks.
PatchGAN: is a discriminator structure for determining whether a local area of an image (rather than the entire image) is authentic. This structure is typically used in image-to-image conversion tasks, such as pixels 2Pixel and CycleGAN. The purpose of the PatchGAN arbiter is to improve the quality and detail of the generated image, and by focusing only on the local area of the image (i.e. "patch"), the image texture and detail can be captured and improved more carefully.
Symbol notes:
All three-dimensional body x, y, z directions refer to depth, lateral, tangential (or elevation) directions, respectively.
Drawings
FIG. 1 is a flow chart of a three-dimensional isotropic high-resolution photoacoustic tomography method based on deep learning;
Fig. 2 is an example of three-dimensional photoacoustic reconstruction of the present invention.
Detailed Description
In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning solves the technical problem that resolution of the existing photoacoustic tomography is unbalanced in all directions, improves imaging quality, does not need to acquire isotropy resolution three-dimensional images and anisotropy resolution three-dimensional images as paired data sets, and reduces cost for manufacturing the data sets; referring to fig. 1, the present invention includes the following steps:
S1, manufacturing a simulation data set, which specifically comprises the following steps:
S11, generating a three-dimensional simulated blood vessel by utilizing VascuSynth software, and setting the branch position, the branch radius and the branch hierarchical structure parameters of the generated blood vessel according to an actual imaging tissue region;
S12, taking a three-dimensional simulated blood vessel as a sample, performing photoacoustic simulation through a photoacoustic simulation tool bag k-Wave in Matlab to obtain a photoacoustic signal, setting an ultrasonic transducer array, an imaging field of view and an imaging grid according to an actual imaging system, and converting an actual system scanning step from an actual distance to a grid distance to perform setting of a simulated scanning step;
S13, in simulation, a 2D photoacoustic signal sinogram is obtained through single scanning, and after layer-by-layer scanning is carried out on a sample, the signal sinogram is stacked to obtain a 3D photoacoustic signal sinogram; dividing the 3D photoacoustic signal sinogram along the scanning direction, wherein the dividing thickness depends on the thickness of a linear light spot according to an actual system, so as to obtain a 2.5D photoacoustic signal sinogram; the 2.5D photoacoustic signal sinogram, the conventional 2D reconstructed image and the original 2D image form a photoacoustic dataset as a set of paired data.
S2, constructing a signal correction reconstruction sub-network and training, wherein the method specifically comprises the following steps:
s21, constructing a signal channel for correcting a signal sinogram, specifically 3D full convolution layers, and designing convolution kernels according to the system sampling rate, the number of ultrasonic transducer array elements and the proportion of scanning steps, wherein the convolution kernels are 3×35×7, 3×21×5 and 3×13×3 respectively;
s22, constructing an image channel for accelerating the network training speed, adopting ResNet network architecture, stacking 3 images ResBlock, and extracting sample characteristics from the traditional reconstructed image;
S23, constructing a merging channel for merging the output of the signal and the image channel, adopting a 2D-UNet network architecture, merging the feature images from the signal channel and the image channel in the channel dimension, and inputting the feature images into the 2D-UNet.
S24, performing supervision training on the data set formed by the S1, inputting a 2.5D photoacoustic signal sinogram and a traditional 2D reconstruction image, wherein a training label is a three-dimensional simulated blood vessel slice in the S11, and outputting a corrected 2D reconstruction image;
and S25, stacking the corrected 2D reconstructed images to obtain a corrected reconstructed three-dimensional body.
S3, constructing a resolution balance reconstruction sub-network and training, wherein the method specifically comprises the following steps:
S31, constructing CycleGAN networks and training: the generator adopts a U-Net architecture, and the discriminator adopts PatchGAN architecture; the data set is composed of an xy slice and an xz slice, wherein the xy slice is a high-resolution image and the xz slice is a low-resolution image due to imaging characteristics of the system, and the xy slice and the xz slice are obtained by dividing a three-dimensional body formed by stacking the corrected 2D reconstructed images in the step S25 along z and y; learning two processes during training, wherein the generator G A learns to retire the high-resolution image into a low-resolution image, and the generator G B learns to restore the low-resolution image into the high-resolution image; only calling a generator G A during testing, and degenerating the xy slice into a low-resolution image, so as to obtain a paired high-definition and low-definition image data pair;
S32, building a Pixel2Pixel network and training: the generator adopts a U-Net architecture, and the discriminator adopts PatchGAN architecture; the dataset is composed of high-definition and low-definition image pairs generated by CycleGAN, two processes are learned during training, the generator G C learns to degenerate a high-resolution image into a low-resolution image, and the generator G D learns to restore the low-resolution image into the high-resolution image; only generator G D is called during testing to restore the xz slice to a high resolution image.
S33, cascading the networks in S31 and S32, performing joint optimization, and obtaining a reconstructed three-dimensional body, wherein the method specifically comprises the following steps: taking the parameter value during independent training as an initial value after cascading, taking the weighted sum of losses of CycleGAN, pixel <2 > Pixel sub-networks as a loss function after cascading, taking the weight as a super-parameter, and adjusting according to the magnitude order obtained by actual calculation; locking parameters of the Pixel2Pixel, and updating parameters of CycleGAN; locking CycleGAN parameters, and updating the parameters of the Pixel2 Pixel; repeating the parameter updating until the loss converges, and outputting a high-resolution image which keeps the original low-resolution slice content during the test;
S34, stacking the xz slices restored to high resolution along the y direction to obtain an isotropic high-resolution three-dimensional body.
S4, constructing a three-dimensional fidelity reconstruction sub-network and training, wherein the method specifically comprises the following steps:
S41, manufacturing a dataset of a photoacoustic forward operator, wherein the processing mode is as follows: according to the proportion of the signal and the image in xyz three directions, the original three-dimensional body is cut off in an equal proportion to obtain a corresponding signal-image three-dimensional voxel with smaller size;
S42, constructing a photoacoustic forward operator, and pre-training by adopting a network architecture of ResNet + full-connection layers to realize photoacoustic forward operation and serve as a part of three-dimensional fidelity reconstruction sub-network loss calculation;
S43, constructing a three-dimensional fidelity reconstruction sub-network by adopting a 3D-UNet network frame, performing supervision training by taking the three-dimensional photoacoustic signal sinogram output by S1 and the isotropic high-resolution three-dimensional body output by S4 as data sets, inputting the three-dimensional body output by S3, outputting the three-dimensional body as a fidelity reconstruction three-dimensional body, performing operation on the output by using a photoacoustic forward operator to obtain a three-dimensional photoacoustic signal sinogram generated by the fidelity reconstruction three-dimensional body, and performing loss calculation by taking the signal sinogram of S1 as a label.
S5, performing network three-dimensional reconstruction on the photoacoustic three-dimensional photoacoustic signal acquired by the photoacoustic tomography system, wherein the network three-dimensional reconstruction comprises the following steps of:
s51, selecting any scanning step distance of 0.1-0.3mm, and performing photoacoustic tomography by adopting a linear ultrasonic array linear scanning mode to obtain a three-dimensional signal matrix;
S52, carrying out delay superposition summation reconstruction on the signal matrix to obtain a preliminary reconstruction three-dimensional body;
S53, inputting the xz slice of the preliminary reconstructed three-dimensional body into a trained Pixel2Pixel network to obtain a high-resolution xz slice;
S54, stacking the high-resolution xz slices along the y direction to obtain a high-resolution three-dimensional body;
and S55, inputting the high-resolution three-dimensional body into a three-dimensional fidelity reconstruction sub-network to obtain a final high-fidelity three-dimensional photoacoustic reconstruction image with balanced resolution.
For purposes of this disclosure, the terms "one embodiment," "some embodiments," "example," "a particular example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims (10)

1. The three-dimensional isotropy high-resolution photoacoustic tomography method based on the deep learning comprises the following steps of:
step S01, manufacturing a simulation data set: performing photoacoustic simulation by using the three-dimensional simulated vessel set to obtain a photoacoustic signal, and forming a photoacoustic data set by using a 2.5D photoacoustic signal sinogram, a 2D traditional reconstructed image and a 2D original image as a group of paired data;
Step S02, a signal correction reconstruction sub-network is constructed and trained: performing supervision training on the data set formed in the step S01, inputting a 2.5D photoacoustic signal sinogram and a traditional 2D reconstructed image, and outputting a corrected 2D reconstructed image;
Step S03, constructing a resolution balance reconstruction sub-network and training: adopting a CycleGAN-Pixel2Pixel network structure, stacking 2D reconstructed images output in the step S02 to form a three-dimensional body, taking two-direction slices for independent training CycleGAN, independently training the Pixel2Pixel by using an image output by CycleGAN, performing cascade connection after independent training for joint optimization, inputting low-resolution slices in the simulated three-dimensional body after joint training is finished to restore the low-resolution slices to high-resolution slices, and stacking the slices to obtain an isotropic-resolution reconstructed three-dimensional body;
Step S04, constructing a three-dimensional fidelity reconstruction sub-network and training: performing supervision training by taking the three-dimensional photoacoustic signal sinogram output in the step S01 and the reconstructed three-dimensional volume output in the step S03 as a data set, inputting the data set into the reconstructed three-dimensional volume output in the step S03, outputting the data set into a fidelity reconstructed three-dimensional volume, performing operation on the output by using a photoacoustic forward operator to obtain a three-dimensional photoacoustic signal sinogram generated by the fidelity reconstructed three-dimensional volume, and performing loss calculation by taking the sinogram in the step S01 as a label;
And S05, inputting the photoacoustic three-dimensional photoacoustic signal matrix acquired by the photoacoustic tomography system and the traditional reconstructed three-dimensional image matrix into the network to obtain a high-fidelity three-dimensional photoacoustic reconstructed image with balanced final resolution.
2. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the three-dimensional simulated blood vessel set is generated according to the branch position, the branch radius and the branch hierarchy parameters of the generated blood vessel set in the actual imaging tissue region.
3. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the photoacoustic simulation is performed by setting an ultrasonic transducer array, an imaging field of view and an imaging grid according to an actual imaging system, setting an imaging sample to generate blood vessels, and converting an actual system scanning step from an actual distance to a grid distance for setting a simulated scanning step.
4. The three-dimensional isotropic high-resolution photoacoustic tomography method based on deep learning according to claim 1, wherein in the photoacoustic simulation, one 2D photoacoustic signal sinogram is obtained through single scanning, and a 3D photoacoustic signal sinogram is obtained through stacking signal sinograms after scanning samples layer by layer; the 2.5D photoacoustic signal sinogram is segmented from the 3D photoacoustic signal sinogram, and the segmentation thickness depends on the thickness of the linear light spot according to an actual system.
5. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the signal correction reconstruction sub-network adopts a full convolutional neural network architecture and is divided into a signal channel, an image channel and a merging channel;
The signal channel is used for correcting a signal sinogram, and specifically comprises 3D full convolution layers, a convolution kernel is designed according to the system sampling rate, the number of ultrasonic transducer array elements and the proportion of scanning steps, and the convolution kernel sizes are 3×35×7, 3×21×5 and 3×13×3 respectively;
The image channel is used for accelerating the network training speed, adopts ResNet network architecture, is formed by stacking 3 images ResBlock, and extracts sample characteristics from the traditional reconstructed image;
The combining channel is used for combining the output of the signal and the image channel, and a 2D-UNet network architecture is adopted to combine the feature images from the signal channel and the image channel in the channel dimension and then input the feature images into the 2D-UNet.
6. The deep learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein CycleGAN and Pixel2 pixels in the resolution balanced reconstruction sub-network are adopted by the generator, a U-Net architecture is adopted by the generator, and a PatchGAN architecture is adopted by the discriminator.
7. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the data set adopted by the resolution equalization reconstruction sub-network is as follows:
The CycleGAN data set is composed of an xy slice and an xz slice obtained by dividing a three-dimensional body formed by stacking the corrected 2D reconstructed images in the step S02 along z and y, wherein due to imaging characteristics of the system, the xy slice is a high-resolution image, and the xz slice is a low-resolution image; learning two processes during training, wherein the generator G A learns to retire the high-resolution image into a low-resolution image, and the generator G B learns to restore the low-resolution image into the high-resolution image; only calling a generator G A during testing, and degenerating the xy slice into a low-resolution image, so as to obtain a paired high-definition and low-definition image data pair;
The Pixel2Pixel data set consists of high-definition image pairs generated by CycleGAN, two processes are learned during training, a generator G C learns to degenerate a high-resolution image into a low-resolution image, and a generator G D learns to restore the low-resolution image into a high-resolution image; only generator G D is called during testing to restore the xz slice to a high resolution image.
8. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein when the network structure of CycleGAN-Pixel2Pixel is subjected to joint tuning, the parameter value during independent training is used as an initial value after cascading, the weighted sum of losses of the CycleGAN, pixel Pixel sub-network is used as a loss function after cascading, the weight is used as a super parameter, and the adjustment is performed according to the magnitude obtained through actual calculation; parameters CycleGAN are updated firstly, parameters of the Pixel2Pixel are locked at the moment, then the parameters of the Pixel2Pixel are updated, at the moment, the parameters of CycleGAN are locked, the parameters are updated once by CycleGAN, the parameters of the Pixel2Pixel are updated once for one cycle, and the process is repeated until the process is repeated, and a high-resolution image which can keep the original low-resolution slice content can be output.
9. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the three-dimensional fidelity reconstruction sub-network adopts a 3D-UNet network architecture; the photoacoustic forward operator adopts a pre-trained network architecture of a 3D-ResNet + full-connection layer, after training is completed, parameters are fixed and used for loss calculation of a three-dimensional fidelity reconstruction sub-network, so that gradient return can be carried out in the loss calculation process.
10. The depth learning-based three-dimensional isotropic high-resolution photoacoustic tomography method of claim 1, wherein the processing method of the three-dimensional reconstruction volume data set in step S05 is as follows: and according to the proportion of the signal and the image in the xyz three directions, the original three-dimensional body is cut off in an equal proportion to obtain corresponding signal-image three-dimensional voxel pairs with smaller sizes, so that the number of network parameters is reduced.
CN202410524431.1A 2024-04-29 2024-04-29 Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning Active CN118115677B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410524431.1A CN118115677B (en) 2024-04-29 2024-04-29 Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410524431.1A CN118115677B (en) 2024-04-29 2024-04-29 Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning

Publications (2)

Publication Number Publication Date
CN118115677A true CN118115677A (en) 2024-05-31
CN118115677B CN118115677B (en) 2024-08-09

Family

ID=91219541

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410524431.1A Active CN118115677B (en) 2024-04-29 2024-04-29 Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning

Country Status (1)

Country Link
CN (1) CN118115677B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning
CN111784721A (en) * 2020-07-01 2020-10-16 华南师范大学 Ultrasonic endoscopic image intelligent segmentation and quantification method and system based on deep learning
WO2021061710A1 (en) * 2019-09-25 2021-04-01 Subtle Medical, Inc. Systems and methods for improving low dose volumetric contrast-enhanced mri
CN115761109A (en) * 2021-09-01 2023-03-07 华中科技大学同济医学院附属协和医院 Deep learning image reconstruction method applied to three-dimensional photoacoustic imaging

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021061710A1 (en) * 2019-09-25 2021-04-01 Subtle Medical, Inc. Systems and methods for improving low dose volumetric contrast-enhanced mri
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning
CN111784721A (en) * 2020-07-01 2020-10-16 华南师范大学 Ultrasonic endoscopic image intelligent segmentation and quantification method and system based on deep learning
CN115761109A (en) * 2021-09-01 2023-03-07 华中科技大学同济医学院附属协和医院 Deep learning image reconstruction method applied to three-dimensional photoacoustic imaging

Also Published As

Publication number Publication date
CN118115677B (en) 2024-08-09

Similar Documents

Publication Publication Date Title
CN110570353B (en) Super-resolution reconstruction method for generating single image of countermeasure network by dense connection
CN110461228B (en) Improving quality of medical images using multi-contrast and deep learning
JP6772112B2 (en) Medical imaging device and medical image processing method
CN114723842B (en) Sparse visual angle CT imaging method and device based on depth fusion neural network
CN113506222B (en) Multi-mode image super-resolution method based on convolutional neural network
JP5600285B2 (en) Ultrasonic image processing device
CN113160380B (en) Three-dimensional magnetic resonance image super-resolution reconstruction method, electronic equipment and storage medium
CN117036162B (en) Residual feature attention fusion method for super-resolution of lightweight chest CT image
CN114187181B (en) Dual-path lung CT image super-resolution method based on residual information refining
CN109766646B (en) Ultrasonic imaging method and device based on sparse channel echo data reconstruction
Liu et al. Medical image super-resolution method based on dense blended attention network
Iddrisu et al. 3D reconstructions of brain from MRI scans using neural radiance fields
CN110706299A (en) Substance decomposition imaging method for dual-energy CT
JP7568471B2 (en) Medical image processing device, trained model, and medical image processing method
CN118115677B (en) Three-dimensional isotropy high-resolution photoacoustic tomography method based on deep learning
CN117523095A (en) Sparse angle THz-CT image reconstruction method based on deep learning
RU2312602C2 (en) Method for producing tomographic images
CN113838161B (en) Sparse projection reconstruction method based on graph learning
Liu et al. A multiscale residual U-net architecture for super-resolution ultrasonic phased array imaging from full matrix capture data
Wu et al. Deep learning-based low-dose tomography reconstruction with hybrid-dose measurements
CN115908610A (en) Method for obtaining attenuation correction coefficient image based on single-mode PET image
CN113052840B (en) Processing method based on low signal-to-noise ratio PET image
CN116090561A (en) Reasoning device and method, medical image diagnosis device, and learning completion neural network generation method
CN115239836A (en) Extreme sparse view angle CT reconstruction method based on end-to-end neural network
CN114862982A (en) Hybrid domain unsupervised finite angle CT reconstruction method based on generation countermeasure network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant