CN111091616A - Method and device for reconstructing three-dimensional ultrasonic image - Google Patents
Method and device for reconstructing three-dimensional ultrasonic image Download PDFInfo
- Publication number
- CN111091616A CN111091616A CN201911167524.9A CN201911167524A CN111091616A CN 111091616 A CN111091616 A CN 111091616A CN 201911167524 A CN201911167524 A CN 201911167524A CN 111091616 A CN111091616 A CN 111091616A
- Authority
- CN
- China
- Prior art keywords
- dimensional
- loss
- image
- layer
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000003042 antagnostic effect Effects 0.000 claims abstract description 8
- 238000001228 spectrum Methods 0.000 claims abstract description 6
- 238000002604 ultrasonography Methods 0.000 claims description 49
- 230000006870 function Effects 0.000 claims description 27
- 230000004913 activation Effects 0.000 claims description 19
- 238000010606 normalization Methods 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 18
- 238000013507 mapping Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 230000003595 spectral effect Effects 0.000 claims description 11
- 230000008439 repair process Effects 0.000 claims description 7
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 5
- 230000001788 irregular Effects 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000003252 repetitive effect Effects 0.000 description 2
- 208000009119 Giant Axonal Neuropathy Diseases 0.000 description 1
- 230000003187 abdominal effect Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 201000003382 giant axonal neuropathy 1 Diseases 0.000 description 1
- 238000002675 image-guided surgery Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/41—Medical
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Image Processing (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
A method and apparatus for reconstructing a three-dimensional ultrasonic image, which is effective for processing an irregularly shaped missing region and is suitable for reconstructing a missing region at an arbitrary position. It includes: interpolating the collected two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence; replacing the conventional convolutional layer with three-dimensional partial convolutional layers, and each partial convolutional layer is followed by a three-dimensional mask updating step; constructing a new spectrum normalized least squares generative confrontation network, wherein the generative confrontation network comprises a generator and a discriminator; combining the content loss and the antagonistic loss to construct a new loss function for the ultrasonic reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature matching loss; and simultaneously inputting the three-dimensional ultrasonic image with the cavity and the three-dimensional mask to a trained generator of the impedance network, and then generating the three-dimensional ultrasonic image after repairing the cavity.
Description
Technical Field
The present invention relates to the field of ultrasound image reconstruction technologies, and in particular, to a three-dimensional ultrasound image reconstruction method and a three-dimensional ultrasound image reconstruction device.
Background
Two-dimensional ultrasound is widely used in medical diagnosis and image-guided surgery because of its non-invasive, non-ionizing, fast imaging, convenient application and low cost. However, two-dimensional ultrasound images do not provide the physician with complete individual data and spatial information of the tissue organ, as compared to three-dimensional ultrasound volumes. Therefore, three-dimensional ultrasound has important clinical value for the diagnosis and treatment of abdominal diseases. Over the last decade, a number of methods have been proposed for reconstructing three-dimensional ultrasound volume data, and handheld three-dimensional ultrasound technology has received increasing attention due to its low cost and flexibility. For hand-held three-dimensional ultrasound techniques, a series of two-dimensional B-ultrasound slices with position and orientation information are acquired, which are scanned and recorded by a two-dimensional ultrasound probe. These B-ultrasound slices can then be used to reconstruct a three-dimensional ultrasound volume.
Classical methods for three-dimensional ultrasound reconstruction are generally based on known information and explore a priori knowledge of the ultrasound volume for repair. For example, the block matching method: searching for near patches within the known region and copying the information of the nearest patches to fill in the holes is known as the most successful repair method at present due to its higher reconstruction quality. However, this method is not suitable for real-time three-dimensional ultrasound reconstruction because the process of searching for patches and optimization takes a lot of time. A multi-resolution three-dimensional ultrasound reconstruction method based on nearest neighbor search is proposed, which constructs a texture and structure pyramid to retain texture and structure information of an ultrasound volume. However, the above reconstruction methods all require that the information of the missing region can be obtained from somewhere in the background region (e.g., similar voxels, structures or patches) of the input ultrasound volume. If the missing region is large and complex, with a non-repetitive structure, and when the missing region has an arbitrary shape, these methods will not synthesize the content of the missing region that is semantically reasonable.
Recent advances in deep neural networks have shown that they can learn semantic priors and meaningful hidden representations in an end-to-end fashion and can therefore be used in natural image restoration work. One has proposed an end-to-end progressive network that divides the repair task into several subtasks and then concatenates all subtasks using long-short term memory. An image restoration method based on a combination of global GAN (Generative adaptive Networks) and Patch GAN (local Generative countermeasure network) is proposed, which can capture local continuity of texture and global features in general. A multi-scale patch approach based on joint optimization of global content and local texture constraints has been proposed, the texture constraints being the use of texture information from non-hole regions to patch hole regions. A two-step convolutional network has been proposed, the first of which is a simple hole convolutional network with a context-aware layer (small blocks generated by convolutional layer processing using the known characteristics of small blocks) to roughly repair the missing regions. The second stage includes global and local discriminators to preserve the structural and detail information of the image. Similarly, a U-Net network with a shift connection layer (similar to the context-aware layer) is proposed, where the coding features of the known regions are shifted to the decoding layer to complete the estimation of the content of the missing regions. However, the limitations of the above methods are that they focus only on the missing regions of the rectangle, and these regions are usually located at the very center of the image, limiting the utility of these models in ultrasound reconstruction.
Disclosure of Invention
In order to overcome the defects of the prior art, the technical problem to be solved by the invention is to provide a method for reconstructing a three-dimensional ultrasonic image, which is good for processing the missing area with an irregular shape and is suitable for reconstructing the missing area at any position.
The technical scheme of the invention is as follows: the method for reconstructing the three-dimensional ultrasonic image comprises the following steps:
(1) interpolating the collected two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
(2) replacing the conventional convolutional layer with three-dimensional partial convolutional layers, and each partial convolutional layer is followed by a three-dimensional mask updating step;
(3) constructing a spectrum normalization least square generation type countermeasure network, wherein the generation type countermeasure network comprises a generator and a discriminator;
(4) combining the content loss and the antagonistic loss to construct a new loss function for the ultrasonic reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
(5) and simultaneously inputting the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image into a generator of the trained impedance network, and then generating the three-dimensional ultrasonic image after repairing the cavity.
The invention interpolates a two-dimensional B-ultrasonic slice sequence into a three-dimensional image with a hole according to the space position of the two-dimensional B-ultrasonic slice sequence, replaces the traditional convolution layer with three-dimensional partial convolution layers, and each partial convolution layer is followed by a three-dimensional mask updating step to construct a new spectrum normalized least square generation countermeasure network, combines the content loss and the countermeasure loss, simultaneously inputs a three-dimensional ultrasonic image with the hole and a three-dimensional mask image to a generator of the trained countermeasure network, and then generates a three-dimensional ultrasonic image after the hole is repaired, thereby having good effect on processing the missing area with the irregular shape and being suitable for the reconstruction of the missing area at any position.
There is also provided an apparatus for reconstructing a three-dimensional ultrasound image, the apparatus comprising:
the interpolation module is configured to interpolate the acquired two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
a three-dimensional partial convolution and three-dimensional mask update module configured to replace a conventional convolutional layer with a three-dimensional partial convolutional layer, and each partial convolutional layer is followed by a three-dimensional mask update step;
a build network module configured to build a spectral normalized least squares generating confrontation network, the generating confrontation network comprising a generator and a discriminator;
a loss combining module configured to combine the content loss and the antagonistic loss to construct a new loss function for the ultrasound reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
and the restoration module is configured to simultaneously input the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image to the trained generator of the impedance network and then generate the three-dimensional ultrasonic image after the cavity is restored.
Drawings
Fig. 1 is a flowchart illustrating step (1) of the method for reconstructing a three-dimensional ultrasound image according to the present invention.
Fig. 2 is a flow chart of a method for reconstructing a three-dimensional ultrasound image according to the present invention.
Fig. 3 is a concrete network architecture of an encoder in a generator of the reconstruction method of a three-dimensional ultrasound image according to the present invention.
Fig. 4 is a flowchart of a method of reconstructing a three-dimensional ultrasound image according to the present invention.
Detailed Description
As shown in fig. 4, the method for reconstructing a three-dimensional ultrasound image includes the following steps:
(1) interpolating the collected two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
(2) replacing the conventional convolutional layer with three-dimensional partial convolutional layers, and each partial convolutional layer is followed by a three-dimensional mask updating step;
(3) constructing a new spectrum normalized least squares generative confrontation network, wherein the generative confrontation network comprises a generator and a discriminator;
(4) combining the content loss and the antagonistic loss to construct a new loss function for the ultrasonic reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
(5) and simultaneously inputting the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image into a generator of the trained impedance network, and then generating the three-dimensional ultrasonic image after repairing the cavity.
The invention interpolates a two-dimensional B-ultrasonic slice sequence into a three-dimensional image with a hole according to the space position of the two-dimensional B-ultrasonic slice sequence, replaces the traditional convolution layer with three-dimensional partial convolution layers, and each partial convolution layer is followed by a three-dimensional mask updating step to construct a new spectrum normalized least square generation type countermeasure network, combines the content loss and the countermeasure loss, simultaneously inputs a three-dimensional ultrasonic image with the hole and a three-dimensional mask image to a generator of the trained countermeasure network, and then generates a three-dimensional ultrasonic image after the hole is repaired, thereby having good effect on processing the missing area with the irregular shape and being suitable for the reconstruction of the missing area at any position.
Preferably, in the step (1), an individual ultrasonic slice sequence with spatial positioning information is acquired, and 3D volume data is interpolated according to the spatial positioning information between the sequences; the interpolation process is as follows: firstly, establishing an empty three-dimensional body with a space coordinate system, wherein the space coordinate system comprises an original point, a size and a body grid interval; and mapping the pixels in the two-dimensional ultrasonic image to the corresponding voxels near the three-dimensional body according to the spatial position information.
Preferably, in the step (2), the partial convolutional layer is defined by the formula (1):
where W is the weight of the convolution filtering, b is the corresponding deviation, X represents the eigenvalue of the current convolution window, M is the corresponding binary mask, 1 represents that the voxel at position (X, y, z) is valid, 0 represents that the voxel at (X, y, z) is invalid; the convolution output values depend only on the unmasked input, and a scaling factor of 1/sum (M) is applied to adjust the variation of the unmasked input, and given a three-dimensional binary mask, the three-dimensional convolution result depends only on the content of the known area of each layer;
the mask update is performed after each partial convolution operation, and is expressed as equation (2):
if the convolution is able to adjust its output over at least one valid input value, the mask for that location is deleted; if the input includes any valid voxels, with full application of the partial convolutional layer, even the larger mask area will shrink, and any mask will eventually be all 1's.
Preferably, in the step (2), all normal convolutions are replaced by three-dimensional partial convolutions, the image is transmitted together with the mask through the network, and a residual block structure and jump connection are introduced into a decoder of a 3D U-Net network architecture; all convolutional layers were convolved with a 3 x 3 convolution kernel, using a three-dimensional learnelu activation layer with alpha 0.2 at the decoder stage; the three-dimensional Relu activation layer is used in all coding layers and all layers of the discriminator; except the first layer and the last layer, a three-dimensional standardization layer is used between each three-dimensional partial convolution layer and the three-dimensional LEAKyrelu activation layer and between each three-dimensional partial convolution layer and the three-dimensional Relu activation layer; in the decoding stage, the three-dimensional normalization layer is followed by a droupout layer with a rate of 0.5 to prevent overfitting of the training data. Defining a mask with the size of DxHxWxC, wherein the mask and the image have the same size, and then realizing the process of updating the mask by utilizing a fixed layer, wherein the size of a convolution kernel of the layer is the same as that of a partial convolution operation, but the weight is set to be 1, and the offset is set to be 0; all three-dimensional dropout layers, three-dimensional leakyrelu activation layers and three-dimensional Relu activation layers only act on partial convolution operations and do not act on mask update layers. Different learning rates are set for the generator and the discriminator.
Preferably, the training of the generator and discriminator network is further stabilized in said step (3) using a three-dimensional spectral normalization method, which is a method aiming at controlling the Lipschitz constants of the generator and discriminator by normalizing the weight of each convolution layer in the network, the new weights after normalization
Is defined as:
where W is the weight of each layer, σ (W) is the largest singular value of the weight; the method applies global regularization to the discriminators and generators.
Preferably, in the step (4), the image I with the missing region is giveninInitial binary mask M, 0 representing a hole, generating image IoutAnd gold standard image Igt；
First a context loss L is definedcTo receive an input image IinThe remaining available information is captured, the context loss is based on the assumption that voxels far from the missing region are less important to the repair process, and the importance W is defined as formula (4):
where i denotes the index of the voxel, WiRepresenting the importance of the voxel at position i, n (i) representing the neighborhood set of voxel i in the local window, and the cardinality of n (i) at | n (i) |;
context loss LcIs defined as:
Lc＝||W⊙(Iout-Igt)||1(5)
where ⊙ denotes the multiplication of elements by elements.
Total variation loss LtvIs equation (6), is a smoothing penalty on the generated P, where P is the 1 voxel dilated region of the missing region,
wherein, IcompIs to generate an output image IoutBut directly replacing the pixels of the non-missing region with the corresponding gold standard image IgtThe voxel (b);
loss of feature matching LFMDefined by formula (7):
where L is the last layer of the discriminator, NiIs the total number of elements of the ith layer,
is the activation mapping of the arbiter ith layer;
against loss LGAN(Gsn，Dsn) The equation (8) is calculated by the least square GAN introduced in the training phase, and arg min is solved by training the generator and the discriminator simultaneouslyGmaxDLGAN(Gsn，Dsn) Obtaining:
wherein D issnDiscriminator indicating spectral normalization, GsnRepresenting a spectrally normalized ultrasound generator, inputs an ultrasound volume y with a missing region.
Preferably, in the step (5), the training process for generating the antagonistic network is as follows: obtaining a content loss function of the real ultrasonic image according to the output result of the generator and the corresponding real ultrasonic image; based on the least square loss function of a generator and a discriminator in the generated countermeasure network, obtaining the total loss function of the generator according to the output result of the discriminator and the content loss function of the real ultrasonic image, and obtaining the total loss function of the discriminator according to the output result of the discriminator; and respectively updating parameters in the network structures of the arbiter and the generator according to the total loss function of the arbiter and the total loss function of the generator until the generation of the confrontation network convergence.
It will be understood by those skilled in the art that all or part of the steps in the method of the above embodiments may be implemented by hardware instructions related to a program, the program may be stored in a computer-readable storage medium, and when executed, the program includes the steps of the method of the above embodiments, and the storage medium may be: ROM/RAM, magnetic disks, optical disks, memory cards, and the like. Therefore, corresponding to the method of the present invention, the present invention also includes a three-dimensional ultrasound image reconstruction apparatus, which is generally represented in the form of functional blocks corresponding to the steps of the method. The device includes:
the interpolation module is configured to interpolate the acquired two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
a three-dimensional partial convolution and three-dimensional mask update module configured to replace a conventional convolutional layer with a three-dimensional partial convolutional layer, and each partial convolutional layer is followed by a three-dimensional mask update step;
a build network module configured to build a new spectral normalized least squares generating confrontation network, the generating confrontation network comprising a generator and a discriminator;
a loss combining module configured to combine the content loss and the antagonistic loss to construct a new loss function for the ultrasound reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
and the restoration module is configured to simultaneously input the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image to the trained generator of the impedance network and then generate the three-dimensional ultrasonic image after the cavity is restored.
The present invention is described in more detail below.
And acquiring ultrasonic slice sequences with spatial positioning information of the same patient, and interpolating into 3D volume data according to the spatial information between the sequences. The interpolation process first creates an empty three-dimensional volume with a spatial coordinate system (including origin, size and volume grid spacing). After the grid is constructed, the pixels in the two-dimensional ultrasonic image are mapped to the corresponding voxels near the three-dimensional body according to the spatial position information. The coordinate relationship between the ultrasound slice and the three-dimensional volume is shown in figure 1. Due to the subjectivity of handheld scans, the two-dimensional ultrasound images collected are typically highly sparse. Thus, after the above voxel mapping is completed, there will be some gaps in the three-dimensional ultrasound volume, as shown in the right part of fig. 1. The invention aims to fill the gap part by using a three-dimensional least square generation countermeasure network, and the three-dimensional least square generation countermeasure network not only can fill the gap by using the information of the known voxels, but also can synthesize non-repetitive textures and structures.
The definition of the partial convolutional layer is as follows:
where W is the weight of the convolution filtering, b is the corresponding deviation, and X represents the eigenvalue of the current convolution window. M is the corresponding binary mask, 1 indicates that the voxel at position (x, y, z) is valid, and 0 indicates that the voxel at (x, y, z) is invalid. From equation (1), it can be seen that the output value of the convolution depends only on the unmasked input. The scaling factor 1/sum (M) is applied to adjust the amount of change in the unmasked input. Given a three-dimensional binary mask, the three-dimensional convolution results of the present invention depend only on the contents of the known regions of each layer.
After each partial convolution operation we perform an update of the mask, which is expressed as:
if the convolution is able to adjust its output over at least one valid input value, the mask at that location is removed. This process may be implemented in any network architecture. If the input includes any valid voxels, with sufficient application of the partial convolution layer, even the larger mask area will shrink and any mask will eventually be updated to 1.
The proposed network architecture is a network architecture similar to 3D U-Net in which all normal convolutions are replaced by three-dimensional partial convolutions, so in this case the image is passed through the network together with the mask, and figure 2 provides the architecture of the entire network. In the decoder of the 3D U-Net network architecture, a residual block structure and a jump connection are introduced, as shown in fig. 3, similar to the structure of the SRGAN. Hopping connections mitigates the network architecture that models self-mapping, which may not be easy to represent using a convolution kernel. All convolutional layers were convolved with a convolution kernel of 3 x 3, using a three-dimensional learnelu activation layer with alpha 0.2 at the decoder stage. The three-dimensional Relu activation layer is used in all coding layers and all layers of the arbiter. Three-dimensional normalization layers were used between each three-dimensional partial convolution layer and the three-dimensional LeakyRelu/Relu layers, except for the first and last layers. In the decoding stage, the three-dimensional normalization layer is followed by a droupout layer (to counteract the internal covariate shift) at a rate of 0.5 to prevent overfitting of the training data.
A mask of size D × H × W × C is defined, and the same size as the image is obtained, and then the process of updating the mask is implemented using a fixed layer whose convolution kernel size is the same as that of the partial convolution operation, but with a weight set to 1 and an offset set to 0. It is noted that all three-dimensional dropout layers, the three-dimensional leakrelu/Relu layers, only act on a partial convolution operation and not on the mask update layer.
In training the network, different learning rates may be set for the generator and the arbiter in order to balance the training speeds of both the generator and the arbiter.
The training of the generator and the discriminator network is further stabilized by a three-dimensional spectral normalization method. Three-dimensional spectral normalization is a method that aims at controlling the Lipschitz constants (assuming statistical bounding) of the generator and the arbiter by normalizing the weights of each convolutional layer in the network. The new weights after normalization are defined as:
where W is the weight of each layer and σ (W) is the largest singular value of the weight. Unlike other weight normalizations, spectral normalization allows the parameter matrix to use as many features as possible while satisfying the local 1-lipschitz constraint, and can improve the quality of the generated image. The method applies global regularization to the discriminators and generators and can be easily combined with least squares generation of competing networks or other forms of GANs.
In order to improve the quality of the reconstructionContent loss is designed for the training of the generator, and consists of three parts, namely context loss, total variation loss and feature matching loss. Given an image I with missing regionsinInitial binary mask M (0 representing a hole), generating image IoutAnd gold standard image Igt. First a context loss L is definedcTo receive an input image IinThe remaining available information is captured. The context loss is based on the assumption that voxels far from the missing region are less important to the repair process. Defining the importance W:
where i denotes the index of the voxel, WiRepresenting the importance of the voxel at position i, n (i) representing the neighborhood set of voxel i in the local window, and the cardinality of n (i) at | n (i) |. Context loss LcIs defined as:
Lc＝||W⊙(Iout-Igt)||1(5)
where ⊙ denotes the multiplication of elements by elements.
Next, the total variation loss L is definedtv. The total variation loss is a smooth penalty on the generated P, where P is the 1 voxel dilated region of the missing region.
Wherein, IcompIs to generate an output image IoutBut directly replacing the pixels of the non-missing region with the corresponding gold standard image IgtThe voxel (2).
The final loss is the feature matching loss LFMSimilar to the perceptual loss of perceptual differences between measured images. The loss-aware activation map is computed using a pre-trained VGG19 network. The feature matching penalty comparison is the activation map of the middle layer of the discriminator, as shown in FIG. 2. Measuring the difference between the output image and the real image and forcing the generator to generate classesSimilar to the output of a real output. With the context loss and the total variation loss, the intrinsic similarity between ultrasound volumes is not measured, but only their surface differences in euclidean distance. By comparing the internal structures of the ultrasound volume, they should be projected onto the manifold and their geodesic distances calculated, so exploiting the feature matching penalty helps to produce a more sharp detailed reconstruction result. Loss of feature matching LFMIs defined as:
where L is the last layer of the discriminator, NiIs the total number of elements of the ith layer,
is the activation map for the i-th layer of the arbiter.
The resistance loss is calculated by least square GAN introduced in the training phase, and argmin is solved by simultaneously training the generator and the discriminatorGmaxDLGAN(Gsn，Dsn) Obtaining:
wherein D issnDiscriminator indicating spectral normalization, GsnRepresenting a spectrally normalized ultrasound generator, inputs an ultrasound volume y with a missing region.
Therefore, the training process for generating the countermeasure network is to obtain a content loss function of the real ultrasonic image according to the output result of the generator and the corresponding real ultrasonic image; based on the least square loss function of a generator and a discriminator in the generated countermeasure network, obtaining the total loss function of the generator according to the output result of the discriminator and the content loss function of the real ultrasonic image, and obtaining the total loss function of the discriminator according to the output result of the discriminator; and respectively updating parameters in the network structures of the arbiter and the generator according to the total loss function of the arbiter and the total loss function of the generator until the generation of the confrontation network convergence.
Generating an impedance network by the deep network learning method, simultaneously inputting a three-dimensional ultrasonic image with a cavity and a three-dimensional mask image into a generator of the trained impedance network, and then generating a three-dimensional ultrasonic image after repairing the cavity, wherein the synthesized ultrasonic image is a highly simulated ultrasonic image with a real ultrasonic image.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and all simple modifications, equivalent variations and modifications made to the above embodiment according to the technical spirit of the present invention still belong to the protection scope of the technical solution of the present invention.
Claims (7)
1. A method for reconstructing a three-dimensional ultrasound image, comprising: which comprises the following steps:
(1) interpolating the collected two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
(2) replacing the conventional convolutional layer with three-dimensional partial convolutional layers, and each partial convolutional layer is followed by a three-dimensional mask updating step;
(3) constructing a new spectrum normalization least square generation type countermeasure network, wherein the generation type countermeasure network comprises a generator and a discriminator;
(4) combining the content loss and the antagonistic loss to construct a new loss function for the ultrasonic reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
(5) and simultaneously inputting the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image into a generator of the trained impedance network, and then generating the three-dimensional ultrasonic image after repairing the cavity.
2. The method for reconstructing a three-dimensional ultrasound image according to claim 1, wherein: in the step (1), an individual ultrasonic slice sequence with spatial positioning information is collected, and 3D volume data is obtained through interpolation according to the spatial positioning information between the sequences; the interpolation process is as follows: firstly, establishing an empty three-dimensional body with a space coordinate system, wherein the space coordinate system comprises an original point, a size and a body grid interval; and mapping the pixels in the two-dimensional ultrasonic image to the corresponding voxels near the three-dimensional body according to the spatial position information.
3. The method for reconstructing a three-dimensional ultrasound image according to claim 1, wherein: in the step (2), the definition of the partial convolution layer is represented by formula (1):
where W is the weight of the convolution filtering, b is the corresponding deviation, X represents the eigenvalue of the current convolution window, M is the corresponding binary mask, 1 represents that the voxel at position (X, y, z) is valid, 0 represents that the voxel at (X, y, z) is invalid; the convolution output values depend only on the unmasked input, and a scaling factor of 1/sum (M) is applied to adjust the variation of the unmasked input, and given a three-dimensional binary mask, the three-dimensional convolution result depends only on the content of the known area of each layer;
the mask is updated after each partial convolution operation, the mask update step being represented by equation (2):
if the convolution is able to adjust its output over at least one valid input value, the mask for that location is deleted; if the input includes any valid voxels, with sufficient application of the partial convolution layer, even the larger mask area will shrink and any mask will eventually be updated to 1.
4. The method for reconstructing a three-dimensional ultrasound image according to claim 1, wherein: in the step (3), all normal convolutions are replaced by three-dimensional partial convolutions, and the image is transmitted into the image through the network together with the maskThe generator introduces a residual block structure and a jump connection in a decoder of a 3D U-Net network architecture; all convolutional layers were convolved with a 3 x 3 convolution kernel, using a three-dimensional learnelu activation layer with alpha 0.2 at the decoder stage; the three-dimensional Relu activation layer is used in all coding layers and all layers of the discriminator; except the first layer and the last layer, a three-dimensional standardization layer is used between each three-dimensional partial convolution layer and the three-dimensional LEAKyrelu activation layer and between each three-dimensional partial convolution layer and the three-dimensional Relu activation layer; in the decoding stage, the three-dimensional normalization layer is followed by a droupout layer with a rate of 0.5 to prevent overfitting of the training data. Defining a mask with the size of DxHxWxC, wherein the mask and the image have the same size, and then realizing the process of updating the mask by utilizing a fixed layer, wherein the size of a convolution kernel of the layer is the same as that of a partial convolution operation, but the weight is set to be 1, and the offset is set to be 0; all three-dimensional dropout layers, three-dimensional leakyrelu activation layers and three-dimensional Relu activation layers only act on partial convolution operations and do not act on mask update layers. Different learning rates are set for the generator and the discriminator. Training of the generator and discriminator networks is further stabilized using three-dimensional spectral normalization, a method aimed at controlling the Lipschitz constants of the generator and discriminator by normalizing the weight of each convolutional layer in the network, the new weights after normalization
Is defined as:
where W is the weight of each layer, σ (W) is the largest singular value of the weight; the method applies global regularization to the discriminators and generators.
5. The method for reconstructing a three-dimensional ultrasound image according to claim 1, wherein: in the step (4), the step of (C),
given an image I with missing regionsinInitial binary mask M, 0 representing a hole, generating image IoutAnd gold standard image Igt；
First a context loss L is definedcTo receive an input image IinThe remaining available information is captured, the context loss is based on the assumption that voxels far from the missing region are less important to the repair process, and the importance W is defined as formula (4):
where i denotes the index of the voxel, WiRepresenting the importance of the voxel at position i, n (i) representing the neighborhood set of voxel i in the local window, and the cardinality of n (i) at | n (i) |;
context loss LcIs defined as:
Lc＝||W⊙(Iout-Igt)||1(5)
where ⊙ denotes the multiplication of elements by elements.
Total variation loss itvIs equation (6), is a smoothing penalty on the generated P, where P is the 1 voxel dilated region of the missing region,
wherein, IcompIs to generate an output image IoutBut directly replacing the pixels of the non-missing region with the corresponding gold standard image IgtThe voxel (b);
loss of feature matching LFMDefined by formula (7):
where L is the last layer of the discriminator, NiIs the total number of elements of the ith layer,
is the activation mapping of the arbiter ith layer;
against loss LGAN(Gsn,Dsn) The equation (8) is calculated by the least square GAN introduced in the training phase, and arg min is solved by training the generator and the discriminator simultaneouslyGmaxDLGAN(Gsn,Dsn) Obtaining:
wherein D issnDiscriminator indicating spectral normalization, GsnRepresenting a spectrally normalized ultrasound generator, inputs an ultrasound volume y with a missing region.
6. The method for reconstructing a three-dimensional ultrasound image according to claim 1, wherein: in the step (5), the training process for generating the countermeasure network is as follows: obtaining a content loss function of the real ultrasonic image according to the output result of the generator and the corresponding real ultrasonic image; based on the least square loss function of a generator and a discriminator in the generated countermeasure network, obtaining the total loss function of the generator according to the output result of the discriminator and the content loss function of the real ultrasonic image, and obtaining the total loss function of the discriminator according to the output result of the discriminator; and respectively updating parameters in the network structures of the arbiter and the generator according to the total loss function of the arbiter and the total loss function of the generator until the generation of the confrontation network convergence.
7. An apparatus for reconstructing a three-dimensional ultrasound image, comprising: the device includes:
the interpolation module is configured to interpolate the acquired two-dimensional B-ultrasonic slice sequence with the spatial positioning information into a three-dimensional image with a cavity according to the spatial position of the two-dimensional B-ultrasonic slice sequence;
a three-dimensional partial convolution and three-dimensional mask update module configured to replace a conventional convolutional layer with a three-dimensional partial convolutional layer, and each partial convolutional layer is followed by a three-dimensional mask update step;
a build network module configured to build a spectral normalized least squares generating confrontation network, the generating confrontation network comprising a generator and a discriminator;
a loss combining module configured to combine the content loss and the antagonistic loss to construct a new loss function for the ultrasound reconstruction, wherein the content loss comprises: context loss, total variation loss, and feature mapping loss;
and the restoration module is configured to simultaneously input the three-dimensional ultrasonic image with the cavity and the three-dimensional mask image to the trained generator of the impedance network and then generate the three-dimensional ultrasonic image after the cavity is restored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911167524.9A CN111091616B (en) | 2019-11-25 | 2019-11-25 | Reconstruction method and device of three-dimensional ultrasonic image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911167524.9A CN111091616B (en) | 2019-11-25 | 2019-11-25 | Reconstruction method and device of three-dimensional ultrasonic image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111091616A true CN111091616A (en) | 2020-05-01 |
| CN111091616B CN111091616B (en) | 2024-01-05 |
Family
ID=70393163
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911167524.9A Active CN111091616B (en) | 2019-11-25 | 2019-11-25 | Reconstruction method and device of three-dimensional ultrasonic image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111091616B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111815764A (en) * | 2020-07-21 | 2020-10-23 | 西北工业大学 | Ultrasonic three-dimensional reconstruction method based on self-supervision 3D full convolution neural network |
| CN112508821A (en) * | 2020-12-21 | 2021-03-16 | 南阳师范学院 | Stereoscopic vision virtual image hole filling method based on directional regression loss function |
| CN112614070A (en) * | 2020-12-28 | 2021-04-06 | 南京信息工程大学 | DefogNet-based single image defogging method |
| CN113034388A (en) * | 2021-03-12 | 2021-06-25 | 西北大学 | Ancient painting virtual repairing method and construction method of repairing model |
| CN114463480A (en) * | 2020-11-09 | 2022-05-10 | 北京理工大学 | Ultrasonic volume reconstruction method and device based on pose parameter regularization |
| CN114723611A (en) * | 2022-06-10 | 2022-07-08 | 季华实验室 | Image reconstruction model training method, reconstruction method, device, equipment and medium |
| CN116152441A (en) * | 2023-03-21 | 2023-05-23 | 电子科技大学 | Multi-resolution U-net curved surface reconstruction method based on depth priori |
| CN116958468A (en) * | 2023-07-05 | 2023-10-27 | 中国科学院地理科学与资源研究所 | Mountain snow environment simulation method and system based on SCycleGAN |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN109462747A (en) * | 2018-12-11 | 2019-03-12 | 成都美律科技有限公司 | Based on the DIBR system gap filling method for generating confrontation network |
| CN109636905A (en) * | 2018-12-07 | 2019-04-16 | 东北大学 | Environment semanteme based on depth convolutional neural networks builds drawing method |
| CN110223370A (en) * | 2019-05-29 | 2019-09-10 | 南京大学 | A method of complete human body's texture mapping is generated from single view picture |
| CN110381845A (en) * | 2017-01-05 | 2019-10-25 | 皇家飞利浦有限公司 | Ultrasonic image-forming system with the neural network for exporting imaging data and organizational information |
| CN110443867A (en) * | 2019-08-01 | 2019-11-12 | 太原科技大学 | Based on the CT image super-resolution reconstructing method for generating confrontation network |
| CN110458939A (en) * | 2019-07-24 | 2019-11-15 | 大连理工大学 | The indoor scene modeling method generated based on visual angle |
-
2019
- 2019-11-25 CN CN201911167524.9A patent/CN111091616B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN110381845A (en) * | 2017-01-05 | 2019-10-25 | 皇家飞利浦有限公司 | Ultrasonic image-forming system with the neural network for exporting imaging data and organizational information |
| CN109636905A (en) * | 2018-12-07 | 2019-04-16 | 东北大学 | Environment semanteme based on depth convolutional neural networks builds drawing method |
| CN109462747A (en) * | 2018-12-11 | 2019-03-12 | 成都美律科技有限公司 | Based on the DIBR system gap filling method for generating confrontation network |
| CN110223370A (en) * | 2019-05-29 | 2019-09-10 | 南京大学 | A method of complete human body's texture mapping is generated from single view picture |
| CN110458939A (en) * | 2019-07-24 | 2019-11-15 | 大连理工大学 | The indoor scene modeling method generated based on visual angle |
| CN110443867A (en) * | 2019-08-01 | 2019-11-12 | 太原科技大学 | Based on the CT image super-resolution reconstructing method for generating confrontation network |
Non-Patent Citations (1)
| Title |
|---|
| AI DAN NI: "《Multiresolution generalized N dimension PCA for ultrasound image denoising》" * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111815764A (en) * | 2020-07-21 | 2020-10-23 | 西北工业大学 | Ultrasonic three-dimensional reconstruction method based on self-supervision 3D full convolution neural network |
| CN114463480A (en) * | 2020-11-09 | 2022-05-10 | 北京理工大学 | Ultrasonic volume reconstruction method and device based on pose parameter regularization |
| CN112508821A (en) * | 2020-12-21 | 2021-03-16 | 南阳师范学院 | Stereoscopic vision virtual image hole filling method based on directional regression loss function |
| CN112508821B (en) * | 2020-12-21 | 2023-02-24 | 南阳师范学院 | Stereoscopic vision virtual image hole filling method based on directional regression loss function |
| CN112614070A (en) * | 2020-12-28 | 2021-04-06 | 南京信息工程大学 | DefogNet-based single image defogging method |
| CN112614070B (en) * | 2020-12-28 | 2023-05-30 | 南京信息工程大学 | defogNet-based single image defogging method |
| CN113034388A (en) * | 2021-03-12 | 2021-06-25 | 西北大学 | Ancient painting virtual repairing method and construction method of repairing model |
| CN113034388B (en) * | 2021-03-12 | 2024-04-05 | 西北大学 | Ancient painting virtual repair method and construction method of repair model |
| CN114723611A (en) * | 2022-06-10 | 2022-07-08 | 季华实验室 | Image reconstruction model training method, reconstruction method, device, equipment and medium |
| CN116152441A (en) * | 2023-03-21 | 2023-05-23 | 电子科技大学 | Multi-resolution U-net curved surface reconstruction method based on depth priori |
| CN116152441B (en) * | 2023-03-21 | 2023-11-14 | 电子科技大学 | Multi-resolution U-net curved surface reconstruction method based on depth priori |
| CN116958468A (en) * | 2023-07-05 | 2023-10-27 | 中国科学院地理科学与资源研究所 | Mountain snow environment simulation method and system based on SCycleGAN |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111091616B (en) | 2024-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111091616B (en) | Reconstruction method and device of three-dimensional ultrasonic image | |
| US11756160B2 (en) | ML-based methods for pseudo-CT and HR MR image estimation | |
| Abdi et al. | Automatic quality assessment of echocardiograms using convolutional neural networks: feasibility on the apical four-chamber view | |
| US9153047B1 (en) | Systems and methods for data and model-driven image reconstruction and enhancement | |
| CN110648337A (en) | Hip joint segmentation method, hip joint segmentation device, electronic apparatus, and storage medium | |
| CN110599528A (en) | Unsupervised three-dimensional medical image registration method and system based on neural network | |
| JP2023550844A (en) | Liver CT automatic segmentation method based on deep shape learning | |
| CN116402865B (en) | Multi-mode image registration method, device and medium using diffusion model | |
| CN113112559A (en) | Ultrasonic image segmentation method and device, terminal equipment and storage medium | |
| CN112036506A (en) | Image recognition method and related device and equipment | |
| CN116188452A (en) | Medical image interlayer interpolation and three-dimensional reconstruction method | |
| CN115830016B (en) | Medical image registration model training method and equipment | |
| CN111260667A (en) | Neurofibroma segmentation method combined with space guidance | |
| CN115239716A (en) | Medical image segmentation method based on shape prior U-Net | |
| CN114119474A (en) | Method for automatically segmenting human tissues in ultrasonic image through deep learning | |
| CA2873918C (en) | Method and system for the three-dimensional reconstruction of structures | |
| Habijan et al. | Generation of artificial CT images using patch-based conditional generative adversarial networks | |
| Chen et al. | Medprompt: Cross-modal prompting for multi-task medical image translation | |
| Dong et al. | Hole-filling based on content loss indexed 3D partial convolution network for freehand ultrasound reconstruction | |
| CN116152235A (en) | Cross-modal synthesis method for medical image from CT (computed tomography) to PET (positron emission tomography) of lung cancer | |
| CN116363332A (en) | Prostate ultrasonic correction method based on medical image segmentation and feature matching | |
| Li et al. | Hrinet: Alternative supervision network for high-resolution ct image interpolation | |
| CN114092643A (en) | Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN | |
| Bermejo et al. | Coral reef optimization for intensity-based medical image registration | |
| Zyuzin et al. | Generation of echocardiographic 2D images of the heart using cGAN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Cong Weijian Inventor after: Wu Chan Inventor after: Dong Jiahui Inventor before: Cong Weijian Inventor before: Wu Chan Inventor before: Dong Jiahui |
|
| CB03 | Change of inventor or designer information | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210929 Address after: 100081 0810 Haidian Science and technology building, Zhongguancun South Street, Haidian District, Beijing Applicant after: ARIEMEDI MEDICAL SCIENCE (BEIJING) Co.,Ltd. Address before: 050000 3rd floor, unit 1, building 7, Runjiang international headquarters, 319 Changjiang Avenue, Yuhua District, Shijiazhuang City, Hebei Province Applicant before: AIRUI MAIDI TECHNOLOGY SHIJIAZHUANG Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |