CN112700534A - Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration - Google Patents
Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration Download PDFInfo
- Publication number
- CN112700534A CN112700534A CN202011623215.0A CN202011623215A CN112700534A CN 112700534 A CN112700534 A CN 112700534A CN 202011623215 A CN202011623215 A CN 202011623215A CN 112700534 A CN112700534 A CN 112700534A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- tensor
- loss
- multiplied
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a three-dimensional reconstruction method of an ultrasonic or CT medical image based on feature migration, which is characterized in that unsupervised learning is adopted, a visual method is utilized according to the characteristics of medical image acquisition, a feature migration principle is adopted, and a mechanism of sharing network parameters is adopted, so that the feature migration from the ultrasonic image to the CT image is realized, the three-dimensional reconstruction of the ultrasonic image and the CT image can be realized, the three-dimensional reconstruction of the ultrasonic or CT image can be effectively realized by utilizing the method, and a quick 3D visual reconstruction result can be provided in artificial intelligent auxiliary diagnosis, so that the auxiliary diagnosis efficiency is improved.
Description
Technical Field
The invention belongs to the technology of related ultrasonic or CT image intelligent auxiliary diagnosis in the technical field of computers, and relates to a three-dimensional reconstruction method for auxiliary diagnosis.
Background
In recent years, artificial intelligence technology is rapidly developed, and the significance of the key technology research of medical auxiliary diagnosis is great. At present, in the study of the three-dimensional reconstruction technology of the medical image, because the parameter recovery of the camera has certain difficulty, the study of the three-dimensional reconstruction technology of the medical image has certain difficulty, and particularly, the reconstruction of a complex model can bring a serious problem of high time complexity to the three-dimensional reconstruction, which is not beneficial to the application of clinical medical auxiliary diagnosis. How to establish an effective deep learning network coding model and effectively solve the rapid problem of three-dimensional reconstruction of multi-modal images, which is a practical problem to be solved urgently.
Disclosure of Invention
The invention provides an ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration.
The specific technical scheme for realizing the aim of the invention is as follows:
a ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration inputs an ultrasonic or CT image sequence, the image resolution is MxN, M is more than or equal to 100 and less than or equal to 1500, N is more than or equal to 100 and less than or equal to 1500, the three-dimensional reconstruction process specifically comprises the following steps:
step 1: building a data set
(a) Constructing a natural image dataset D
Selecting a natural image website, requiring image sequences and corresponding internal parameters of a camera, downloading a image sequences and the corresponding internal parameters of the sequences from the natural image website, wherein a is more than or equal to 1 and less than or equal to 20, for each image sequence, recording every 3 adjacent frames of images as an image b, an image c and an image d, splicing the image b and the image d according to a color channel to obtain an image tau, forming a data element by the image c and the image tau, wherein the image c is a natural target image, the sampling viewpoint of the image c is used as a target viewpoint, and the internal parameters of the image b, the image c and the image d are all et(t ═ 1, 2, 3, 4) in which e1Is a horizontal focal length, e2Is a vertical focal length, e3And e4Are two components of the principal point coordinates; if the last remaining image in the same image sequence is less than 3 framesThen abandon; constructing a data set D by using all the sequences, wherein the data set D has f elements, and f is more than or equal to 3000 and less than or equal to 20000;
(b) constructing an ultrasound image dataset E
Sampling g ultrasonic image sequences, wherein g is more than or equal to 1 and less than or equal to 20, recording every adjacent 3 frames of images of each sequence as an image i, an image j and an image k, splicing the image i and the image k according to a color channel to obtain an image pi, forming a data element by the image j and the image pi, wherein the image j is an ultrasonic target image, and a sampling viewpoint of the image j is used as a target viewpoint;
(c) construction of a CT image dataset G
Sampling h CT image sequences, wherein h is more than or equal to 1 and less than or equal to 20, recording every adjacent 3 frames of each sequence as an image l, an image m and an image n, splicing the image l and the image n according to a color channel to obtain an image sigma, forming a data element by the image m and the image sigma, taking the image m as a CT target image, taking a sampling viewpoint of the image m as a target viewpoint, if the last residual image in the same image sequence is less than 3 frames, abandoning, and constructing a data set G by using all the sequences, wherein the data set G has xi elements, and xi is more than or equal to 1000 and less than or equal to 20000;
step 2: constructing neural networks
The resolution of the image or video processed by the neural network is p x o, p is the width, o is the height, and the resolution is 100-2000, 100-2000;
(1) structure of network A
Taking tensor H as input, the scale is alpha multiplied by o multiplied by p multiplied by 3, taking tensor I as output, the scale is alpha multiplied by o multiplied by p multiplied by 1, and alpha is the number of batches;
the network A consists of an encoder and a decoder, and for the tensor H, the output tensor I is obtained after encoding and decoding processing is carried out in sequence;
the encoder consists of 5 residual error units, the 1 st to 5 th units respectively comprise 2, 3, 4, 6 and 3 residual error modules, each residual error module performs convolution for 3 times, the shapes of convolution kernels are 3 multiplied by 3, the number of the convolution kernels is 64, 64, 128, 256 and 512, and a maximum pooling layer is included behind the first residual error unit;
the decoder is composed of 6 decoding units, each decoding unit comprises two steps of deconvolution and convolution, the shapes and the numbers of convolution kernels of the deconvolution and convolution are the same, the shapes of convolution kernels of the 1 st to 6 th decoding units are all 3x3, the numbers of the convolution kernels are 512, 256, 128, 64, 32 and 16 respectively, cross-layer connection is carried out between network layers of the encoder and the decoder, and the corresponding relation of the cross-layer connection is as follows: 1 and 4, 2 and 3, 3 and 2, 4 and 1;
(2) structure of network B
Tensor J and tensor K are used as input, the scales are respectively alpha multiplied by O multiplied by p multiplied by 3 and alpha multiplied by O multiplied by p multiplied by 6, tensor L, tensor O and tensor
As output, the scales are α × 2 × 6, α × 4 × 1, and α × 1 × 1, respectively, α being the number of batches;
the network B is composed of a module P, a module Q and a module mu, and has 14 layers of convolution units in total, firstly, a tensor J and a tensor K are spliced according to the last channel to obtain a tensor with the scale of alpha multiplied by O multiplied by P multiplied by 9, and after the tensor is processed by the module P, the module Q and the module mu, an output tensor L, a tensor O and a tensor are respectively obtained
For the module P, except for sharing 4 layers, the module P occupies convolution units from the 5 th layer to the 7 th layer of the network B, the scale of convolution kernels is 3 multiplied by 3, the number of the convolution kernels is 256, and after the convolution processing is carried out on the processing result of the 7 th layer by using 12 convolution kernels of 3 multiplied by 3, the result of the tensor L is obtained from 12 channels;
for the module Q, except for 1 to 4 layers of the shared network B, 8 th to 11 th layers of convolution units of the network B are occupied, 2 nd layer output of the network B is used as 8 th layer input of the network B, the shapes of convolution kernels in the 8 th to 11 th layers of convolution units are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after convolution processing is carried out on the 11 th layer result by using 4 convolution kernels of 3 multiplied by 3, tensor O results are obtained from 4 channels;
for the module mu, except 1 to 4 layers of the shared network B, the module mu also occupies convolution units from 12 th layer to 14 th layer of the network B, the 4 th layer output of the network B is used as the 12 th layer input, the shapes of convolution kernels in the 12 th layer to the 14 th layer are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after the convolution processing is carried out on the 14 th layer result by using 1 convolution kernel of 3 multiplied by 3, tensor is obtained
The result of (1);
(3) structure of network C
Taking tensor R and tensor S as network input, wherein the scales are both alpha multiplied by o multiplied by p multiplied by 3, taking tensor T as network output, the scales are alpha multiplied by o multiplied by p multiplied by 2, and alpha is the number of batches;
the network C is designed into a coding and decoding structure, firstly, a tensor R and a tensor S are spliced according to a last channel to obtain a tensor with the scale of alpha multiplied by o multiplied by p multiplied by 6, and an output tensor T is obtained after the tensor is subjected to coding and decoding processing;
for the coding structure, the coding structure is composed of 6 layers of coding units, each layer of coding unit comprises 1 convolution processing, 1 batch normalization processing and 1 activation processing, wherein the 1 st layer of coding unit adopts 7x7 convolution kernels, other layer of coding units all adopt 3x3 convolution kernels, the convolution step length of the 1 st and 3 rd layer of coding units is 1, the convolution step length of other layer of coding units is 2, for each layer of coding unit, the coding units are all activated by Relu function, and the number of the convolution kernels of the 1-6 layer of coding units is respectively 16, 32, 64, 128, 256 and 512;
for a decoding structure, the decoding structure comprises 6 layers of decoding units, each layer of decoding unit comprises a deconvolution unit, a connection processing unit and a convolution unit, wherein the deconvolution unit comprises deconvolution processing and Relu activation processing, the sizes of 1-6 layers of deconvolution kernels are all 3x3, for the 1 st-2 layers of decoding units, the deconvolution step length is 1, the deconvolution step length of the 3-6 layers of decoding units is 2, the number of the 1-6 layers of deconvolution kernels is 512, 256, 128, 64, 32 and 16 in sequence, the connection processing unit connects the deconvolution results of the coding unit and the corresponding decoding units and inputs the results into the convolution units, the convolution kernel size of the 1-5 layers of convolution units is 3x3, the convolution kernel size of the 6 th layer of convolution unit is 7x7, the convolution step lengths of the 1-6 layers of convolution units are all 2, and after the convolution results of the 6 th layer are processed by 2 3x3, obtaining a result T; and step 3: training of neural networks
Respectively dividing samples in a data set D, a data set E and a data set G into a training set and a testing set according to a ratio of 9:1, wherein data in the training set is used for training, data in the testing set is used for testing, training data are respectively obtained from the corresponding data sets when the following steps are trained, the training data are uniformly scaled to a resolution ratio p x o and input into a corresponding network, iterative optimization is carried out, and loss of each batch is minimized by continuously modifying network model parameters;
in the training process, the calculation method of each loss is as follows:
internal parameter supervision synthesis loss: in the network model parameter training of the natural image, the output tensor I of the network A is taken as the depth, and the output result L of the network B and the internal parameter label e of the training data are taken as the deptht(t is 1, 2, 3, 4) respectively used as a pose parameter and a camera internal parameter, respectively synthesizing two images at the viewpoint of the image c by using the image b and the image d according to a computer vision algorithm, and respectively calculating by using the image c and the two images according to the sum of the intensity difference of pixel-by-pixel and color-by-color channels;
unsupervised synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and a camera internal parameter, images at the viewpoint of a target image are respectively synthesized by using two adjacent images of the target image according to a computer vision algorithm, and the target image and the images at the viewpoint of the target image are respectively used for calculation according to the sum of the intensity differences of pixel-by-pixel and color-by-color channels;
internal parameter error loss: utilizing output result O of network B and internal parameter label e of training datat(t is 1, 2, 3, 4) is calculated according to the sum of absolute values of the difference of each component；
Spatial structure error loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the tensor O of the network B are respectively used as pose parameters and camera internal parameters, the target image is reconstructed by taking the viewpoint of the target image as the origin of a camera coordinate system according to a computer vision algorithm, a RANSAC algorithm is adopted to fit the spatial structure of reconstruction points, and the Euclidean distance between each reconstruction point of the target image and the spatial geometric structure is calculated;
transform synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
Taking the output tensor L and the tensor O of the network B as a pose parameter and an internal parameter of a camera respectively as a depth, and obtaining a new position of each pixel by adding the coordinate of each pixel to a displacement result of each pixel output by the network C to form a synthesis result image after obtaining the position of each pixel for each image in the two synthesized images in the process of synthesizing the two images at the viewpoint of the target image by using two adjacent images of the target image according to a computer vision algorithm, wherein the output tensor L and the tensor O of the network B are used as the pose parameter and the internal parameter of the camera respectively;
(1) on the data set D, the modules P of the network A and the network B are respectively trained 80000 times
Taking out training data from the data set D each time, uniformly scaling to a resolution ratio P x o, inputting the image c into the network A, inputting the image c and the image r into the network B, training the module P of the network B, and calculating the training loss of each batch by the supervision and synthesis loss of internal parameters;
(2) on data set D, model Q of network B was trained 80000 times
Taking out training data from the data set D each time, uniformly scaling to a resolution ratio p x o, inputting the image c into the network A, inputting the image c and the image t into the network B, and training the module Q of the network B, wherein the training loss of each batch is obtained by calculating the sum of the supervised synthesis loss of internal parameters and the error loss of the internal parameters;
(3) on the data set E, the module Q and the module mu of the network B are trained 80000 times for feature migration
Taking out the ultrasonic training data from the data set E each time, uniformly scaling the ultrasonic training data to the resolution p × o, inputting the image j and the image pi into the network B, and training the module Q and the module μ of the network B, wherein the training loss of each batch is calculated as follows:
z＝v+W+χ (1)
wherein v is unsupervised synthesis loss, W is space structure error loss, and constant depth loss χ is calculated by means of the mean square error of the output result of the module μ;
(4) on data set E, three modules of network B were trained 80000 times according to the following steps
Taking out ultrasonic training data from a data set E every time, uniformly scaling the ultrasonic training data to a resolution ratio p x o, inputting an image j and an image pi into a network B, and during training, continuously modifying parameters of three modules of the network B, and performing iterative optimization to minimize the loss of each image of each batch, wherein the training loss of each batch is composed of the sum of unsupervised synthesis loss, spatial structure error loss and constant depth loss, and the constant depth loss is calculated by using the mean square error of an output result of a module mu of the network B;
(5) on data set E, three modules of network C and network B were trained 80000 times
Every time ultrasonic image training data are taken out from the data set E, the data are uniformly scaled to the resolution ratio p x o, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is output
Using the output tensor L and the output tensor O of the network B as the pose parameter and the internal parameter of the camera respectively as the depth, synthesizing two images at the sight point of the image j according to the image i and the image k respectively, and combining the two imagesInputting the two images into a network C, and continuously modifying parameters of the network C and the network B, and performing iterative optimization to minimize the loss of each image in each batch, wherein the loss of each batch is calculated as the sum of transformation synthesis loss, spatial structure error loss and constant depth loss, and the constant depth loss is calculated by using the mean square error of the output result of a module mu of the network B;
(6) on a data set E, three modules of a network C and a network B are trained 50000 times to obtain a model rho
During training, ultrasonic image training data are taken out from the data set E each time, the data are uniformly scaled to the resolution p multiplied by o, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is output
As the depth, the tensor L and the tensor O output by the network B are respectively used as a pose parameter and a camera internal parameter, two images at the visual point of the image j are synthesized according to the image i and the image k respectively, the two images are input into the network C, the parameters of the network C and the network B are continuously modified, iterative optimization is carried out, the loss of each image in each batch is enabled to be minimum, an optimal network model parameter rho is obtained after iteration, and the loss of each batch is calculated as the sum of transformation synthesis loss and space structure error loss;
(7) on data set G, three modules of network C and network B were trained 80000 times
During training, CT image training data are taken out from a data set G each time, the CT image training data are uniformly scaled to resolution p multiplied by o, an image m and an image sigma are input into a network B, and the output tensor of a module mu of the network B is output
And as the depth, respectively taking the tensor L and the tensor O output by the network B as a pose parameter and an internal parameter of the camera, respectively synthesizing two images at the viewpoint of the image m according to the image L and the image n, inputting the two images into the network C, and continuously modifying the parameters of the network C and the network B to perform iterative optimization so as to minimize the loss of each image in each batch, wherein the loss in each batch is calculated as transformationThe sum of synthesis loss, spatial structure error loss, constant depth loss and camera translational motion loss Y is obtained, wherein the constant depth loss is obtained by utilizing the mean square error calculation of the output result of a module mu of a network B, and the Y is obtained by the output pose parameter of the network B through the constraint calculation of the camera translational motion;
(8) on a data set G, three modules of a network C and a network B are trained 50000 times to obtain a model rho'
Taking out CT image training data from the data set G each time, uniformly scaling to resolution p x o, inputting the image m and the image sigma into the network B, and outputting tensor of module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of the camera, two images at the viewpoint of an image m are synthesized according to an image L and an image n respectively, the two images are input into a network C, parameters of the network C and the network B are continuously modified, iterative optimization is carried out, the loss of each image in each batch is minimized, an optimal network model parameter rho' is obtained after iteration, the loss of each batch is calculated to be the sum of transformation synthesis loss, space structure error loss and translational motion loss Y of the camera, and the Y is calculated from the output pose parameter of the network B according to the constraint of the translational motion of the camera;
and 4, step 4: three-dimensional reconstruction of ultrasound or CT images
Utilizing a self-sampled ultrasonic or CT sequence image to uniformly scale each frame of image to resolution ratio p x o, using model parameter p or model parameter p' to make prediction, inputting image j and image pi into network B for ultrasonic sequence image, inputting image m and image sigma into network B for CT sequence image, and inputting output tensor of module mu of network B
Using the output tensor L and the output tensor O of the network B as a pose parameter and an internal parameter of the camera respectively as depth, selecting a key frame according to the following steps, using the first frame in the sequence as a current key frame, and sequentially using the current key frame as the current key frameTaking each frame in the sequence image as a target frame, synthesizing an image at a viewpoint of the target frame by utilizing a camera pose parameter and an internal parameter according to a current key frame, calculating an error lambda by utilizing the magnitude of the sum of pixel-by-pixel color channel intensity differences between the synthesized image and the target frame, synthesizing an image at the viewpoint of the target frame by utilizing the camera pose parameter and the internal parameter according to adjacent frames of the target frame, calculating an error gamma by utilizing the magnitude of the sum of pixel-by-pixel color channel intensity differences between the synthesized image and the target frame, further calculating a synthesis error ratio Z by utilizing a formula (2), and when Z is greater than a threshold eta, 1<η<2, updating the current key frame to the current target frame;
and (3) for any target frame, the resolution ratio of the target frame is scaled to MxN, the three-dimensional coordinates in the camera coordinate system of each pixel of each frame of image are calculated according to the internal parameters of the camera and the reconstruction algorithm of computer vision, further, the viewpoint of the first frame is used as the origin of the world coordinate system, and the three-dimensional coordinates in the world coordinate system of each pixel of each frame of image of the sequence are calculated by utilizing the geometric transformation of three-dimensional space and combining the pose parameters of all key frames.
The method can effectively realize the rapid three-dimensional reconstruction of the ultrasonic or CT image, can improve the auxiliary diagnosis efficiency in the artificial intelligent auxiliary diagnosis, and can show the CT or ultrasonic slice image with a 3D visual effect so as to improve the auxiliary diagnosis accuracy.
Drawings
FIG. 1 is a three-dimensional reconstruction result of an ultrasound image of the present invention;
fig. 2 is a three-dimensional reconstruction result diagram of a CT image according to the present invention.
Detailed Description
Examples
The invention is further described below with reference to the accompanying drawings.
The embodiment is implemented under a Windows 1064-bit operating system on a PC, and the hardware configuration of the embodiment is CPU i7-9700F, a memory 16G and a GPU NVIDIA GeForce GTX 20708G. The deep learning library adopts Tensorflow1.14, and the programming adopts Python language.
A three-dimensional reconstruction method of ultrasonic or CT medical images based on feature migration is disclosed, the method inputs an ultrasonic or CT image sequence, the resolution ratio is M multiplied by N, for ultrasonic images, M is 450, N is 300, for CT images, M and N are both 512, the three-dimensional reconstruction process specifically comprises the following steps:
step 1: building a data set
(a) Constructing a natural image dataset D
Selecting a natural image website, requiring image sequences and corresponding internal parameters of a camera, downloading 19 image sequences and the corresponding internal parameters of the sequences from the website, recording every adjacent 3 frames of images as an image b, an image c and an image d for each image sequence, splicing the image b and the image d according to a color channel to obtain an image tau, forming a data element by the image c and the image tau, wherein the image c is a natural target image, the sampling viewpoint of the image c is used as a target viewpoint, and the internal parameters of the image b, the image c and the image d are all et(t ═ 1, 2, 3, 4) in which e1Is a horizontal focal length, e2Is a vertical focal length, e3And e4Are two components of the principal point coordinates; if the last residual image in the same image sequence is less than 3 frames, discarding; constructing a data set D by using all the sequences, wherein the data set D has 3600 elements;
(b) constructing an ultrasound image dataset E
Sampling 10 ultrasonic image sequences, recording 3 adjacent images of each sequence as an image i, an image j and an image k, splicing the image i and the image k according to a color channel to obtain an image pi, forming a data element by the image j and the image pi, wherein the image j is an ultrasonic target image, a sampling viewpoint of the image j is used as a target viewpoint, if the last residual image in the same image sequence is less than 3 frames, discarding the image j, and constructing a data set E by using all the sequences, wherein the data set E has 1600 elements;
(c) construction of a CT image dataset G
Sampling 1 CT image sequence, regarding the sequence, marking every adjacent 3 frames as an image l, an image m and an image n, splicing the image l and the image n according to a color channel to obtain an image sigma, forming a data element by the image m and the image sigma, wherein the image m is a CT target image, a sampling viewpoint of the image m is used as a target viewpoint, if the last residual image in the same image sequence is less than 3 frames, discarding, and constructing a data set G by using all the sequences, wherein the data set G has 2000 elements; step 2: constructing neural networks
The resolution of the image or video processed by the neural network is p x o, p is the width, o is the height, and the resolution is 100-2000, 100-2000;
(1) structure of network A
Taking tensor H as input, the scale is 16 multiplied by 128 multiplied by 416 multiplied by 3, taking tensor I as output, and the scale is 16 multiplied by 128 multiplied by 416 multiplied by 1;
the network A consists of an encoder and a decoder, and for the tensor H, the output tensor I is obtained after encoding and decoding processing is carried out in sequence;
the encoder consists of 5 residual error units, the 1 st to 5 th units respectively comprise 2, 3, 4, 6 and 3 residual error modules, each residual error module performs convolution for 3 times, the shapes of convolution kernels are 3 multiplied by 3, the number of the convolution kernels is 64, 64, 128, 256 and 512, and a maximum pooling layer is included behind the first residual error unit;
the decoder is composed of 6 decoding units, each decoding unit comprises two steps of deconvolution and convolution, the shapes and the numbers of convolution kernels of the deconvolution and convolution are the same, the shapes of convolution kernels of the 1 st to 6 th decoding units are all 3x3, the numbers of the convolution kernels are 512, 256, 128, 64, 32 and 16 respectively, cross-layer connection is carried out between network layers of the encoder and the decoder, and the corresponding relation of the cross-layer connection is as follows: 1 and 4, 2 and 3, 3 and 2, 4 and 1;
(2) structure of network B
Tensor J and tensor K are used as input, the scales are respectively 16 × 128 × 416 × 3 and 16 × 128 × 416 × 6, tensor L, tensor O and tensor
As an output, the scales are 16 × 2 × 6, 16 × 4 × 1, and 16 × 1 × 1, respectively;
the network B is composed of a module P, a module Q and a module mu, a convolution unit with 14 layers is shared, firstly, a tensor J and a tensor K are spliced according to the last channel to obtain a tensor with the dimension of 16 multiplied by 128 multiplied by 416 multiplied by 9, and after the tensor is processed by the module P, the module Q and the module mu, an output tensor L, a tensor O and a tensor are respectively obtained
For the module P, except for sharing 4 layers, the module P occupies convolution units from the 5 th layer to the 7 th layer of the network B, the scale of convolution kernels is 3 multiplied by 3, the number of the convolution kernels is 256, and after the convolution processing is carried out on the processing result of the 7 th layer by using 12 convolution kernels of 3 multiplied by 3, the result of the tensor L is obtained from 12 channels;
for the module Q, except for 1 to 4 layers of the shared network B, 8 th to 11 th layers of convolution units of the network B are occupied, 2 nd layer output of the network B is used as 8 th layer input of the network B, the shapes of convolution kernels in the 8 th to 11 th layers of convolution units are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after convolution processing is carried out on the 11 th layer result by using 4 convolution kernels of 3 multiplied by 3, tensor O results are obtained from 4 channels;
for the module mu, except 1 to 4 layers of the shared network B, the module mu also occupies convolution units from 12 th layer to 14 th layer of the network B, the 4 th layer output of the network B is used as the 12 th layer input, the shapes of convolution kernels in the 12 th layer to the 14 th layer are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after the convolution processing is carried out on the 14 th layer result by using 1 convolution kernel of 3 multiplied by 3, tensor is obtained
The result of (1);
(3) structure of network C
Taking tensor R and tensor S as network input, wherein the scales of the tensor R and the tensor S are both 16 multiplied by 128 multiplied by 416 multiplied by 3, taking tensor T as network output, and the scales of the tensor R and the tensor S are 16 multiplied by 128 multiplied by 416 multiplied by 2;
the network C is designed into a coding and decoding structure, firstly, a tensor R and a tensor S are spliced according to a last channel to obtain a tensor with the dimension of 16 multiplied by 128 multiplied by 416 multiplied by 6, and an output tensor T is obtained after the tensor is subjected to coding and decoding processing;
for the coding structure, the coding structure is composed of 6 layers of coding units, each layer of coding unit comprises 1 convolution processing, 1 batch normalization processing and 1 activation processing, wherein the 1 st layer of coding unit adopts 7x7 convolution kernels, other layer of coding units all adopt 3x3 convolution kernels, the convolution step length of the 1 st and 3 rd layer of coding units is 1, the convolution step length of other layer of coding units is 2, for each layer of coding unit, the coding units are all activated by Relu function, and the number of the convolution kernels of the 1-6 layer of coding units is respectively 16, 32, 64, 128, 256 and 512;
for a decoding structure, the decoding structure comprises 6 layers of decoding units, each layer of decoding unit comprises a deconvolution unit, a connection processing unit and a convolution unit, wherein the deconvolution unit comprises deconvolution processing and Relu activation processing, the sizes of 1-6 layers of deconvolution kernels are all 3x3, for the 1 st-2 layers of decoding units, the deconvolution step length is 1, the deconvolution step length of the 3-6 layers of decoding units is 2, the number of the 1-6 layers of deconvolution kernels is 512, 256, 128, 64, 32 and 16 in sequence, the connection processing unit connects the deconvolution results of the coding unit and the corresponding decoding units and inputs the results into the convolution units, the convolution kernel size of the 1-5 layers of convolution units is 3x3, the convolution kernel size of the 6 th layer of convolution unit is 7x7, the convolution step lengths of the 1-6 layers of convolution units are all 2, and after the convolution results of the 6 th layer are processed by 2 3x3, obtaining a result T;
and step 3: training of neural networks
Respectively dividing samples in a data set D, a data set E and a data set G into a training set and a testing set according to a ratio of 9:1, wherein data in the training set is used for training, data in the testing set is used for testing, training data are respectively obtained from corresponding data sets when the following steps are trained, the training data are uniformly scaled to a resolution of 416 x 128 and input into corresponding networks, iterative optimization is carried out, and loss of each batch is minimized by continuously modifying network model parameters;
in the training process, the calculation method of each loss is as follows:
internal parameter supervision synthesis loss: in the network model parameter training of natural images, the output tensor I of a network A is used as depth, the output result L of a network B and an internal parameter label et (t is 1, 2, 3 and 4) of training data are respectively used as pose parameters and camera internal parameters, two images at the viewpoint of an image c are respectively synthesized by using an image B and an image d according to a computer vision algorithm, and the image c and the two images are respectively obtained by calculation according to the sum of the intensity differences of pixel-by-pixel and color-by-color channels;
unsupervised synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and a camera internal parameter, images at the viewpoint of a target image are respectively synthesized by using two adjacent images of the target image according to a computer vision algorithm, and the target image and the images at the viewpoint of the target image are respectively used for calculation according to the sum of the intensity differences of pixel-by-pixel and color-by-color channels;
internal parameter error loss: calculating the output result O of the network B and an internal parameter label et (t is 1, 2, 3 and 4) of the training data according to the sum of absolute values of all component differences;
spatial structure error loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the tensor O of the network B are respectively used as pose parameters and camera internal parameters, the target image is reconstructed by taking the viewpoint of the target image as the origin of a camera coordinate system according to a computer vision algorithm, a RANSAC algorithm is adopted to fit the spatial structure of reconstruction points, and the Euclidean distance between each reconstruction point of the target image and the spatial geometric structure is calculated;
transform synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
Taking the output tensor L and the tensor O of the network B as a pose parameter and an internal parameter of a camera respectively as a depth, and obtaining a new position of each pixel by adding the coordinate of each pixel to a displacement result of each pixel output by the network C to form a synthesis result image after obtaining the position of each pixel for each image in the two synthesized images in the process of synthesizing the two images at the viewpoint of the target image by using two adjacent images of the target image according to a computer vision algorithm, wherein the output tensor L and the tensor O of the network B are used as the pose parameter and the internal parameter of the camera respectively;
(1) on the data set D, the modules P of the network A and the network B are respectively trained 80000 times
Taking out training data from the data set D each time, uniformly scaling the training data to a resolution of 416 multiplied by 128, inputting the image c into the network A, inputting the image c and the image tau into the network B, and training the module P of the network B, wherein the training loss of each batch is obtained by calculating the internal parameter supervision synthesis loss;
(2) on data set D, model Q of network B was trained 80000 times
Taking out training data from the data set D each time, uniformly scaling the training data to a resolution of 416 multiplied by 128, inputting the image c into the network A, inputting the image c and the image tau into the network B, and training the module Q of the network B, wherein the training loss of each batch is calculated by the sum of the supervised synthesis loss of internal parameters and the error loss of the internal parameters;
(3) on the data set E, the module Q and the module mu of the network B are trained 80000 times for feature migration
Each time, taking out the ultrasonic training data from the data set E, uniformly scaling the data to the resolution of 416 multiplied by 128, inputting the image j and the image pi into the network B, and training the module Q and the module mu of the network B, wherein the training loss of each batch is calculated as follows:
z＝v+W+χ (1)
wherein v is unsupervised synthesis loss, W is space structure error loss, and constant depth loss χ is calculated by means of the mean square error of the output result of the module μ;
(4) on data set E, three modules of network B were trained 80000 times according to the following steps
Taking out ultrasonic training data from a data set E every time, uniformly scaling the ultrasonic training data to a resolution of 416 x 128, inputting an image j and an image pi into a network B, and during training, continuously modifying parameters of three modules of the network B, and performing iterative optimization to minimize the loss of each image in each batch, wherein the training loss in each batch is composed of the sum of unsupervised synthesis loss, spatial structure error loss and constant depth loss, and the constant depth loss is calculated by using the mean square error of an output result of a module mu of the network B;
(5) on data set E, three modules of network C and network B were trained 80000 times
Every time ultrasonic image training data are taken out from the data set E and are uniformly scaled to the resolution ratio of 416 multiplied by 128, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is output
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of a camera, two images at the visual point of an image j are synthesized respectively according to an image i and an image k, the two images are input into a network C, the parameters of the network C and the network B are continuously modified, and iterative optimization is performed, so that the loss of each image in each batch is minimized, the loss of each batch is calculated as the sum of transformation synthesis loss, spatial structure error loss and constant depth loss, wherein the constant depth loss is calculated by using the mean square error of the output result of a module mu of the network B;
(6) on a data set E, three modules of a network C and a network B are trained 50000 times to obtain a model rho
During training, ultrasonic image training data are taken out from the data set E each time, the data are uniformly zoomed to the resolution of 416 multiplied by 128, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is input
As depth, network B is output with tensors L and ORespectively serving as pose parameters and camera internal parameters, respectively synthesizing two images at the visual point of an image j according to an image i and an image k, inputting the two images into a network C, continuously modifying parameters of the network C and a network B, and performing iterative optimization to minimize the loss of each image of each batch, obtaining an optimal network model parameter rho after iteration, wherein the loss of each batch is calculated as the sum of transformation synthesis loss and spatial structure error loss;
(7) on data set G, three modules of network C and network B were trained 80000 times
During training, CT image training data are taken out from a data set G each time, the CT image training data are uniformly scaled to the resolution of 416 multiplied by 128, an image m and an image sigma are input into a network B, and the output tensor of a module mu of the network B is output
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of a camera, two images at the viewpoint of an image m are synthesized according to an image L and an image n respectively, the two images are input into a network C, and the loss of each batch of images is minimized by continuously modifying the parameters of the network C and the network B and carrying out iterative optimization, wherein the loss of each batch is calculated as the sum of transformation synthesis loss, spatial structure error loss, constant depth loss and camera translational motion loss Y, the constant depth loss is calculated by using the mean square error of the output result of a module mu of the network B, and the Y is calculated by the output pose parameter of the network B according to the constraint of the camera translational motion;
(8) on a data set G, three modules of a network C and a network B are trained 50000 times to obtain a model rho'
Each time CT image training data is taken out from the data set G, the resolution is uniformly scaled to 416 multiplied by 128, the image m and the image sigma are input into the network B, and the output tensor of the module mu of the network B is output
As the depth, the tensor L and the tensor O output by the network B are respectively used as the pose parameter and the camera internal parameter, and are respectively based on the imagel and n are synthesized into two images at the viewpoint of the image m, the two images are input into a network C, the loss of each image in each batch is minimized by continuously modifying the parameters of the network C and the network B and iterative optimization is carried out, the optimal network model parameter rho' is obtained after iteration, the loss in each batch is calculated into the sum of transformation synthesis loss, space structure error loss and the translational motion loss Y of the camera, and the Y is obtained by the output pose parameter of the network B and the constraint calculation of the translational motion of the camera;
and 4, step 4: three-dimensional reconstruction of ultrasound or CT images
Utilizing a self-sampled ultrasonic or CT sequence image to uniformly scale each frame of image to resolution ratio 416 x 128, using model parameter rho or model parameter rho' to make prediction, inputting image j and image pi into network B for ultrasonic sequence image, inputting image m and image sigma into network B for CT sequence image, and inputting output tensor of module mu of network B
The output tensors L and O of the network B are respectively used as pose parameters and camera internal parameters, selecting key frames according to the following steps, wherein the first frame in the sequence is used as a current key frame, each frame in the sequence image is used as a target frame in turn, synthesizing an image at a viewpoint of a target frame according to a current key frame by using a pose parameter and an internal parameter of a camera, calculating an error lambda by using the sum of intensity differences of pixel-by-pixel color-by-color channels between the synthesized image and the target frame, synthesizing an image at the viewpoint of the target frame by using the pose parameter and the internal parameter of the camera according to an adjacent frame of the target frame, calculating an error gamma by using the sum of intensity differences of pixel-by-pixel color-by-color channels between the synthesized image and the target frame, further calculating a synthesis error ratio Z by using a formula (2), and updating the current key frame into the current target frame when Z is greater than a threshold value 1.2;
the method comprises the steps of scaling the resolution of any target frame to MxN, taking M450 and N300 for ultrasonic images and taking M and N512 for CT images, calculating the three-dimensional coordinates in the camera coordinate system of each pixel of each frame of image according to the internal parameters of a camera and the reconstruction algorithm of computer vision, further taking the viewpoint of a first frame as the origin of a world coordinate system, combining pose parameters of all key frames, and calculating the three-dimensional coordinates in the world coordinate system of each pixel of each frame of image of the sequence by using three-dimensional space geometric transformation.
In the examples, the experimental hyper-parameters are as follows: the optimizer adopts an Adam optimizer, the network learning rate is 0.0002, and the momentum coefficient is 0.9;
in this embodiment, network training is performed on a data set D, a data set E, and a training set of a data set G, and respective tests are performed on the data set E and a test set of the data set G, where table 1 is an error result of ultrasonic image synthesis, which is obtained by calculation using formula (1), and a 3D reconstruction result is generated by segmenting an ultrasonic image using DenseNet, and fig. 1 shows a three-dimensional reconstruction result of the ultrasonic image; table 2 shows an error result of CT image synthesis calculated by formula (1), and a 3D reconstruction result is generated by segmenting the CT image by using DenseNet, and fig. 2 shows a three-dimensional reconstruction result of the CT image; from these results, the effectiveness of the present invention can be seen.
TABLE 1
| Serial number | Error of the measurement |
| 1 | 0.24817425412586994 |
| 2 | 0.2128944972906263 |
| 3 | 0.2523211232711172 |
| 4 | 0.16108245516912195 |
| 5 | 0.1296228699255319 |
| 6 | 0.16174850045371253 |
| 7 | 0.109522666112984 |
| 8 | 0.16590744004359323 |
| 9 | 0.18864673126611387 |
| 10 | 0.10801564672849703 |
TABLE 2
| 1 | 0.15917478001790017 |
| 2 | 0.19462372936315264 |
| 3 | 0.199991624672326 |
| 4 | 0.1968527306861435 |
| 5 | 0.2187002130410199 |
| 6 | 0.20469571469345407 |
| 7 | 0.24732626433002053 |
| 8 | 0.3044730817847928 |
| 9 | 0.24487321087759462 |
| 10 | 0.24453120950986149 |
Claims (1)
1. A three-dimensional reconstruction method of ultrasonic or CT medical images based on feature migration is characterized in that an ultrasonic or CT image sequence is input, the image resolution is MxN, M is more than or equal to 100 and less than or equal to 1500, N is more than or equal to 100 and less than or equal to 1500, and the three-dimensional reconstruction process specifically comprises the following steps:
step 1: building a data set
(a) Constructing a natural image dataset D
Selecting a natural image website, requiring image sequences and corresponding camera internal parameters, downloading a image sequences andand (2) internal parameters corresponding to the sequences, a is more than or equal to 1 and less than or equal to 20, for each image sequence, every 3 adjacent frames of images are marked as an image b, an image c and an image d, the image b and the image d are spliced according to a color channel to obtain an image tau, the image c is a natural target image, the sampling viewpoint of the image c is used as a target viewpoint, and the internal parameters of the image b, the image c and the image d are all et(t ═ 1, 2, 3, 4) in which e1Is a horizontal focal length, e2Is a vertical focal length, e3And e4Are two components of the principal point coordinates; if the last residual image in the same image sequence is less than 3 frames, discarding; constructing a data set D by using all the sequences, wherein the data set D has f elements, and f is more than or equal to 3000 and less than or equal to 20000;
(b) constructing an ultrasound image dataset E
Sampling g ultrasonic image sequences, wherein g is more than or equal to 1 and less than or equal to 20, recording every adjacent 3 frames of images of each sequence as an image i, an image j and an image k, splicing the image i and the image k according to a color channel to obtain an image pi, forming a data element by the image j and the image pi, wherein the image j is an ultrasonic target image, and a sampling viewpoint of the image j is used as a target viewpoint;
(c) construction of a CT image dataset G
Sampling h CT image sequences, wherein h is more than or equal to 1 and less than or equal to 20, recording every adjacent 3 frames of each sequence as an image l, an image m and an image n, splicing the image l and the image n according to a color channel to obtain an image sigma, forming a data element by the image m and the image sigma, taking the image m as a CT target image, taking a sampling viewpoint of the image m as a target viewpoint, if the last residual image in the same image sequence is less than 3 frames, abandoning, and constructing a data set G by using all the sequences, wherein the data set G has xi elements, and xi is more than or equal to 1000 and less than or equal to 20000; step 2: constructing neural networks
The resolution of the image or video processed by the neural network is p x o, p is the width, o is the height, and the resolution is 100-2000, 100-2000;
(1) structure of network A
Taking tensor H as input, the scale is alpha multiplied by o multiplied by p multiplied by 3, taking tensor I as output, the scale is alpha multiplied by o multiplied by p multiplied by 1, and alpha is the number of batches;
the network A consists of an encoder and a decoder, and for the tensor H, the output tensor I is obtained after encoding and decoding processing is carried out in sequence;
the encoder consists of 5 residual error units, the 1 st to 5 th units respectively comprise 2, 3, 4, 6 and 3 residual error modules, each residual error module performs convolution for 3 times, the shapes of convolution kernels are 3 multiplied by 3, the number of the convolution kernels is 64, 64, 128, 256 and 512, and a maximum pooling layer is included behind the first residual error unit;
the decoder is composed of 6 decoding units, each decoding unit comprises two steps of deconvolution and convolution, the shapes and the numbers of convolution kernels of the deconvolution and convolution are the same, the shapes of convolution kernels of the 1 st to 6 th decoding units are all 3x3, the numbers of the convolution kernels are 512, 256, 128, 64, 32 and 16 respectively, cross-layer connection is carried out between network layers of the encoder and the decoder, and the corresponding relation of the cross-layer connection is as follows: 1 and 4, 2 and 3, 3 and 2, 4 and 1;
(2) structure of network B
Tensor J and tensor K are used as input, the scales are respectively alpha multiplied by O multiplied by p multiplied by 3 and alpha multiplied by O multiplied by p multiplied by 6, tensor L, tensor O and tensor
As output, the scales are α × 2 × 6, α × 4 × 1, and α × 1 × 1, respectively, α being the number of batches;
the network B is composed of a module P, a module Q and a module mu, and has 14 layers of convolution units in total, firstly, a tensor J and a tensor K are spliced according to the last channel to obtain a tensor with the scale of alpha multiplied by O multiplied by P multiplied by 9, and after the tensor is processed by the module P, the module Q and the module mu, an output tensor L, a tensor O and a tensor are respectively obtained
For the module P, except for sharing 4 layers, the module P occupies convolution units from the 5 th layer to the 7 th layer of the network B, the scale of convolution kernels is 3 multiplied by 3, the number of the convolution kernels is 256, and after the convolution processing is carried out on the processing result of the 7 th layer by using 12 convolution kernels of 3 multiplied by 3, the result of the tensor L is obtained from 12 channels;
for the module Q, except for 1 to 4 layers of the shared network B, 8 th to 11 th layers of convolution units of the network B are occupied, 2 nd layer output of the network B is used as 8 th layer input of the network B, the shapes of convolution kernels in the 8 th to 11 th layers of convolution units are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after convolution processing is carried out on the 11 th layer result by using 4 convolution kernels of 3 multiplied by 3, tensor O results are obtained from 4 channels;
for the module mu, except 1 to 4 layers of the shared network B, the module mu also occupies convolution units from 12 th layer to 14 th layer of the network B, the 4 th layer output of the network B is used as the 12 th layer input, the shapes of convolution kernels in the 12 th layer to the 14 th layer are all 3 multiplied by 3, the number of the convolution kernels is all 256, and after the convolution processing is carried out on the 14 th layer result by using 1 convolution kernel of 3 multiplied by 3, tensor is obtained
The result of (1);
(3) structure of network C
Taking tensor R and tensor S as network input, wherein the scales are both alpha multiplied by o multiplied by p multiplied by 3, taking tensor T as network output, the scales are alpha multiplied by o multiplied by p multiplied by 2, and alpha is the number of batches;
the network C is designed into a coding and decoding structure, firstly, a tensor R and a tensor S are spliced according to a last channel to obtain a tensor with the scale of alpha multiplied by o multiplied by p multiplied by 6, and an output tensor T is obtained after the tensor is subjected to coding and decoding processing;
for the coding structure, the coding structure is composed of 6 layers of coding units, each layer of coding unit comprises 1 convolution processing, 1 batch normalization processing and 1 activation processing, wherein the 1 st layer of coding unit adopts 7x7 convolution kernels, other layer of coding units all adopt 3x3 convolution kernels, the convolution step length of the 1 st and 3 rd layer of coding units is 1, the convolution step length of other layer of coding units is 2, for each layer of coding unit, the coding units are all activated by Relu function, and the number of the convolution kernels of the 1-6 layer of coding units is respectively 16, 32, 64, 128, 256 and 512;
for a decoding structure, the decoding structure comprises 6 layers of decoding units, each layer of decoding unit comprises a deconvolution unit, a connection processing unit and a convolution unit, wherein the deconvolution unit comprises deconvolution processing and Relu activation processing, the sizes of 1-6 layers of deconvolution kernels are all 3x3, for the 1 st-2 layers of decoding units, the deconvolution step length is 1, the deconvolution step length of the 3-6 layers of decoding units is 2, the number of the 1-6 layers of deconvolution kernels is 512, 256, 128, 64, 32 and 16 in sequence, the connection processing unit connects the deconvolution results of the coding unit and the corresponding decoding units and inputs the results into the convolution units, the convolution kernel size of the 1-5 layers of convolution units is 3x3, the convolution kernel size of the 6 th layer of convolution unit is 7x7, the convolution step lengths of the 1-6 layers of convolution units are all 2, and after the convolution results of the 6 th layer are processed by 2 3x3, obtaining a result T;
and step 3: training of neural networks
Respectively dividing samples in a data set D, a data set E and a data set G into a training set and a testing set according to a ratio of 9:1, wherein data in the training set is used for training, data in the testing set is used for testing, training data are respectively obtained from the corresponding data sets when the following steps are trained, the training data are uniformly scaled to a resolution ratio p x o and input into a corresponding network, iterative optimization is carried out, and loss of each batch is minimized by continuously modifying network model parameters;
in the training process, the calculation method of each loss is as follows:
internal parameter supervision synthesis loss: in the network model parameter training of natural images, the output tensor I of a network A is used as depth, the output result L of a network B and an internal parameter label et (t is 1, 2, 3 and 4) of training data are respectively used as pose parameters and camera internal parameters, two images at the viewpoint of an image c are respectively synthesized by using an image B and an image d according to a computer vision algorithm, and the image c and the two images are respectively obtained by calculation according to the sum of the intensity differences of pixel-by-pixel and color-by-color channels;
unsupervised synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and a camera internal parameter, images at the viewpoint of a target image are respectively synthesized by using two adjacent images of the target image according to a computer vision algorithm, and the target image and the images at the viewpoint of the target image are respectively used for calculation according to the sum of the intensity differences of pixel-by-pixel and color-by-color channels;
internal parameter error loss: utilizing output result O of network B and internal parameter label e of training datat(t is 1, 2, 3, 4) calculated as the sum of the absolute values of the differences of the components;
spatial structure error loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
As the depth, the output tensor L and the tensor O of the network B are respectively used as pose parameters and camera internal parameters, the target image is reconstructed by taking the viewpoint of the target image as the origin of a camera coordinate system according to a computer vision algorithm, a RANSAC algorithm is adopted to fit the spatial structure of reconstruction points, and the Euclidean distance between each reconstruction point of the target image and the spatial geometric structure is calculated;
transform synthesis loss: in the network model parameter training of ultrasonic or CT image, the output tensor of the module mu of the network B
Using the output tensor L and tensor O of the network B as the position and internal parameters of the camera, respectively, synthesizing two images at the viewpoint of the target image by using two adjacent images of the target image according to the computer vision algorithm, adding the coordinate of each pixel to the displacement result of each pixel output by the network C after obtaining the position of each pixel for each image in the two synthesized images to obtain the new position of each pixel to form a synthesized result image, and using the pixel-by-pixel and color-by-color channel between the synthesized result image and the image jCalculating the sum of the intensity differences;
(1) on the data set D, the modules P of the network A and the network B are respectively trained 80000 times
Taking out training data from the data set D each time, uniformly scaling to a resolution ratio P x o, inputting the image c into the network A, inputting the image c and the image r into the network B, training the module P of the network B, and calculating the training loss of each batch by the supervision and synthesis loss of internal parameters;
(2) on data set D, model Q of network B was trained 80000 times
Taking out training data from the data set D each time, uniformly scaling to a resolution ratio p x o, inputting the image c into the network A, inputting the image c and the image t into the network B, and training the module Q of the network B, wherein the training loss of each batch is obtained by calculating the sum of the supervised synthesis loss of internal parameters and the error loss of the internal parameters;
(3) on the data set E, the module Q and the module mu of the network B are trained 80000 times for feature migration
Taking out the ultrasonic training data from the data set E each time, uniformly scaling the ultrasonic training data to the resolution p × o, inputting the image j and the image pi into the network B, and training the module Q and the module μ of the network B, wherein the training loss of each batch is calculated as follows:
z＝v+W+χ (1)
wherein v is unsupervised synthesis loss, W is space structure error loss, and constant depth loss χ is calculated by means of the mean square error of the output result of the module μ;
(4) on data set E, three modules of network B were trained 80000 times according to the following steps
Taking out ultrasonic training data from a data set E every time, uniformly scaling the ultrasonic training data to a resolution ratio p x o, inputting an image j and an image pi into a network B, and during training, continuously modifying parameters of three modules of the network B, and performing iterative optimization to minimize the loss of each image of each batch, wherein the training loss of each batch is composed of the sum of unsupervised synthesis loss, spatial structure error loss and constant depth loss, and the constant depth loss is calculated by using the mean square error of an output result of a module mu of the network B;
(5) on data set E, three modules of network C and network B were trained 80000 times
Every time ultrasonic image training data are taken out from the data set E, the data are uniformly scaled to the resolution ratio p x o, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is output
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of a camera, two images at the visual point of an image j are synthesized respectively according to an image i and an image k, the two images are input into a network C, the parameters of the network C and the network B are continuously modified, and iterative optimization is performed, so that the loss of each image in each batch is minimized, the loss of each batch is calculated as the sum of transformation synthesis loss, spatial structure error loss and constant depth loss, wherein the constant depth loss is calculated by using the mean square error of the output result of a module mu of the network B;
(6) on a data set E, three modules of a network C and a network B are trained 50000 times to obtain a model rho
During training, ultrasonic image training data are taken out from the data set E each time, the data are uniformly scaled to the resolution p multiplied by o, the image j and the image pi are input into the network B, and the output tensor of the module mu of the network B is output
As the depth, the tensor L and the tensor O output by the network B are respectively used as a pose parameter and a camera internal parameter, two images at the visual point of the image j are synthesized according to the image i and the image k respectively, the two images are input into the network C, the parameters of the network C and the network B are continuously modified, iterative optimization is carried out, the loss of each image in each batch is enabled to be minimum, an optimal network model parameter rho is obtained after iteration, and the loss of each batch is calculated as the sum of transformation synthesis loss and space structure error loss;
(7) on data set G, three modules of network C and network B were trained 80000 times
During training, CT image training data are taken out from the data set G every time, and are uniformly zoomed to the resolution ratiop x o, inputting the image m and the image sigma into the network B, and the output tensor of the module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of a camera, two images at the viewpoint of an image m are synthesized according to an image L and an image n respectively, the two images are input into a network C, and the loss of each batch of images is minimized by continuously modifying the parameters of the network C and the network B and carrying out iterative optimization, wherein the loss of each batch is calculated as the sum of transformation synthesis loss, spatial structure error loss, constant depth loss and camera translational motion loss Y, the constant depth loss is calculated by using the mean square error of the output result of a module mu of the network B, and the Y is calculated by the output pose parameter of the network B according to the constraint of the camera translational motion;
(8) on a data set G, three modules of a network C and a network B are trained 50000 times to obtain a model rho'
Taking out CT image training data from the data set G each time, uniformly scaling to resolution p x o, inputting the image m and the image sigma into the network B, and outputting tensor of module mu of the network B
As the depth, the output tensor L and the output tensor O of the network B are respectively used as a pose parameter and an internal parameter of the camera, two images at the viewpoint of an image m are synthesized according to an image L and an image n respectively, the two images are input into a network C, parameters of the network C and the network B are continuously modified, iterative optimization is carried out, the loss of each image in each batch is minimized, an optimal network model parameter rho' is obtained after iteration, the loss of each batch is calculated to be the sum of transformation synthesis loss, space structure error loss and translational motion loss Y of the camera, and the Y is calculated from the output pose parameter of the network B according to the constraint of the translational motion of the camera;
and 4, step 4: three-dimensional reconstruction of ultrasound or CT images
Using a self-sampled ultrasound or CT sequence imageUniformly scaling each frame of image to resolution ratio p x o, predicting by using model parameter rho or model parameter rho', inputting image j and image pi into network B for ultrasonic sequence image, inputting image m and image sigma into network B for CT sequence image, and inputting output tensor of module mu of network B
Using the output tensor L and the tensor O of the network B as a pose parameter and an internal parameter of a camera respectively as a depth, selecting key frames according to the following steps, using a first frame in a sequence as a current key frame, using each frame in a sequence image as a target frame in sequence, synthesizing an image at a viewpoint of the target frame by using a pose parameter and an internal parameter of the camera according to the current key frame, calculating an error lambda by using the magnitude of the sum of intensity differences of pixel-by-pixel color channels between the synthesized image and the target frame, synthesizing an image at the viewpoint of the target frame by using a pose parameter and an internal parameter of the camera according to adjacent frames of the target frame, calculating an error gamma by using the magnitude of the sum of intensity differences of pixel-by-pixel color channels between the synthesized image and the target frame, further calculating a synthesized error ratio Z by using a formula (2), and when Z is greater than a threshold eta, 1<η<2, updating the current key frame to the current target frame;
and (3) for any target frame, the resolution ratio of the target frame is scaled to MxN, the three-dimensional coordinates in the camera coordinate system of each pixel of each frame of image are calculated according to the internal parameters of the camera and the reconstruction algorithm of computer vision, further, the viewpoint of the first frame is used as the origin of the world coordinate system, and the three-dimensional coordinates in the world coordinate system of each pixel of each frame of image of the sequence are calculated by utilizing the geometric transformation of three-dimensional space and combining the pose parameters of all key frames.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011623215.0A CN112700534B (en) | 2020-12-30 | 2020-12-30 | Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011623215.0A CN112700534B (en) | 2020-12-30 | 2020-12-30 | Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112700534A true CN112700534A (en) | 2021-04-23 |
| CN112700534B CN112700534B (en) | 2022-07-08 |
Family
ID=75511249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011623215.0A Active CN112700534B (en) | 2020-12-30 | 2020-12-30 | Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112700534B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113689546A (en) * | 2021-08-02 | 2021-11-23 | 华东师范大学 | Cross-modal three-dimensional reconstruction method for ultrasonic or CT image of two-view twin transducer |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109584164A (en) * | 2018-12-18 | 2019-04-05 | 华中科技大学 | Medical image super-resolution three-dimensional rebuilding method based on bidimensional image transfer learning |
| CN109978778A (en) * | 2019-03-06 | 2019-07-05 | 浙江工业大学 | Convolutional neural networks medicine CT image denoising method based on residual error study |
| CN110197254A (en) * | 2019-04-25 | 2019-09-03 | 中国科学院自动化研究所南京人工智能芯片创新研究院 | Line character description method based on depth migration study |
| CN111950404A (en) * | 2020-07-29 | 2020-11-17 | 南京大学 | Single-image three-dimensional reconstruction method based on deep learning video surveillance |
| CN112102276A (en) * | 2020-09-10 | 2020-12-18 | 西安电子科技大学 | Low-field-intensity MR stomach segmentation method based on transfer learning image enhancement |
-
2020
- 2020-12-30 CN CN202011623215.0A patent/CN112700534B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109584164A (en) * | 2018-12-18 | 2019-04-05 | 华中科技大学 | Medical image super-resolution three-dimensional rebuilding method based on bidimensional image transfer learning |
| CN109978778A (en) * | 2019-03-06 | 2019-07-05 | 浙江工业大学 | Convolutional neural networks medicine CT image denoising method based on residual error study |
| CN110197254A (en) * | 2019-04-25 | 2019-09-03 | 中国科学院自动化研究所南京人工智能芯片创新研究院 | Line character description method based on depth migration study |
| CN111950404A (en) * | 2020-07-29 | 2020-11-17 | 南京大学 | Single-image three-dimensional reconstruction method based on deep learning video surveillance |
| CN112102276A (en) * | 2020-09-10 | 2020-12-18 | 西安电子科技大学 | Low-field-intensity MR stomach segmentation method based on transfer learning image enhancement |
Non-Patent Citations (5)
| Title |
|---|
| ASAD B等: "fast 3D reconstruction of Ultrasonic images based on generalized regression neural network", 《WORLD CONGRESS ON MEDICAL PHYSICS AND BIOMEDICAL ENGINEERING 2006》 * |
| MARTIN WEIGERT等: "Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks", 《ARXIV:1704.01510V1 [CS.CV]》 * |
| YINJIE LEI等: "an efficient 3D face recognition approach based on the fusion of novel local low-levelfeatures", 《PATTERN RECOGNITION》 * |
| 唐久驭: "基于卷积神经网络和迁移学习的医学影像辅助诊断研究", 《中国优秀博硕士学位论文全文数据库（硕士） 信息科技辑》 * |
| 雷若鸣: "医学图像三维重建技术研究", 《中国优秀博硕士学位论文全文数据库（硕士） 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113689546A (en) * | 2021-08-02 | 2021-11-23 | 华东师范大学 | Cross-modal three-dimensional reconstruction method for ultrasonic or CT image of two-view twin transducer |
| CN113689546B (en) * | 2021-08-02 | 2023-06-23 | 华东师范大学 | Cross-modal three-dimensional reconstruction method for ultrasound or CT image of two-view twin transducer |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112700534B (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112767532B (en) | Ultrasonic or CT medical image three-dimensional reconstruction method based on transfer learning | |
| CN111310707B (en) | Bone-based graph annotation meaning network action recognition method and system | |
| Kumar | Video based traffic forecasting using convolution neural network model and transfer learning techniques | |
| Fu et al. | Bidirectional 3D quasi-recurrent neural network for hyperspectral image super-resolution | |
| CN113689545B (en) | 2D-to-3D end-to-end ultrasound or CT medical image cross-modal reconstruction method | |
| CN110930378B (en) | Emphysema image processing method and system based on low data demand | |
| CN111583285A (en) | Liver image semantic segmentation method based on edge attention strategy | |
| Bao et al. | SCTANet: A spatial attention-guided CNN-transformer aggregation network for deep face image super-resolution | |
| Gao et al. | Bayesian image super-resolution with deep modeling of image statistics | |
| CN112700534B (en) | Ultrasonic or CT medical image three-dimensional reconstruction method based on feature migration | |
| CN112734906B (en) | Three-dimensional reconstruction method of ultrasonic or CT medical image based on knowledge distillation | |
| CN112700535B (en) | Ultrasonic image three-dimensional reconstruction method for intelligent medical auxiliary diagnosis | |
| CN112734907B (en) | Ultrasonic or CT medical image three-dimensional reconstruction method | |
| CN118015396A (en) | Unsupervised medical image organ segmentation model-based pre-training method | |
| CN113689548B (en) | Medical image three-dimensional reconstruction method based on mutual attention transducer | |
| CN113689542B (en) | Ultrasonic or CT medical image three-dimensional reconstruction method based on self-attention transducer | |
| CN116935044B (en) | Endoscopic polyp segmentation method with multi-scale guidance and multi-level supervision | |
| CN113689544B (en) | Cross-view geometric constraint medical image three-dimensional reconstruction method | |
| CN113689546B (en) | Cross-modal three-dimensional reconstruction method for ultrasound or CT image of two-view twin transducer | |
| CN111696167A (en) | Single image super-resolution reconstruction method guided by self-example learning | |
| Rezaei et al. | Brain abnormality detection by deep convolutional neural network | |
| CN114612305B (en) | Event-driven video super-resolution method based on stereogram modeling | |
| CN113689547B (en) | Ultrasonic or CT medical image three-dimensional reconstruction method of cross-view visual transducer | |
| CN113689543B (en) | Epipolar constrained sparse attention mechanism medical image three-dimensional reconstruction method | |
| CN112419283A (en) | Neural network for estimating thickness and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |