CN117830449A - Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network - Google Patents
Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network Download PDFInfo
- Publication number
- CN117830449A CN117830449A CN202311828772.XA CN202311828772A CN117830449A CN 117830449 A CN117830449 A CN 117830449A CN 202311828772 A CN202311828772 A CN 202311828772A CN 117830449 A CN117830449 A CN 117830449A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- ray
- radiation field
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005855 radiation Effects 0.000 title claims abstract description 62
- 238000000034 method Methods 0.000 title claims abstract description 56
- 210000005036 nerve Anatomy 0.000 title claims abstract description 30
- 230000001537 neural effect Effects 0.000 claims abstract description 33
- 238000009877 rendering Methods 0.000 claims abstract description 21
- 238000005516 engineering process Methods 0.000 claims abstract description 5
- 238000000605 extraction Methods 0.000 claims abstract description 5
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 230000000694 effects Effects 0.000 claims abstract description 4
- 238000005070 sampling Methods 0.000 claims description 21
- 238000003384 imaging method Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000009966 trimming Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 230000015556 catabolic process Effects 0.000 claims description 2
- 230000002708 enhancing effect Effects 0.000 claims 1
- 238000002591 computed tomography Methods 0.000 description 15
- 210000003127 knee Anatomy 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 230000000007 visual effect Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000004913 activation Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 230000000052 comparative effect Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000000137 annealing Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 210000000629 knee joint Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000399 orthopedic effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to a method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network, comprising: acquiring a medical X-ray data set, and preprocessing an X-ray image; constructing an enhanced neural radiation field consisting of position coding, linear latent coding, an MLP network and volume rendering; constructing a discriminator consisting of a feature encoding network module, a feature decoding network module and a discriminator network module to optimize the nerve radiation field, and carrying out feature extraction, reconstruction and discrimination on the real data set and the image generated by the nerve radiation field; in the reasoning stage, a frequency regularization technology is adopted to finely adjust parameters of a current generation model; the invention also relates to a corresponding system, device, processor and storage medium thereof. The method, system, device, processor and storage medium thereof have better effect of reconstructing new view angle image from single Zhang Yixue X-ray image than baseline model.
Description
Technical Field
The invention relates to the technical field of digital images, in particular to the technical field of computer vision, and specifically relates to a method, a system, a device, a processor and a computer readable storage medium for realizing reconstruction of a new view image from a single Zhang Yixue X-ray image based on generation of a nerve radiation field of an countermeasure network.
Background
Three-dimensional medical data generated by Computed Tomography (CT) generally provides sufficient visual information to assist a physician in making a diagnosis. However, this is often accompanied by high costs. Taking CT data as an example, the basic principle is to scan a layer of a certain thickness of the examination region of the human body with an X-ray beam, generate a plurality of slices, and superimpose the slice information, thereby obtaining three-dimensional data, but this requires a long exposure of the patient to higher radiation than in a single medical X-ray image. For cost-effective reasons, a physician will typically choose to judge three-dimensional information from several medical X-ray images, but this is largely dependent on a priori knowledge of the physician. It is therefore a very promising task to assist doctors in diagnosing conditions by generating new view X-ray images from a small number of medical X-ray images. However, the traditional algorithm and the algorithm based on deep learning are often used for obtaining the medical X-ray new view angle image by adopting an indirect imaging scheme, and more medical X-ray images are needed. Such as: the application number is: the invention patent application of CN201711097204.1, the algorithm carries out residual operation on a reconstructed image with artifacts reconstructed from sparse CT projections through a filtered back projection reconstruction algorithm and an artifact image generated by the reconstructed image with artifacts through a residual network, and a clear CT image is restored, but if a new view projection is required to be obtained, a digital reconstruction radiographic image technology is also required to carry out analog imaging; the application number is: the invention patent application of CN202210569044.0 also uses a filtered back projection reconstruction algorithm to obtain a reconstructed image with artifacts, then uses a depth fusion neural network-based optimized reconstructed image, but if a new view projection is desired, a digital reconstructed radiogram technique is also required for analog imaging. Obtaining new view X-ray images from a small number of medical X-ray images is a challenging task.
Disclosure of Invention
It is an object of the present invention to overcome the above-described drawbacks of the prior art by providing a method, system, device, processor and computer readable storage medium thereof for enabling reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network.
To achieve the above object, the method, system, apparatus, processor and computer readable storage medium thereof for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network of the present invention are as follows:
the method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network is mainly characterized by comprising the following steps of:
(1) Acquiring a medical X-ray data set, and preprocessing an X-ray image;
(2) Constructing an enhanced nerve radiation field consisting of position coding, linear potential coding, an MLP network and volume rendering, and finishing mapping from a coordinate domain to a value domain;
(3) Constructing a discriminator consisting of a feature encoding network module, a feature decoding network module and a discriminator network module to optimize the nerve radiation field, and carrying out feature extraction, reconstruction and discrimination on the real data set and the image generated by the nerve radiation field so as to obtain a clearer and real generated image;
(4) In the reasoning stage, the parameters of the current generation model are finely adjusted by adopting a frequency regularization technology so as to obtain accurate new view imaging with a smaller range.
Preferably, the step (1) specifically includes the following steps:
(1.1) downloading a CT data set from a data set official network to obtain original medical CT three-dimensional data;
(1.2) obtaining an X-ray projection medical image with 128X 128 pixels from the original medical CT three-dimensional data in a radiographic reconstruction mode, and processing the zoomed image in a data enhancement mode;
preferably, the data enhancement mode includes:
and randomly carrying out vertical and horizontal translation processing on the image subjected to scaling processing, and carrying out data normalization processing on the image.
Preferably, the step (2) specifically includes the following steps:
(2.1) constructing a position coding module: each pixel of the image corresponds to a ray, and the ray is subjected to discrete sampling to obtain N sampling points. Then, using Fourier feature mapping to construct an orthogonal base formed by sine and cosine functions, mapping the coordinate x and the ray direction d of the sampling point from a low-dimensional space to a high-dimensional space to obtain a coordinate code gamma (x) and a direction code gamma (d), wherein p is any input, and L is the dimension of the raised high dimension, as shown in the following formula:
γ(p)＝(sin(2 0 πp)，cos(2 0 πp)，...，sin(2 L-1 πp)，cos(2 L-1 πp))
(2.2) constructing a linear coding module:
generating slope latent codes Z using a Gaussian distribution with variance of 0.05 and mean of 1 s Generating biased potential codes Z by using Gaussian distribution with variance of 1 and mean of 0 a . Using Z a And Z s Coding the coordinates gamma (x)A row line operation, generating enhanced coordinate code gamma * (x) The specific formula is as follows:
γ * (x)＝γ(x)⊙Z s +Z a
(2.3) constructing an MLP network and calculating: an implicit mapping function is formed by using a multi-layer MLP, firstly, the enhanced coordinate code generated in the step (2.2) is subjected to a coding network to obtain a coordinate code characteristic, on one hand, the point density sigma is obtained through a density decoder, on the other hand, the point density sigma is connected with the direction code generated in the step (2.1), and then, the point color value c is obtained through a color decoder;
(2.4) construct rendering mode and rendering: using a discrete volume rendering formula, according to the color c of the sampling point i on ray r i And density sigma i Rendering the ray to obtain pixel value of corresponding point of the imageAs shown in the following formula, N is the total number of sampling points on the ray r, t i Coordinates for the i-th point:
more preferably, the step (3) specifically includes the following steps:
(3.1) constructing a learnable network module, and extracting features of the input generator image and the real image to obtain a feature map of the image;
(3.2) constructing a learnable feature decoding network module, and restoring the feature diagram code generated by the feature encoding network module to an original image through the module, and carrying out loss calculation with the original image to avoid network breakdown;
(3.3) constructing a learnable authentication network module, classifying the feature map output by the step (3.1) to enable the image generated by the generator to trend to the distribution of the real image, wherein the loss function adopts the R1 regularized unsaturated GAN loss.
More preferably, the step (4) specifically includes:
when a small-range image is generated, frequency regularization is carried out, namely, a mask for coding the coordinates is gradually released along with the increase of the trimming frequency, and the method is concretely implemented by adopting the following formula, wherein j is a certain dimension of the coordinate coding, T is the current trimming frequency, T is the set maximum trimming frequency, K is amplitude, and as follows, the sum of Hadamard products:
γ(x)′＝γ(x)⊙α(t)，
α(t)＝(α 0 ，α 1 ...α j ...α len(γ(x))-1 )
when the frequency mask is completely released, fine tuning can be stopped when a proper index value is reached, and a fine tuned network is used to obtain a small-range new view imaging.
The system for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network by using the method is mainly characterized in that the system comprises:
a linear latent coding module for randomly generating slope codes and bias codes from the Gaussian distribution;
the nerve radiation field module is connected with the linear potential coding module and is used for mapping the coordinate system space domain to the pixel domain by combining the linear potential coding to generate medical X-rays of the new view of each view angle;
the discriminator module is connected with the nerve radiation field module and is used for carrying out feature coding, feature decoding and discrimination on the input image by using the feature coding network module, the feature decoding network module and the discrimination network module so that the image distribution generated by combining the linear potential coding module with the nerve radiation field module tends to be a real image and is finer;
the frequency regularization module is connected with the nerve radiation field module and used for imaging in a small range, and in an inference stage, the frequency regularization module is used for fine tuning the nerve radiation field module through frequency mask construction by frequency regularization so as to enable the generated image in the small range to be finer and more accurate.
The device for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network is mainly characterized by comprising:
a processor configured to execute computer-executable instructions;
a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the method described above for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network.
The processor for realizing the reconstruction of the new view image from the single Zhang Yixue X-ray image based on the nerve radiation field of the generated countermeasure network is mainly characterized in that the processor is configured to execute computer executable instructions, and the computer executable instructions realize the steps of the method for realizing the reconstruction of the new view image from the single Zhang Yixue X-ray image based on the nerve radiation field of the generated countermeasure network when being executed by the processor.
The computer readable storage medium is characterized in that it has stored thereon a computer program executable by a processor to perform the steps of the method for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network as described above.
The method, system, device, processor and computer readable storage medium thereof for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field of an countermeasure network of the present invention are employed, using a latest implicit network representation model as a generator module and a latest generated countermeasure network model as a discriminator module of a medical X-ray image. In order to enhance the network generated image precision and generalization capability, the invention also innovatively introduces a linear latent coding module and a frequency regularization scheme. The linear potential coding module is formed by Gaussian distribution of two different variances and means, and the combination of the linear potential coding module, the coordinate coding and the direction coding can improve the robustness and generalization of the model. The small-range accurate imaging is realized by frequency regularization, and the mask is gradually released along with the increase of fine tuning times in a dynamic frequency mask mode, so that the generalization capability and the low-frequency and high-frequency learning capability of the network are enhanced. The technical scheme performs experimental verification on the knee dataset, and has more prominent effect of reconstructing a new view angle image in a single view compared with a baseline model.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the present invention.
FIG. 2 is a graph of the visual results of the present model using a linear encoding module and its comparison model to achieve reconstruction of other view angles from single Zhang Yixue X-ray in an embodiment of the present invention.
FIG. 3 is a graph of the visual results of the present model using a frequency regularization scheme and its comparison model to achieve small-scale imaging in an embodiment of the present invention.
Detailed Description
In order to more clearly describe the technical contents of the present invention, a further description will be made below in connection with specific embodiments.
Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, the method for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network is implemented, wherein the method comprises the steps of:
(1) Collecting knee X-ray data sets, and preprocessing X-ray images;
(2) Constructing an enhanced nerve radiation field consisting of position coding, linear potential coding, an MLP network and volume rendering, and finishing mapping from a coordinate domain to a value domain;
(3) Constructing a discriminator consisting of a feature encoding network module, a feature decoding network module and a discriminator network module to optimize the nerve radiation field, and carrying out feature extraction, reconstruction and discrimination on the real data set and the image generated by the nerve radiation field so as to obtain a clearer and real generated image;
(4) In the reasoning stage, the parameters of the current generation model are finely adjusted by adopting a frequency regularization technology so as to obtain accurate new view imaging with a smaller range.
As a preferred embodiment of the present invention, the step (1) specifically includes the steps of:
(1.1) downloading a knee CT data set from a data set official network to obtain original knee CT three-dimensional data;
(1.2) obtaining 128X 128 pixel X-ray projection knee images from the original knee CT three-dimensional data by a reconstructed radiogram technique, and processing the scaled images by a data enhancement method;
in practical application, the step (1) specifically includes:
the knee CT data set is downloaded from the data set grid to obtain raw knee X-ray CT three-dimensional data of 512X H pixels in size. Then generating an X-ray projection image with 128X 128 pixels by using a tigre open-source project original knee X-ray CT three-dimensional data simulation X-ray projection mode, and obtaining final training and test images by using a data enhancement mode.
As a preferred embodiment of the present invention, the data enhancement mode includes:
and randomly carrying out vertical and horizontal translation processing on the image subjected to scaling processing, and carrying out data normalization processing on the image to obtain a final training and tested image.
As a preferred embodiment of the present invention, the step (2) specifically includes the following steps:
(2.1) constructing a position coding module: each pixel of the image corresponds to a ray, and the ray is subjected to discrete sampling to obtain N sampling points. Then, using Fourier feature mapping to construct an orthogonal base formed by sine and cosine functions, mapping the coordinate x and the ray direction d of the sampling point from a low-dimensional space to a high-dimensional space to obtain a coordinate code gamma (x) and a direction code gamma (d), wherein p is any input, and L is the dimension of the raised high dimension, as shown in the following formula:
γ(p)＝(sin(2 0 πp)，cos(2 0 πp)，...，sin(2 L-1 πp)，cos(2 L-1 πp))
(2.2) constructing a linear coding module:
generating slope latent codes Z using a Gaussian distribution with variance of 0.05 and mean of 1 s Generating biased potential codes Z by using Gaussian distribution with variance of 1 and mean of 0 a Using Z a And Z s Performing linear operation on the coordinate code gamma (x) to generate enhanced coordinate code gamma * (x) The specific formula is as follows:
γ * (x)＝γ(x)⊙Z s +Z a
(2.3) constructing an MLP network and calculating: an implicit mapping function is formed by using a multi-layer MLP, and the direction codes in the step (2.1) and the enhancement coordinate codes generated in the step (2.2) are decoded into the color c and the density sigma of the sampling points. In particular enhanced coordinate coding gamma * (x) Firstly, obtaining a coordinate coding characteristic through a coding network, then, on one hand, obtaining the point density sigma through a density decoder, and on the other hand, performing connection operation with a direction code gamma (d), and then, obtaining a point color value c through a color decoder;
(2.4) construct rendering mode and rendering: using a discrete volume rendering formula, according to the color c of the sampling point i on ray r i And density sigma i Rendering the ray to obtain pixel value of corresponding point of the imageAs shown in the following formula, N is the total number of sampling points on the ray r, t i Coordinates for the i-th point:
in practical application, the step (2) specifically includes:
step 2.1: position encoder module design:
the position encoder maps the position of the coordinate point and the direction of the ray from a low-dimensional space to a high-dimensional space using fourier feature mapping, mainly using an orthogonal basis composed of sine and cosine functions. L of the coordinate encoder takes 10 and L of the direction encoder takes 4. The dimension of the coordinate x and the direction d is 3, the dimension is changed into 60 and 24 after the gamma (·) coding function, and the dimension of the final square coordinate coding gamma (x) and the direction coding gamma (d) is 63 and 27;
step 2.2: constructing a linear coding module:
the present invention achieves sampling of slope latent codes and bias latent codes by using classical distribution (gaussian distribution). Slope latent coding Z s Is 63, with the first 36 dimensions set to 1, and the variance and mean are 0.05 and 1. Biasing potential code Z a The first 36 dimensions of (3) set to 0, and the variance and mean values were 1 and 0. The variance and the mean of the two can be adjusted according to actual conditions. In obtaining Z s And Z a Then, the gamma (x) generated in the step (2.1) is subjected to linear operation to generate gamma with 63 dimensions * (x) The formula is as follows:
γ * (x)＝γ(x)⊙Z s +Z a
step 2.3: building an MLP network and calculating:
the MLP network mainly adopts the network architecture of NeRF to construct an implicit mapping function. The NERF network architecture consists of an encoding module, a density decoder and a color decoder. The coding module is an 8-layer simple full-connection layer, the output of each layer is subjected to RuLU activation, and the 1 st layer and the 4 th layer are subjected to residual operation. The density decoder is composed of a full-connection layer, and decodes the output of the coding module to output the density sigma. The color decoder consists of three fully connected layers, wherein the first layer fuses the input gamma (d) with the output of the coding module, and the second two layers are used for decoding, wherein the output of the second layer is used for performing ReLU activation and outputting the color c.
Step 2.4: construct rendering mode and rendering:
discretizing a continuous domain volume rendering formula according to the volume rendering principle of computer graphics, as shown in the following formula, wherein N is the total number of sampling points on the ray r, t i Coordinates for the i-th point:
the ray r is generated by this formula to the value of the corresponding pixel.
As a preferred embodiment of the present invention, the step (3) specifically includes the following steps:
(3.1) constructing the learnable feature encoding network module, and carrying out feature extraction on the input generator image and the real image to obtain a feature map of the image;
(3.2) constructing the learnable feature decoding network module, restoring the feature code generated by the feature encoding network module into an original image through the module, carrying out loss calculation on the feature code and the original image, and preventing the generator and the discriminator from being crashed, wherein the step is only adopted when training a real data set;
(3.3) constructing the learnable authentication network module, classifying the feature map output by (3.1) to enable the image generated by the generator to trend to the distribution of the real image, wherein the loss function adopts the R1 regularized unsaturated GAN loss.
In practical application, the step (3) specifically includes:
step 3.1: constructing a feature coding network module and carrying out feature coding on the image:
the improvement is made on the discriminator architecture of GRAF, the characteristic diagram of the input image is extracted, and the characteristic coding network consists of 3 coding modules. Each coding module consists of 2 groups of convolution layers, batch normalization layers and a leakage ReLU activation function;
step 3.2: constructing a feature decoding network module and decoding image features:
and decoding the output of the feature encoding network module to output an image, and performing perception loss calculation with the input image, so that the discriminator module avoids the dilemma of mode collapse and the like. The decoding network is composed of 4 decoding modules and a reconstruction layer, wherein the decoding modules comprise an up-sampling layer and a convolution layer, and the up-sampling module adopts a nearest neighbor algorithm. Batch normalization and Gating Linear Unit (GLU) activation functions are used to further adjust the output after each decoding module is executed. The reconstruction layer consists of a convolution layer and a hyperbolic tangent function (Tanh) activation layer;
step 3.3: constructing an authentication network module and classifying images:
the construction identification network module carries out classification judgment on the feature map, the feature map consists of a convolution layer, batch normalization, a Leak Rule and the convolution layer, and loss calculation is carried out by adopting R1 regularized unsaturated GAN loss, so that the image generated by the generator tends to the distribution of a real image.
As a preferred embodiment of the present invention, the step (4) specifically includes:
in practical applications, small-range precise imaging is often more common, so that frequency regularization is adopted to complete precise small-range imaging. The method comprises the following steps: with the increase of the fine tuning times, gradually releasing a mask alpha (T) for coordinate coding, and specifically adopting the following formula, wherein j is a certain dimension of the coordinate coding, gamma (x) is T which is the current fine tuning times, T is set to 2000, and K is set to 33:
γ(x)′＝γ(x)⊙α(t)，
α(t)＝(α 0 ，α 1 ...α j ...α len(γ(x))-1 )
when the frequency mask is completely released, fine tuning can be stopped when a proper index value is reached, and a fine tuned network is used to obtain a small-range new view imaging.
The system for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network using the method described above, wherein the system comprises:
a linear latent coding module for randomly generating slope codes and bias codes from the Gaussian distribution;
the neural radiation field module is connected with the linear potential coding module and is used for mapping the coordinate system space domain to the pixel domain by combining the linear potential coding to generate knee X-ray images of new views of all visual angles;
the discriminator module is connected with the nerve radiation field module and is used for carrying out feature coding, feature decoding and discrimination on the input image by using the feature coding network module, the feature decoding network module and the discrimination network module so that the image distribution generated by combining the linear potential coding module with the nerve radiation field module tends to be a real image and is finer;
the frequency regularization module is connected with the nerve radiation field module and used for imaging in a small range, and in an inference stage, the frequency regularization module is used for fine tuning the nerve radiation field module through frequency mask construction by frequency regularization so as to enable the generated image in the small range to be finer and more accurate.
In a specific embodiment of the present invention, the single image reconstruction method adopting the present technical solution is tested as follows:
(1) Experimental data set
The present invention provides a data set from a natural human knee joint using a study staff from the university of denver, usa, orthopedics biomechanics center. The dataset contained CT and MRI images of 7 knee specimens. Training and testing the test dataset is obtained by simulating X-ray projections of the source dataset by means of reconstructed radiological imaging techniques. The 6 groups in the constructed dataset were trained and one group tested.
(2) Training process
The training image is scaled to 128 x 128 pixels and the data enhancement mode is randomly translated up, down, left, right. The initial learning rate is set to be le-4, the cosine annealing algorithm is adopted to attenuate the learning rate, the batch is set to be 8, and 100000 rounds of training are performed.
(3) Reasoning process
In the face of full-range imaging, the neural radiation field is fixed, and the linear latent code is finely tuned until a better result is obtained, and the fine tuning is stopped, so that a new view angle image is generated. Frequency regularization is employed for small range imaging.
(4) Test results
In order to verify the effectiveness of the proposed method, three new view angle generation methods based on a nerve radiation field are selected in the experiment, and the three methods are used as a comparison group for comparison experiments.
Comparative group 1 is: new methods of view generation are mentioned in the publication PixelNeRF Neural Radiance Fields from One or Few Images.
Comparative group 2 is: new methods of generating visual angles are mentioned in the publication FreeNeRF: improving Few-shot Neural Rendering with Free Frequency Regularization.
The comparative group 3 is: new methods of view angle generation are mentioned in the publication GRAF Generative Radiance Fields for D-Aware Image Synthesis.
As shown in fig. 2, a visual result diagram of the linear coding module and the contrast model thereof proposed in the present application for reconstructing a new view angle image from a single Zhang Yixue X-ray image is provided. The first column represents the actual original X-ray image at that view angle, and the other columns generate X-ray images at that view angle. Each instance first acts as a new view and the second acts generates a depth map. Compared with the comparison model, the linear coding model provided by the invention can better generate a new view angle image.
As shown in fig. 3, the frequency regularization scheme and the contrast model thereof provided by the invention realize a visual result graph imaged from a single X-ray in a small range. Compared with a comparison model, the frequency regularization scheme provided by the invention can better generate a new view angle image in a small range.
The device for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network comprises:
a processor configured to execute computer-executable instructions;
a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the method described above for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network.
The processor for implementing reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network, wherein the processor is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of the method for implementing reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network described above.
The computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the method for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network as described above.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.
The method, system, device, processor and computer readable storage medium thereof for reconstructing a new view image from a single Zhang Yixue X-ray image using the generated countermeasure network based neural radiation field implementation of the present invention uses the latest implicit network representation model or the latest generated countermeasure network based implicit network representation model as a discriminator module for medical X-ray images. In order to enhance the network generated image precision and generalization capability, the invention also innovatively introduces a linear latent coding module and a frequency regularization scheme. The linear potential coding module is formed by Gaussian distribution of two different variances and means, and the combination of the linear potential coding module, the coordinate coding and the direction coding can improve the robustness and generalization of the model. The small-range accurate imaging is realized by frequency regularization, and the mask is gradually released along with the increase of fine tuning times in a dynamic frequency mask mode, so that the generalization capability and the low-frequency and high-frequency learning capability of the network are enhanced. The technical scheme performs experimental verification on the knee dataset, and has more prominent effect of reconstructing a new view angle image in a single view compared with a baseline model.
In this specification, the invention has been described with reference to specific embodiments thereof. It will be apparent, however, that various modifications and changes may be made without departing from the spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
Claims (9)
1. A method for effecting reconstruction of a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network, said method comprising the steps of:
(1) Acquiring a medical X-ray data set, and preprocessing an X-ray image;
(2) Constructing an enhanced nerve radiation field consisting of position coding, linear potential coding, an MLP network and volume rendering, and finishing mapping from a coordinate domain to a value domain;
(3) Constructing a discriminator consisting of a feature encoding network module, a feature decoding network module and a discriminator network module to optimize the nerve radiation field, and carrying out feature extraction, reconstruction and discrimination on the real data set and the image generated by the nerve radiation field so as to obtain a clearer and real generated image;
(4) In the reasoning stage, the parameters of the current generation model are finely adjusted by adopting a frequency regularization technology so as to obtain accurate new view imaging with a smaller range.
2. The method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network according to claim 1, wherein said step (1) comprises the steps of:
(1.1) downloading a medical CT dataset from a dataset bureau to obtain original three-dimensional CT data thereof;
(1.2) obtaining an X-ray projection medical image with 128 multiplied by 128 pixels from the original three-dimensional CT data by a radiographic reconstruction technique, and processing the zoomed image by a data enhancement method, specifically, randomly carrying out vertical and horizontal translation processing on the zoomed image, and carrying out data normalization on the image, thereby completing data preprocessing operation.
3. The method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network as recited in claim 2, wherein said step (2) comprises the steps of:
(2.1) constructing a position coding module: each pixel of the image corresponds to a ray, the ray is subjected to discrete sampling to obtain N sampling points, then Fourier feature mapping is used to construct an orthogonal base formed by sine and cosine functions, the coordinate x of the sampling points and the ray direction d are mapped from a low-dimensional space to a high-dimensional space, and coordinate coding gamma (x) and direction coding gamma (d) are obtained, as shown in the following formula, wherein p is any input, and L is the raised high-dimensional dimension:
γ(p)＝(sin(2 0 πp)，cos(2 0 πp)，...，sin(2 L-1 πp)，cos(2 L-1 πp))
(2.2) constructing a linear latent encoding module:
generating slope latent codes Z using a Gaussian distribution with variance of 0.05 and mean of 1 s Generating biased latent codes Z using a Gaussian distribution with variance of 1 and mean of 0 a Using Z a And Z s Performing linear operation on the coordinate code gamma (x) to generate enhanced coordinate code gamma * (x) Enhancing the editability of the nerve radiation field, the specific formula is as follows:
γ * (x)＝γ(x)⊙Z s +Z a
(2.3) constructing an MLP network and calculating: using a multi-layer MLP to form an implicit mapping function, coding the direction of the step (2.1) into gamma (d) and the enhanced coordinate obtained in the step (2.2) * (x) Decoding into color c and density sigma of sampling points, in particular enhancement coordinate encoding gamma * (x) The coordinate coding feature is obtained through a coding network, the point density sigma is obtained through a density decoder, the connection operation is carried out with the direction coding gamma (d), and the point color is obtained through a color decoderA value c;
(2.4) construct rendering mode and rendering: using a discrete volume rendering formula, according to the color c of the sampling point i on ray r i And density sigma i Rendering the ray to obtain pixel value of corresponding point of the imageAs shown in the following formula, N is the total number of sampling points on the ray r, t i T is the position of the ith point i+1 Is the position of the (i+1) th point, sigma i Density of the ith point, t j Is the position of the j-th point, t j+1 Is the position of the j+1st point, sigma j Density of j-th point:
4. a method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network as claimed in claim 3, wherein said step (3) comprises the steps of:
(3.1) constructing a learnable feature coding network module, and extracting features of an input generator image and a real image to obtain a feature map of the image;
(3.2) constructing a learnable feature decoding network module, restoring the feature diagram code generated by the feature encoding network module into an original image through the module, and carrying out loss calculation on the feature diagram code and the original image to prevent the generator and the discriminator from falling into breakdown;
(3.3) constructing a learnable authentication network module, classifying the feature map output by the step (3.1) to enable the image generated by the generator to trend to the distribution of the real image, wherein the loss function adopts the R1 regularized unsaturated GAN loss.
5. The method for reconstructing a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network according to claim 4, wherein said step (4) comprises:
when a small-range image is generated, frequency regularization is carried out, namely, a mask alpha (T) for coding coordinates is gradually released along with the increase of the trimming frequency, and the small-range image is concretely realized by adopting the following formula, wherein gamma (x)' is a frequency mask, j is a certain dimension of the coordinate coding, T is the current trimming frequency, T is the set maximum trimming frequency, K is amplitude, and as a result, the following is Hadamard product:
γ(x)′＝γ(x)⊙α(t)，
α(t)＝(α 0 ，α 1 …α j …α len(γ(x))-1 )
when the frequency mask gamma (x)' is completely released, the fine tuning can be stopped when a proper index value is reached, and a small-range new view imaging is obtained by using the network after fine tuning.
6. A system for implementing reconstruction of new view images from single Zhang Yixue X-ray images based on generating neural radiation fields against a network using the method of any one of claims 1 to 5, said system comprising:
a linear latent coding module for randomly generating slope codes and bias codes from the Gaussian distribution;
the nerve radiation field module is connected with the linear potential coding module and is used for mapping the coordinate system space domain to the pixel domain by combining the linear potential coding to generate medical X-ray images of new views of each view angle;
and the discriminator module is connected with the nerve radiation field module and is used for carrying out feature encoding, feature decoding and discrimination on the input image by using the feature encoding network module, the feature decoding network module and the discrimination network module so that the image distribution generated by combining the linear potential encoding module with the nerve radiation field module tends to be a real image and is finer.
7. An apparatus for effecting reconstruction of a new view angle image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network, said apparatus comprising:
a processor configured to execute computer-executable instructions;
a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the method of any one of claims 1 to 5 for implementing reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network.
8. A processor for effecting reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network, wherein the processor is configured to execute computer-executable instructions that, when executed by the processor, effect the steps of the method of effecting reconstruction of a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network as claimed in any one of claims 1 to 6.
9. A computer-readable storage medium, having stored thereon a computer program executable by a processor to perform the steps of the method of any one of claims 1 to 6 for reconstructing a new view image from a single Zhang Yixue X-ray image based on generating a neural radiation field against a network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311828772.XA CN117830449A (en) | 2023-12-28 | 2023-12-28 | Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311828772.XA CN117830449A (en) | 2023-12-28 | 2023-12-28 | Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117830449A true CN117830449A (en) | 2024-04-05 |
Family
ID=90505406
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311828772.XA Pending CN117830449A (en) | 2023-12-28 | 2023-12-28 | Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117830449A (en) |
-
2023
- 2023-12-28 CN CN202311828772.XA patent/CN117830449A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11126914B2 (en) | Image generation using machine learning | |
| CN110462689B (en) | Tomographic reconstruction based on deep learning | |
| US11039805B2 (en) | Deep learning based estimation of data for use in tomographic reconstruction | |
| US10438350B2 (en) | Material segmentation in image volumes | |
| US20220245400A1 (en) | Autonomous segmentation of three-dimensional nervous system structures from medical images | |
| EP3716214B1 (en) | Medical image processing apparatus and method for acquiring training images | |
| KR102560911B1 (en) | Image processing apparatus, image processing method, and storage medium | |
| Jia et al. | GPU-based fast low-dose cone beam CT reconstruction via total variation | |
| US7978886B2 (en) | System and method for anatomy based reconstruction | |
| US9824468B2 (en) | Dictionary learning based image reconstruction | |
| CN112396560A (en) | System and method for deblurring medical images using a deep neural network | |
| Hegazy et al. | Image denoising by transfer learning of generative adversarial network for dental CT | |
| Wang et al. | Inner-ear augmented metal artifact reduction with simulation-based 3D generative adversarial networks | |
| US20160335785A1 (en) | Method of repeat computer tomography scanning and system thereof | |
| CN113592968B (en) | Method and device for reducing metal artifacts in tomographic images | |
| CN117830449A (en) | Method for realizing reconstruction of new view image from single Zhang Yixue X-ray image based on nerve radiation field of generation countermeasure network | |
| Vlontzos et al. | 3D probabilistic segmentation and volumetry from 2D projection images | |
| Rossi et al. | Comparison of Supervised and Unsupervised Approaches for the Generation of Synthetic CT from Cone-Beam CT. Diagnostics 2021; 11: 1435 | |
| Pandey et al. | A Framework for Mathematical Methods in Medical Image Processing | |
| Lee et al. | Framework for dual‐energy‐like chest radiography image synthesis from single‐energy computed tomography based on cycle‐consistent generative adversarial network | |
| Gajera | Ct-Scan Image Denoising with Generative Adversarial Networks | |
| Rahman | Low Signal Algorithms in X-ray Computed Tomography | |
| Ahmed et al. | Clinical application of image fusion | |
| Khudair | Faculty of Engineering Department of Biomedical Engineering | |
| Ostrek et al. | Hypodensity extractor: A phantom study |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |