WO2022094911A1 - 一种共享权重的双区域生成对抗网络及其图像生成方法 - Google Patents

一种共享权重的双区域生成对抗网络及其图像生成方法 Download PDF

Info

Publication number
WO2022094911A1
WO2022094911A1 PCT/CN2020/127030 CN2020127030W WO2022094911A1 WO 2022094911 A1 WO2022094911 A1 WO 2022094911A1 CN 2020127030 W CN2020127030 W CN 2020127030W WO 2022094911 A1 WO2022094911 A1 WO 2022094911A1
Authority
WO
WIPO (PCT)
Prior art keywords
artifact
image
feature
generator
discriminator
Prior art date
Application number
PCT/CN2020/127030
Other languages
English (en)
French (fr)
Inventor
胡战利
郑海荣
梁栋
刘新
邓富权
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Priority to PCT/CN2020/127030 priority Critical patent/WO2022094911A1/zh
Publication of WO2022094911A1 publication Critical patent/WO2022094911A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/003—Reconstruction from projections, e.g. tomography
    • G06T11/008—Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/404—Angiography
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/441—AI-based methods, deep learning or artificial neural networks

Definitions

  • the present invention relates to the technical field of medical image processing, and more particularly, to a dual-area generative confrontation network with shared weights and an image generation method thereof.
  • CCTA Coronary Computed Tomography Angiography
  • imaging methods it refers to a non-invasive method that uses computers and X-rays to obtain a patient's cardiac tomographic image after intravenous injection of an appropriate contrast agent.
  • imaging methods CCTA has the advantages of short scanning time, extensive component information and non-invasive visualization of the vessel wall.
  • CCTA-acquired images may exhibit motion artifacts and require re-examination.
  • a large amount of X-ray exposure will cause the cumulative effect of radiation doses, increasing the possibility of various diseases, thereby affecting human physiological functions, destroying human tissues and organs, and even endangering the life safety of patients. Therefore, research and development to remove motion artifacts in images that produce artifacts has important scientific significance and broad application prospects for the current medical diagnosis field.
  • motion artifact during coronary CT imaging is due to the displacement of image pixels when the CT acquires projection data from different angles.
  • the degree of motion artifact depends on the rate of displacement and the correction result of the image reconstruction algorithm.
  • motion artifacts are eliminated from two aspects: the first is to control the heart rate, reduce the subject's heart rate, prolong the cardiac cycle, slow down the coronary artery movement and prolong the time of the low-velocity movement of the coronary artery, thereby reducing the time resolution in imaging.
  • the second is to improve the temporal resolution.
  • the solution to improve the time resolution is usually carried out from the hardware aspect or from the software aspect.
  • the time resolution is improved by increasing the rotation speed of the tube ball, using a wide-body detector and adopting a dual-detector technology.
  • the use of multi-sector reconstruction technology, image reconstruction technology based on compressed sensing (Prior Image Constrained Compressed Sensing, PICCS), motion estimation and compensation algorithms and motion correction technology (Snap Shot Freeze, SSF) can effectively improve the temporal resolution .
  • the increase of the rotation speed of the tube is limited by the physical characteristics; the use of multi-detector technology is limited by space; and the use of wide-body detector technology is limited by economic costs.
  • the use of multi-sector reconstruction technology needs to maintain the patient's heart rate stable, and is limited by the ball rotation time and scanning pitch; image reconstruction technology based on compressed sensing has not been verified; motion estimation and Compensation algorithms rely on a large number of calculations and evaluations; motion correction technology needs to acquire a phase with relatively small relative motion artifacts and good image quality in the image, and can eliminate motion artifacts through complex calculations.
  • the purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a novel generative adversarial network and an image generation method thereof, which utilizes a generative adversarial network based on two-region multi-generators with shared weights to eliminate motion artifacts of medical images, In addition, while removing artifacts and generating image features, the peak signal-to-noise ratio and structural similarity of the image are improved, thereby obtaining a clearer medical image that meets the needs of diagnosis.
  • a dual-area generative adversarial network with shared weights which includes an artifact-free area processing module and an artifact-existing area processing module, wherein:
  • the artifact-free area processing module includes a first generator and a first discriminator, the first generator includes a first feature generator and a first feature parser, the first feature generator is used to extract the image features of the artifact-free image, the first A feature parser is used to re-analyze the feature image generated by the second feature generator with the artifact area into an image without the artifact, and the first discriminator is used to calculate the adversarial loss of the artifact-free area;
  • the artifact region processing module includes a second generator and a second discriminator, wherein the second generator includes a second feature generator, an artifact generator, a reconstruction parser and a second feature parser, and the second feature generator is used for Extract the image features of the artifact image; the artifact generator is used to extract the artifact features of the artifact image; the reconstruction parser is used to add the feature images generated by the second feature generator and the artifact generator, and get The feature image and the image with artifact are used as constraints; the second feature parser is used to add the feature image of the artifact-free area and the artifact feature image of the artifact area to generate the artifact-free image as the base. image, the second discriminator is used to compute the adversarial loss for the artifacted regions.
  • the second generator includes a second feature generator, an artifact generator, a reconstruction parser and a second feature parser, and the second feature generator is used for Extract the image features of the artifact image
  • an image generation method comprising:
  • the medical image to be processed is divided into image blocks containing artifacts and image blocks not containing artifacts, which are used as inputs to the artifact-existing area processing module and the artifact-free area processing module, respectively, to obtain a generated image.
  • the present invention has the advantages that, by designing a two-region multi-generator generative adversarial network, the motion artifacts of medical images can be effectively eliminated, and in terms of hardware, it is not affected by physical characteristics, space and economic costs. Limitations; on the software side, artifact-free images can be obtained without complex calculations and without being affected by changes in the patient's heart rate.
  • FIG. 1 is a structural diagram of a feature generator according to an embodiment of the present invention.
  • FIG. 2 is a structural diagram of a feature parser according to an embodiment of the present invention.
  • FIG. 3 is a structural diagram of a residual module according to an embodiment of the present invention.
  • FIG. 4 is a structural diagram of a discriminator according to an embodiment of the present invention.
  • FIG. 5 is a structural diagram of an artifact generator according to an embodiment of the present invention.
  • FIG. 6 is a structural diagram of a content discriminator according to an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of a training process of a content discriminator according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a discriminator training process for an artifact area and an artifact-free area according to an embodiment of the present invention
  • FIG. 9 is a schematic diagram of a training process of generators with an artifact region and an artifact-free region according to an embodiment of the present invention.
  • FIG. 10 is a comparison diagram of experimental results according to an embodiment of the present invention.
  • the present invention designs a generative adversarial network with shared weight training.
  • the images of the generator and the parser can be significantly improved.
  • the sharing of features improves the learning efficiency of the generator and the parser for the same kind of features, and generates image features from the artifact area and the artifact-free area, which improves the detail expressiveness of the generated image.
  • the motion artifacts of the image can be effectively eliminated.
  • CCTA images are used as an example for description, but it should be understood that the present invention can also be applied to other types of medical image noise reduction and de-artifact removal in addition to CCTA image de-artifact removal.
  • the shared weight dual-area generative adversarial network provided by the present invention generally includes an artifact-free area processing module and an artifact-existing area processing module, and the specific design process and operation steps are as follows.
  • Step S110 Design an artifact-free area processing module.
  • the artifact-free region processing module includes a generator Gen b and a discriminator Dis b .
  • the generator further includes a feature generator Encoder cont_b and a feature parser Decoder cont_b , the feature generator is used to extract the features of the image without artifacts, and the feature parser re-parses the feature image generated by the first feature generator in the area with artifacts into a feature image without artifacts. Artifact image.
  • the discriminator is used to compute the adversarial loss for artifact-free regions.
  • the feature generator is shown in Fig. 1.
  • the input images are artifact-free images of size 1 ⁇ H ⁇ W, and the generator is trained to obtain the feature images of the artifact-free images.
  • the generator is trained to obtain the feature images of the artifact-free images.
  • first perform the convolution operation on the input image use the 7 ⁇ 7 convolution operation to obtain the feature image of l ⁇ H ⁇ W, and add multiple channels to the feature map.
  • Angular learns features of images.
  • the feature image is down-sampled through two 4 ⁇ 4 convolutions to obtain a reduced feature image.
  • RES 4-layer residual block
  • the low-dimensional and high-dimensional feature information of the feature image is learned to obtain
  • the feature image is the final output of the feature generator.
  • each convolutional layer is followed by a non-linear activation function leaky ReLU.
  • the feature parser is shown in Figure 2, and the input size is The feature image of , learns two layers of information through the residual module, and restores the feature image to the original image size l ⁇ H ⁇ W through two upsampling (Up sample) and a convolutional layer with a convolution kernel of 5 ⁇ 5. After another layer of convolutional layer with a convolution kernel of 7 ⁇ 7, the feature image is restored to the same artifact-free image as the original size (1 ⁇ H ⁇ W). Except for the last convolutional layer, the rest of the convolutional The activation function leaky ReLU is included after the layer.
  • the process of the feature parser is equivalent to the inverse process of the feature generator.
  • Figure 3 is a schematic diagram of the structure of the residual module in Figures 1 and 2.
  • the feature image is obtained through two layers of 3 ⁇ 3 convolution, and the first layer of convolution is followed by nonlinear activation.
  • the function leaky ReLU the feature image obtained by the second layer of convolution is added to the input image to obtain the final output of the residual module. This way of using multiple layers of residual modules is able to learn the deep and shallow information of the feature image.
  • Figure 4 is an example of the structure of the discriminator.
  • An image of 1 ⁇ H ⁇ W is input.
  • a least squares loss function can be used as a constraint.
  • the least squares loss function of the generated image itself is used.
  • the least squares loss function of the generated image and the target image is used.
  • Step S120 an artifact area processing module is designed.
  • the artifact region processing module includes a generator Gen a and a discriminator Dis a .
  • the generator further includes a feature generator Encoder cont_a , an artifact generator Encoder art_a , a reconstruction parser Decoder recons_a and a feature parser Decoder cont_a .
  • the feature generator is used to extract the image features of the image with artifacts; the artifact generator is to extract the artifact features of the image with artifacts; the reconstruction parser is used to add the feature images generated by the feature generator and the artifact generator, and analyze the The obtained feature image and the image with artifact are used as constraints; the feature parser is used to add the feature image of the artifact-free area and the artifact feature image of the artifact area to generate the artifact-free image as the base. image.
  • the discriminator is used to compute the adversarial loss for artifact regions.
  • FIG. 1 For the structures of the feature generator, the reconstruction parser and the feature parser included in the artifact region processing module, the examples shown in FIG. 1 , FIG. 2 and FIG. 2 can also be used respectively.
  • Figure 5 is an example of the structure of the artifact generator. After one convolution layer with a convolution kernel of 7 ⁇ 7, and then two convolution layers with a convolution kernel of 4 ⁇ 4 and a stride of 2, it is also downsampling. . The size of the feature image obtained by downsampling should be consistent with the feature generator and the feature generator of the artifact-free area. Finally, a convolutional layer with a convolution kernel of 1 ⁇ 1 is used to obtain the feature image of the artifact.
  • the reconstructed parser and feature parser with artifact regions and the feature parser without artifact regions should have the same size of feature images obtained by up-sampling.
  • Step S130 designing a content discriminator.
  • the designed two-region generative adversarial network may further include a content discriminator or a style discriminator.
  • the content discriminator Dis cont is used to extract the features of the image style, which can be realized by a traditional convolutional neural network, as shown in Figure 6, which includes 5 layers of convolutional layers with a convolution kernel of 3 ⁇ 3 and a stride of 2.
  • the pooling layer (Avg pool) and finally through a convolution layer with a convolution kernel of 1 ⁇ 1, the feature image of the image style is obtained.
  • the content discriminator is used to calculate the adversarial loss for the feature generator of the artifact-free area and the artifact area generator. By adding the content discriminator, it plays an important role in maintaining the training stability of the feature generators of the two areas, which is beneficial to improve the Universality of dual-region generative adversarial networks for images with different content and styles.
  • Step S140 train a dual-region generative adversarial network with the set joint loss function as the target.
  • the overall training process includes: step S141 , training a content discriminator; step S142 , training a discriminator with artifact and artifact-free regions; and step S143 , training a generator with artifact and artifact-free regions.
  • step S141 for the content discriminator, in the training process of updating its own weight, as shown in Figure 7, the loss value is calculated using the least squares loss function, wherein the feature image label generated by the input image is 0, and the target image The resulting feature image has the label 1.
  • the formula is as follows:
  • x represents the input image and y represents the target image.
  • 0 and 1 represent the labels of artifact- and no-artifact regions, respectively.
  • step S142 for the discriminator with the artifact area and the artifact-free area, the training process of updating its own weight is shown in Figure 8.
  • the input target image through the Encoder cont_b and the Decoder cont_b , Obtain the output image b, input the target image and image b to the discriminator Dis b , and obtain the loss value loss dis_b ;
  • the area processing module with artifacts input the input image, pass through the feature generator Encoder cont_a and the artifact generator Encoder art_a respectively , obtain two feature images, add the two feature images, and then pass through the reconstruction parser Decoder recons_a to obtain the output image a, input the input image and image a into the discriminator Dis a , and calculate the loss value loss dis_a ; , input the feature images obtained by the feature generator Encoder cont_b of the artifact-free area and the feature generator Encoder cont_a of the artifact area into the content discrimin
  • the loss functions of each discriminator in the training process are:
  • the loss (loss) calculation formula of the content discriminator is expressed as:
  • GAN_weight represents the set hyperparameters.
  • step S143 for generators with and without artifacts, the training process for updating their own weights is shown in Figure 9.
  • the input image marked as x
  • the target image marked as y
  • the generator and discriminator are updated to complete the overall training process.
  • the weight training of the entire framework includes steps S141 and S142 in addition to those shown in FIG. 9 .
  • the joint loss function contains the following:
  • the input image x passes through the feature generator Encoder cont_a and the artifact generator Encoder art_a in the artifact area to obtain the feature images x c and x a respectively, add x c and x a , and obtain the reconstructed image through the reconstruction parser Decoder recons_a , calculate the reconstruction loss between the reconstructed image and the input image, the formula is expressed as:
  • x c passes through the feature parser Decoder cont_b of the artifact-free area to obtain the feature image x cc , and x cc then passes through the feature generator Decoder cont_b of the artifact-free area to obtain the feature image x ccc ; the target image y passes through the artifact-free area
  • the feature parser Decoder cont_b obtains the feature image y c ; y c and x a are added to obtain y ca , and y ca passes through the feature parser Decoder cont_a of the artifact area to obtain the generated image y cr with artifacts; y cr passes through
  • the feature generator Encoder cont_a with the artifact area and the artifact generator Encoder art_a obtain the feature images y crc and y cra respectively , the feature image y cra and x ccc are added, and then pass the feature parser Decoder cont_a with the artifact area, get
  • the feature images generated in the area with artifacts are used to calculate the content feature images generated for the first time, the artifact feature images generated for the first time, the content feature images generated for the second time, and the artifact feature images generated for the second time in the same training.
  • Kernel loss the formula is expressed as:
  • the feature image y cr generated by the artifact area is used to calculate the adversarial loss of the feature image with the least squares loss, and the formula is expressed as:
  • the target image is input to the feature generator and feature parser of the artifact-free area, and the reconstruction loss function is calculated between the feature image obtained from the feature parser and the target image, and the formula is expressed as:
  • the feature images generated in the artifact-free area are used to calculate the kernel loss of the first generated content feature image and the second generated content feature image in the same training.
  • the formula is expressed as:
  • the feature image x cc generated by the artifact-free area is calculated with the least squares loss for the adversarial loss of the image of this feature, and the formula is expressed as:
  • the perceptual loss calculation formula for the artifact-free area is expressed as:
  • the content loss of the content discriminator is expressed as:
  • GAN w is the calculated weight of the adversarial loss in the area with artifacts, the adversarial loss in the area without artifacts and the content loss of the style discriminator;
  • KL w is the content kernel loss in the area with artifact, the artifact kernel in the area with artifact Calculation weights of loss and content kernel loss for artifact-free areas;
  • KL cycw is the calculated weight for cyclic content kernel loss with artifact areas, cyclic artifact kernel loss with artifact areas, and cyclic content kernel loss in artifact-free areas ;
  • VGG w is the calculated weight of the perceptual loss of the two regions;
  • REC w is the calculated weight of the reconstruction loss of the area with and without artifacts, respectively;
  • REC_cyc_w is the cycle consistency loss of the area with and without artifacts Calculate weights.
  • the present invention can effectively ensure the stability of the model and the quality of the output image by using multiple generators and discriminators and designing a joint multiple loss function. Moreover, compared with the traditional hardware-based method for collecting additional data to perform the de-artifact operation, the present invention can complete the de-artifact operation without additional information. Training the model with two regions can be done both supervised and unsupervised.
  • the existing Adam optimizer can be used for optimization, and the training network gradually reaches a convergence state.
  • the CCTA images without artifacts and those with artifacts can be converted into multiple image blocks of the same size, and the image blocks are used as the input of the artifact-free area and the area with artifacts respectively, and the artifact-free CCTA image Reference.
  • the artifact removal of medical images can be achieved by using a trained two-region generative adversarial network.
  • the medical image to be processed is divided into image blocks containing artifacts and image blocks not containing artifacts, and the generated images are obtained as inputs to the artifact region processing module and the artifact-free region processing module, respectively.
  • the area without artifacts and the areas containing artifacts in the image to be processed are respectively converted into multiple image blocks of the same size, and the image blocks are regarded as areas without artifacts and areas with artifacts respectively. shadow area input.
  • the actual processing process of medical images is similar to the training process, and will not be repeated here.
  • the present invention uses the shared weight dual-region generative adversarial network to improve the generalization ability of the model and its application in the field of medical images.
  • Artifact images are directly upscaled to high-quality images using deep learning networks without additional information.
  • the present invention can also be applied to the field of image super-resolution after proper modification.
  • the present invention extracts image features in two regions, and generates image features from the intersection of the artifact region and the artifact-free region, and further uses the patch method between training models to improve the model's ability to perceive local information, Thereby, the detail representation of the generated medical images is significantly improved.
  • the present invention may be a system, method and/or computer program product.
  • the computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.
  • a computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM erasable programmable read only memory
  • flash memory static random access memory
  • SRAM static random access memory
  • CD-ROM compact disk read only memory
  • DVD digital versatile disk
  • memory sticks floppy disks
  • mechanically coded devices such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.
  • Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.
  • the computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • the computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages.
  • Source or object code written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect).
  • LAN local area network
  • WAN wide area network
  • custom electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)
  • FPGAs field programmable gate arrays
  • PDAs programmable logic arrays
  • Computer readable program instructions are executed to implement various aspects of the present invention.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本发明公开了一种共享权重的双区域生成对抗网络及其图像生成方法。该生成对抗网络包括无伪影区域处理模块和有伪影区域处理模块，其中，无伪影区域处理模块包含第一特征生成器、第一特征解析器和第一鉴别器；有伪影区域处理模块包含第二特征生成器、伪影生成器、重建解析器、第二特征解析器和第二鉴别器。本发明通过多次反复使用相同的生成器和解析器进行训练，可以提高生成器和解析器的图像特征的共享性，并且使用两个区域对模型进行训练既可以进行有监督又可以进行无监督训练。利用本发明能够消除医学图像的运动伪影，并在生成图像特征的同时，提高了图像峰值信噪比和结构相似度，从而得到符合诊断需求的更清晰的医学图像。

Description

一种共享权重的双区域生成对抗网络及其图像生成方法 技术领域
本发明涉及医学图像处理技术领域，更具体地，涉及一种共享权重的双区域生成对抗网络及其图像生成方法。
背景技术
医学图像的伪影去除一直是研究人员致力于解决的难题。以医疗图像冠状动脉计算机断层扫描血管造影(Cornary Computed Tomography Angiography，CCTA)为例，其是指受试者通过静脉注射适当造影剂后，利用计算机和X射线来获取病人心脏断层图像的非侵入式影像学检测方法。CCTA具有扫描时间短，成分信息广泛和以非侵入式可视化血管壁等优点，适用于可疑冠心病的诊断，冠状动脉搭桥术的随访，评估瓣膜性心脏病和评估心脏质量。然而，CCTA获取的影像可能会出现运动伪影，需要重新进行检查。而大量的X射线照射会出现辐射剂量的累计效应，增加各种疾病发生的可能性，进而影响人体生理机能，破坏人体组织器官，甚至危害到患者的生命安全。因此，研究和开发在产生伪影的影像下去除运动伪影，对于目前的医疗诊断领域具有重要的科学意义和广阔的应用前景。
冠状动脉CT成像过程中运动伪影的形成是由于图像像素在CT获取不同角度投影数据时发生了位移，运动伪影的程度取决于位移的速率和图像重建算法的校正结果。通常从两个方面消除运动伪影：第一是控制心率，降低受检者的心率、延长心动周期、减慢冠状动脉运动和延长冠状动脉低速运动的时间，从而降低成像时对时间分辨率的需求；第二是提高时间分辨率，基于MDCT冠状动脉成像基本原理和运动伪影产生的原因，要实现CCTA成像并规避运动伪影，需要将特定扫描方式下获得的最高时间分辨率与冠状动脉运动幅度最小的时相相匹配。
在现有技术中，提高时间分辨率的方案通常从硬件方面或从软件方面 进行。具体地，在硬件方面，从提高管球旋转速度，采用宽体探测器和采用双探测器技术出发来提高时间分辨率。在软件方面，采用多扇区重建技术，基于压缩感知的图像重建技术(Prior Image Constrained Compressed Sensing，PICCS)，运动估算和补偿算法与运动校正技术(Snap Shot Freeze，SSF)可以有效提高时间分辨率。
目前，对于从硬件方面提高时间分辨率的方案，提高管球旋转速度受物理特性的限制；采用多探测器技术则受到空间的限制；而采用宽体探测器技术则受限于经济成本。对于从软件方面提高时间分辨率的方案，采用多扇区重建技术需要维持患者的心率稳定，并且受管球旋转时间和扫描螺距的限制；基于压缩感知的图像重建技术尚未得到验证；运动估算和补偿算法依赖大量的计算和评估；运动校正技术需要采集图像中相对运动伪影较小、图像质量较好的时相，经过复杂计算才能消除运动伪影。
发明内容
本发明的目的是克服上述现有技术的缺陷，提供一种新型的生成对抗网络及其图像生成方法，利用基于共享权重的两区域多生成器的生成对抗网络来消除医学图像的运动伪影，并且在去除伪影，生成图像特征的同时，提高了图像峰值信噪比和结构相似度，从而得到符合诊断需求的更清晰的医学图像。
根据本发明的第一方面，提供一种共享权重的双区域生成对抗网络，其包括无伪影区域处理模块、有伪影区域处理模块，其中：
无伪影区域处理模块包括第一生成器和第一鉴别器，第一生成器包含第一特征生成器和第一特征解析器，第一特征生成器用于提取无伪影图像的图像特征，第一特征解析器用于将有伪影区域的第二特征生成器生成的特征图像重新解析为无伪影的图像，第一鉴别器用于计算无伪影区域的对抗损失；
有伪影区域处理模块包含第二生成器和第二鉴别器，其中第二生成器包含第二特征生成器、伪影生成器、重建解析器和第二特征解析器，第二特征生成器用于提取有伪影图像的图像特征；伪影生成器用于提取有伪影 图像的伪影特征；重建解析器用于将第二特征生成器和伪影生成器所生成的特征图像相加，解析后得到的特征图像与有伪影图像作为约束；第二特征解析器用于将无伪影区域的特征图像和有伪影区域的伪影特征图像相加，生成以无伪影图像为底的有伪影图像，第二鉴别器用于计算有伪影区域的对抗损失。
根据本发明的第二方面，提供一种图像生成方法，包括：
以设定的联合损失函数为目标训练本发明提供的共享权重的双区域生成对抗网络；
将待处理的医学图像划分为包含伪影的图像块和未含伪影的图像块，分别作为所述有伪影区域处理模块和无伪影区域处理模块的输入，获得生成图像。
与现有技术相比，本发明的优点在于，通过设计两区域多生成器的生成对抗网络，能够有效消除医学图像的运动伪影，并且在硬件方面，不受物理特性、空间和经济成本的限制；在软件方面，不需要复杂的计算且不受患者心率变化的影响，即可得到消除伪影的图像。
通过以下参照附图对本发明的示例性实施例的详细描述，本发明的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本发明的实施例，并且连同其说明一起用于解释本发明的原理。
图1是根据本发明一个实施例的特征生成器结构图；
图2是根据本发明一个实施例的特征解析器结构图；
图3是根据本发明一个实施例的残差模块结构图；
图4是根据本发明一个实施例的鉴别器结构图；
图5是根据本发明一个实施例的伪影生成器结构图；
图6是根据本发明一个实施例的内容鉴别器结构图；
图7是根据本发明一个实施例的内容鉴别器的训练过程示意图；
图8是根据本发明一个实施例的有伪影区域和无伪影区域的鉴别器训 练过程示意图；
图9是根据本发明一个实施例的有伪影区域和无伪影区域生成器的训练过程示意图；
图10是根据本发明一个实施例的实验结果对比图。
具体实施方式
现在将参照附图来详细描述本发明的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本发明及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
为了解决医学图像的运动伪影的问题，本发明设计了一种共享权重训练的生成对抗网络，通过多次反复使用相同的生成器和解析器进行训练，可以显著提高生成器和解析器的图像特征的共享性，从而提高生成器和解析器对同种特征的学习效率，并且从有伪影区域和无伪影区域相互交叉生成图像特征，提高了生成图像的细节表现力，从而在图像生成过程中能够有效消除图像的运动伪影。
为清楚起见，本文以CCTA图像为例进行说明，但应理解的是，除应用于CCTA图像去伪影外，本发明也可应用于其他类型的医学图像降噪和去伪影。
本发明提供的共享权重双区域生成对抗网络整体上包括无伪影区域 处理模块和有伪影区域处理模块，具体设计过程和操作步骤如下。
步骤S110：设计无伪影区域处理模块。
在一个实施例中，无伪影区域处理模块包括生成器Gen b和鉴别器Dis b。生成器进一步包含特征生成器Encoder cont_b和特征解析器Decoder cont_b，特征生成器用于提取无伪影的图像的特征，特征解析器将有伪影区域第一特征生成器生成的特征图像重新解析为无伪影的图像。鉴别器用于计算无伪影区域的对抗损失。
特征生成器如图1所示，输入大小为1×H×W的无伪影图像，训练生成器以获得无伪影图像的特征图像。为了获取无伪影图像的特征图像，首先对输入的图像做卷积操作，使用7×7卷积操作获得l×H×W的特征图像，为特征图增加多个通道，多个通道从不同角度学习图像的特征。然后，经过两次4×4卷积对特征图像进行下采样，获得缩小的特征图像。接着，经过4层残差模块(residual block，RES)，学习特征图像低维和高维的特征信息，获得
Figure PCTCN2020127030-appb-000001
的特征图像，该特征图像即为特征生成器的最终输出。图1中，每一个卷积层都紧接着非线性激活函数leaky ReLU。
特征解析器如图2所示，输入大小为
Figure PCTCN2020127030-appb-000002
的特征图像，经过残差模块学习深浅两层信息，通过两次上采样(Up sample)和卷积核为5×5的卷积层将特征图像恢复到原始图像的尺寸l×H×W，再经过一层卷积核为7×7的卷积层，将特征图像还原到与原尺寸(1×H×W)相同的无伪影图像，除最后一层卷积层外，其余卷积层后包含激活函数leaky ReLU。特征解析器的过程相当于特征生成器的逆过程。
图3是图1和图2中残差模块的结构示意，以输入l×H×W图像为例，经过两层3×3卷积得到特征图像，其中第一层卷积后接非线性激活函数leaky ReLU，第二层卷积得到的特征图像与输入图像相加得到残差模块的最终输出。这种使用多层残差模块的方式，能够学习特征图像的深层和浅层信息。
图4是鉴别器的结构示例，输入1×H×W的图像，经过4层卷积核为3×3，步长stride＝2的卷积层，接着通过一层卷积核为1×1，stride为1的卷积层，得到
Figure PCTCN2020127030-appb-000003
的特征图像。在训练过程中，可使用最小二乘法损 失函数作为约束。在对生成器更新过程中，使用的是生成图像自身的最小二乘法损失函数。而在对鉴别器更新过程中，使用的是生成图像和目标图像的最小二乘法损失函数。
步骤S120，设计有伪影区域处理模块。
在一个实施例中，有伪影区域处理模块包含生成器Gen a和鉴别器Dis a。生成器进一步包含特征生成器Encoder cont_a、伪影生成器Encoder art_a、重建解析器Decoder recons_a和特征解析器Decoder cont_a。特征生成器用于提取有伪影图像的图像特征；伪影生成器是提取有伪影图像的伪影特征；重建解析器用于将由特征生成器和伪影生成器所生成的特征图像相加，解析后得到的特征图像与有伪影图像作为约束；特征解析器用于将无伪影区域的特征图像和有伪影区域的伪影特征图像相加，生成以无伪影图像为底的有伪影图像。鉴别器用于计算有伪影区域的对抗损失。
对于有伪影区域处理模块所包含的特征生成器、重建解析器和特征解析器的结构也可分别采用图1、图2和图2的示例。
图5是伪影生成器的结构示例，经过一层卷积核为7×7的卷积层，然后经过两层卷积核为4×4，stride为2的卷积层，同时也是下采样。下采样得到的特征图像尺寸与特征生成器以及无伪影区域的特征生成器应保持一致。最后通过一层卷积核为1×1的卷积层，得到伪影的特征图像。
此外，有伪影区域的重建解析器和特征解析器以及无伪影区域的特征解析器在上采样得到的特征图像尺寸应当保持一致。
步骤S130，设计内容鉴别器。
优选地，所设计的双区域生成对抗网络可进一步包括内容鉴别器或称风格鉴别器。
内容鉴别器Dis cont用于提取图像风格的特征，可通过传统的卷积神经网络实现，如图6所示，其包括5层卷积核为3×3，stride为2的卷积层、平均池化层(Avg pool)、并最后通过一层卷积核为1×1的卷积层，获得图像风格的特征图像。内容鉴别器用于对无伪影区域和有伪影区域生成器的特征生成器计算对抗损失，通过增加内容鉴别器对于维持两个区域的特征生成器的训练稳定性具有重要的作用，有利于提高双区域生成对抗网络对 不同内容、不同风格图像的普适性。
步骤S140，以设定的联合损失函数为目标训练双区域生成对抗网络。
简言之，整体训练过程包括：步骤S141，训练内容鉴别器；步骤S142，训练有伪影和无伪影区域的鉴别器；步骤S143，训练有伪影和无伪影区域的生成器。
下文所有Encoder、Decoder和Dis分别对应特征图像通过相应下标的生成器、解析器和鉴别器的处理，得到处理后的特征图像。
具体地，在步骤S141中，对于内容鉴别器，在更新自身权重的训练过程中，如图7，使用最小二乘损失函数计算损失值，其中，输入图像生成的特征图像标签为0，目标图像生成的特征图像标签为1。公式如下：
Figure PCTCN2020127030-appb-000004
其中，x表示输入图像，y表示目标图像。0和1分别代表有伪影区域和无伪影区域的标签。
在步骤S142中，对于有伪影区域和无伪影区域的鉴别器，更新自身权重的训练过程如图8所示，对于无伪影区域处理模块，输入目标图像，经过Encoder cont_b和Decoder cont_b，得到输出图像b，将目标图像和图像b输入到鉴别器Dis b，得到损失值loss dis_b；对于有伪影区域处理模块，输入输入图像，分别经过特征生成器Encoder cont_a和伪影生成器Encoder art_a，得到两个特征图像，将两个特征图像相加，再经过重建解析器Decoder recons_a，得到输出图像a，将输入图像与图像a输入到鉴别器Dis a中，计算得到损失值loss dis_a；同时，将无伪影区域的特征生成器Encoder cont_b和有伪影区域的特征生成器Encoder cont_a得到的特征图像输入到内容鉴别器Dis cont，计算得到特征图像out_a和out_b。
在一个实施例中，训练过程各鉴别器的损失函数分别为：
内容鉴别器的loss(损失)计算公式表示为：
Figure PCTCN2020127030-appb-000005
由公式(2)可知，在训练两个区域的鉴别器的过程中，为了使内容鉴别器起到稳定两个区域的特征生成器的技术效果，使其生成的特征都是基于图像的特征，训练过程中，没有真实图像和生成图像之分，因此设置 标签为0.5。
无伪影区域鉴别器的loss计算公式表示为：
Figure PCTCN2020127030-appb-000006
有伪影区域鉴别器的loss计算公式表示为：
Figure PCTCN2020127030-appb-000007
结合超参数计算，得到的总体loss值计算公式表示为：
loss total＝GAN _weight×(loss dis_cont+loss dis_b+loss dis_a)    (5)
其中，GAN_weight表示设定的超参数。
在步骤S143中，对于有伪影和无伪影区域的生成器，更新自身权重的训练过程如图9所示，针对输入图像(标记为x)和目标图像(标记为y)，对整个框架的生成器和鉴别器进行更新，完成整体训练的过程。
具体地，整个框架的权重训练除了图9所示，还包含了步骤S141和步骤S142的内容。因此联合损失函数包含以下的内容：
输入图像x经过有伪影区域的特征生成器Encoder cont_a和伪影生成器Encoder art_a，分别得到特征图像x c和x a，将x c和x a相加，通过重建解析器Decoder recons_a获得重建图像，计算重建图像与输入图像的重建损失，公式表示为：
Figure PCTCN2020127030-appb-000008
x c通过无伪影区域的特征解析器Decoder cont_b，得到特征图像x cc，x cc再通过无伪影区域的特征生成器Decoder cont_b，从而得到特征图像x ccc；目标图像y通过无伪影区域的特征解析器Decoder cont_b得到特征图像y c；y c和x a相加获得y ca，y ca通过有伪影区域的特征解析器Decoder cont_a，得到生成的有伪影图像y cr；y cr通过有伪影区域的特征生成器Encoder cont_a与伪影生成器Encoder art_a分别得到特征图像y crc和y cra，特征图像y cra与x ccc相加，再通过有伪影区域的特征解析器Decoder cont_a，得到理论上与输入图像一致的生成图像
Figure PCTCN2020127030-appb-000009
称为有伪影区域的循环一致性损失函数：
Figure PCTCN2020127030-appb-000010
以有伪影区域生成的特征图像分别计算在同一次训练中第一次生成内容特征图像、第一次生成伪影特征图像、第二次生成内容特征图像、第 二次生成伪影特征图像的核损失，公式表示为：
Figure PCTCN2020127030-appb-000011
Figure PCTCN2020127030-appb-000012
Figure PCTCN2020127030-appb-000013
Figure PCTCN2020127030-appb-000014
有伪影区域生成的特征图像y cr以最小二乘损失计算该特征的图像的对抗损失，公式表示为：
Figure PCTCN2020127030-appb-000015
以预训练的VGG19作为感知网络，v(y cr)和v(y)分别代表y cr和y通过感知网络得到的特征图像，而α＝in(x)代表对x进行规范化处理，则有伪影区域的感知损失的计算公式表示为：
Figure PCTCN2020127030-appb-000016
目标图像输入到无伪影区域的特征生成器和特征解析器，从特征解析器中得到的特征图像与目标图像之间计算重建损失函数，公式表示为：
Figure PCTCN2020127030-appb-000017
将y crc输入到无伪影区域的特征生成器中，得到特征图像
Figure PCTCN2020127030-appb-000018
对
Figure PCTCN2020127030-appb-000019
和目标图像计算循环一致性损失，称为无伪影区域的循环一致性损失：
Figure PCTCN2020127030-appb-000020
以无伪影区域生成的特征图像分别计算在同一次训练中第一次生成内容特征图像和第二次生成内容特征图像核损失，公式表示为：
Figure PCTCN2020127030-appb-000021
Figure PCTCN2020127030-appb-000022
无伪影区域生成的特征图像x cc以最小二乘损失计算该特征的图像的对抗损失，公式表示为：
Figure PCTCN2020127030-appb-000023
无伪影区域的感知损失计算公式表示为：
Figure PCTCN2020127030-appb-000024
内容鉴别器的内容损失表示为：
Figure PCTCN2020127030-appb-000025
共享权重的双区域训练生成对抗网络的联合损失函数表示为：
Figure PCTCN2020127030-appb-000026
其中GAN w是有伪影区域的对抗损失、无伪影区域的对抗损失和风格鉴别器的内容损失的计算权重；KL w是有伪影区域的内容核损失、有伪影区域的伪影核损失和无伪影区域的内容核损失的计算权重；KL cycw是有伪影区域的循环内容核损失、有伪影区域的循环伪影核损失和无伪影区域的循环内容核损失的计算权重；VGG w是两个区域的感知损失的计算权重；REC w分别为有伪影区域和无伪影区域的重建损失计算权重；REC_cyc_w是有伪影区域和无伪影区域的循环一致性损失的计算权重。
综上，本发明通过采用多个生成器和鉴别器，设计联合多重损失函数，能够有效保证模型的稳定性以及输出图像的质量。并且，相对于传统的基于硬件方法采集额外数据进行去伪影操作，本发明不需要额外信息即可完成去伪影操作。使用两个区域对模型进行训练既可以进行有监督又可以进行无监督训练。
应理解的是，在整个网络训练优化过程，可使用现有的Adam优化器来优化，训练网络逐步达到收敛状态。训练时，可将不含伪影和含有伪影的CCTA图像转换成多个大小相同的图像块，将图像块分别作为无伪影区域和有伪影区域的输入，将无伪影的CCTA图像作为参考。
进一步地，利用经训练的双区域生成对抗网络即可实现医学图像的伪影去除。例如，将待处理的医学图像划分为包含伪影的图像块和未含伪影的图像块，分别作为有伪影区域处理模块和无伪影区域处理模块的输入，获得生成图像。又如，为了提高对于局部信息的感知能力，将待处理图像的不含伪影和含有伪影的区域分别转换成多个大小相同的图像块，将图像块分别作为无伪影区域和有伪影区域的输入。对医学图像的实际处理过程与训练过程类似，在此不再赘述。
综上所述，本发明使用共享权重的双区域生成对抗网络提高模型的泛化能力以及其医学图像领域的应用。使用深度学习网络直接将伪影图像提 升成高质量图像而无需其他信息。除应用于去伪影和降噪外，本发明经过适当更改后，也可应用于图像超分辨领域。此外，本发明在两个区域内提取图像的特征，从有伪影区域和无伪影区域相互交叉生成图像特征，并且在训练模型之间可进一步使用patch方法提高模型对于局部信息的感知能力，从而显著提高生成的医学图像的细节表现力。
为进一步验证本发明的效果，进行了实验。参见图10所示，从左到右，分别为输入图像，生成图像以及目标图像。可以看出，本发明的方法可以有效提高图像的峰值信噪比和结构相似度，同时，可以在一定程度上恢复图像细节信息。
应理解的是，在不违背本发明精神和范围的前提下，本领域技术人员可对上述实施例进行适当的改变或变型，例如对于特征生成器、特征解析器、伪影生成器、重建解析器、内容鉴别器等设置更多或更少的卷积层，设置不同的卷积核、卷积步长等。又如，对于有伪影区域的生成器和无伪影区域的生成器可采用相同或不同的结构等。
本发明可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本发明的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本发明操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本发明的各个方面。
这里参照根据本发明实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本发明的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计 算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本发明的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本发明的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本发明的范围由所附权利要求来限定。

Claims (10)

  1. 一种共享权重的双区域生成对抗网络，包括无伪影区域处理模块、有伪影区域处理模块，其中：
    无伪影区域处理模块包括第一生成器和第一鉴别器，第一生成器包含第一特征生成器和第一特征解析器，第一特征生成器用于提取无伪影图像的图像特征，第一特征解析器用于将有伪影区域的第二特征生成器生成的特征图像重新解析为无伪影的图像，第一鉴别器用于计算无伪影区域的对抗损失；
    有伪影区域处理模块包含第二生成器和第二鉴别器，其中第二生成器包含第二特征生成器、伪影生成器、重建解析器和第二特征解析器，第二特征生成器用于提取有伪影图像的图像特征；伪影生成器用于提取有伪影图像的伪影特征；重建解析器用于将第二特征生成器和伪影生成器所生成的特征图像相加，解析后得到的特征图像与有伪影图像作为约束；第二特征解析器用于将无伪影区域的特征图像和有伪影区域的伪影特征图像相加，生成以无伪影图像为底的有伪影图像，第二鉴别器用于计算有伪影区域的对抗损失。
  2. 根据权利要求1所述的共享权重的双区域生成对抗网络，其中，还包括内容鉴别器，所述内容鉴别器用于针对无伪影区域的第一特征生成器和有伪影区域的第二特征生成器计算对抗损失。
  3. 根据权利要求1所述的共享权重的双区域生成对抗网络，其中，第一特征生成器依次包括多层卷积层和多层残差模块，所述多层卷积层被设置为利用多个通道从不同角度学习图像特征并对特征图像进行下采样，所述多层残差模块被设置为学习特征图像低维和高维的特征信息。
  4. 根据权利要求1所述的共享权重的双区域生成对抗网络，其中，所述伪影生成器通过设置多层卷积层提取有伪影图像的伪影特征并进行下采样，下采样得到的特征图像尺寸与第二特征生成器和无伪影区域的第一特征生成器保持一致。
  5. 一种图像生成方法，包括：
    以设定的联合损失函数为目标训练根据权利要求1至4任一项所述的 共享权重的双区域生成对抗网络；
    将待处理的医学图像划分为包含伪影的图像块和未含伪影的图像块，分别作为所述有伪影区域处理模块和无伪影区域处理模块的输入，获得生成图像。
  6. 根据权利要求5所述的图像生成方法，其中，所述共享权重的双区域生成对抗网络包括内容鉴别器，所述内容鉴别器用于针对无伪影区域的第一特征生成器和有伪影区域的第二特征生成器计算对抗损失，训练过程包括：
    训练内容鉴别器，以最小二乘损失函数计算损失值；
    训练有伪影区域处理模块的第二鉴别器和无伪影区域处理模块的第一鉴别器；
    训练有伪影区域处理模块的第二生成器和无伪影区域处理模块的第一生成器。
  7. 根据权利要求5所述的图像生成方法，其中，训练有伪影区域处理模块的第二鉴别器和无伪影区域处理模块的第一鉴别器包括：
    对于无伪影区域处理模块，输入目标图像，经过第一特征生成器和第一特征解析器，得到输出图像b，将目标图像和图像b输入到第一鉴别器，得到损失值loss dis_b；对于有伪影区域处理模块，输入输入图像，分别经过第二特征生成器和伪影生成器，得到两个特征图像，将两个特征图像相加，再经过重建解析器，得到输出图像a，将输入图像与图像a输入到第二鉴别器中，计算得到损失值；并且，将无伪影区域处理模块的第一特征生成器和有伪影区域处理模块的第二特征生成器得到的特征图像输入到内容鉴别器，计算得到特征图像out_a和out_b。
  8. 根据权利要求6所述的图像生成方法，其中，所述基于共享权重的双区域生成对抗网络中鉴别器的联合损失函数表示为：
    loss total＝GAN_weight×(loss dis_cont+loss dis_b+loss dis_a)
    其中，loss dis_cont是内容鉴别器的损失，loss dis_b是无伪影区域处理模块的第一鉴别器的损失，loss dis_a是有伪影区域处理模块的第二鉴别器的损失，GAN_weight是设定的超参数。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其中，该程序被处理器执行时实现根据权利要求5至8中任一项所述方法的步骤。
  10. 一种计算机设备，包括存储器和处理器，在所述存储器上存储有能够在处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现权利要求5至8中任一项所述的方法的步骤。
PCT/CN2020/127030 2020-11-06 2020-11-06 一种共享权重的双区域生成对抗网络及其图像生成方法 WO2022094911A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/127030 WO2022094911A1 (zh) 2020-11-06 2020-11-06 一种共享权重的双区域生成对抗网络及其图像生成方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/127030 WO2022094911A1 (zh) 2020-11-06 2020-11-06 一种共享权重的双区域生成对抗网络及其图像生成方法

Publications (1)

Publication Number Publication Date
WO2022094911A1 true WO2022094911A1 (zh) 2022-05-12

Family

ID=81456916

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/127030 WO2022094911A1 (zh) 2020-11-06 2020-11-06 一种共享权重的双区域生成对抗网络及其图像生成方法

Country Status (1)

Country Link
WO (1) WO2022094911A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115086670A (zh) * 2022-06-13 2022-09-20 梧州学院 一种面向高清显微视频的低码率编解码方法及系统
CN115797611A (zh) * 2023-02-10 2023-03-14 真健康（北京）医疗科技有限公司 三维医学图像模态转换模型训练方法及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190128989A1 (en) * 2017-11-01 2019-05-02 Siemens Healthcare Gmbh Motion artifact reduction of magnetic resonance images with an adversarial trained network
US20190197358A1 (en) * 2017-12-21 2019-06-27 International Business Machines Corporation Generative Adversarial Network Medical Image Generation for Training of a Classifier
US20190377047A1 (en) * 2018-06-07 2019-12-12 Siemens Healthcare Gmbh Artifact Reduction by Image-to-Image Network in Magnetic Resonance Imaging
CN110570492A (zh) * 2019-09-11 2019-12-13 清华大学 神经网络训练方法和设备、图像处理方法和设备以及介质
CN110675461A (zh) * 2019-09-03 2020-01-10 天津大学 一种基于无监督学习的ct图像恢复方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190128989A1 (en) * 2017-11-01 2019-05-02 Siemens Healthcare Gmbh Motion artifact reduction of magnetic resonance images with an adversarial trained network
US20190197358A1 (en) * 2017-12-21 2019-06-27 International Business Machines Corporation Generative Adversarial Network Medical Image Generation for Training of a Classifier
US20190377047A1 (en) * 2018-06-07 2019-12-12 Siemens Healthcare Gmbh Artifact Reduction by Image-to-Image Network in Magnetic Resonance Imaging
CN110675461A (zh) * 2019-09-03 2020-01-10 天津大学 一种基于无监督学习的ct图像恢复方法
CN110570492A (zh) * 2019-09-11 2019-12-13 清华大学 神经网络训练方法和设备、图像处理方法和设备以及介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115086670A (zh) * 2022-06-13 2022-09-20 梧州学院 一种面向高清显微视频的低码率编解码方法及系统
CN115086670B (zh) * 2022-06-13 2023-03-10 梧州学院 一种面向高清显微视频的低码率编解码方法及系统
CN115797611A (zh) * 2023-02-10 2023-03-14 真健康（北京）医疗科技有限公司 三维医学图像模态转换模型训练方法及设备

Similar Documents

Publication Publication Date Title
Kang et al. Cycle‐consistent adversarial denoising network for multiphase coronary CT angiography
US20220383565A1 (en) Deep learning based three-dimensional reconstruction method for low-dose pet imaging
Rohkohl et al. Improving best‐phase image quality in cardiac CT by motion correction with MAM optimization
Tang et al. A fully four‐dimensional, iterative motion estimation and compensation method for cardiac CT
Song et al. Deep learning-based automatic segmentation of images in cardiac radiography: a promising challenge
JP2020036877A (ja) 反復的画像再構成フレームワーク
Yuan et al. SIPID: A deep learning framework for sinogram interpolation and image denoising in low-dose CT reconstruction
WO2022094911A1 (zh) 一种共享权重的双区域生成对抗网络及其图像生成方法
US10867375B2 (en) Forecasting images for image processing
Ko et al. Rigid and non-rigid motion artifact reduction in X-ray CT using attention module
Xiao et al. STIR-net: deep spatial-temporal image restoration net for radiation reduction in CT perfusion
Zhang et al. CT super-resolution using multiple dense residual block based GAN
CN112419173A (zh) 一种由pet图像生成ct图像的深度学习框架和方法
Su et al. A deep learning method for eliminating head motion artifacts in computed tomography
CN112419175A (zh) 一种共享权重的双区域生成对抗网络及其图像生成方法
Li et al. Learning non-local perfusion textures for high-quality computed tomography perfusion imaging
Liu et al. MRCON-Net: Multiscale reweighted convolutional coding neural network for low-dose CT imaging
CN112991220A (zh) 一种基于多重约束的卷积神经网络校正图像伪影的方法
Li et al. A comprehensive survey on deep learning techniques in CT image quality improvement
CN117813055A (zh) 用于从快速spect扫描和ct图像合成spect图像的多模态和多尺度特征聚合
Müller et al. Evaluation of interpolation methods for surface‐based motion compensated tomographic reconstruction for cardiac angiographic C‐arm data
Yin et al. Unpaired low-dose CT denoising via an improved cycle-consistent adversarial network with attention ensemble
WO2022011690A1 (zh) 一种自监督学习方法及应用
CN112102251A (zh) 一种分割影像的方法及装置、电子设备和存储介质
WO2022094779A1 (zh) 一种由pet图像生成ct图像的深度学习框架和方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20960372

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20960372

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20960372

Country of ref document: EP

Kind code of ref document: A1