CN109325931A - Based on the multi-modality images fusion method for generating confrontation network and super-resolution network - Google Patents

Based on the multi-modality images fusion method for generating confrontation network and super-resolution network Download PDF

Info

Publication number
CN109325931A
CN109325931A CN201810962126.5A CN201810962126A CN109325931A CN 109325931 A CN109325931 A CN 109325931A CN 201810962126 A CN201810962126 A CN 201810962126A CN 109325931 A CN109325931 A CN 109325931A
Authority
CN
China
Prior art keywords
network
image
super
resolution
fusion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810962126.5A
Other languages
Chinese (zh)
Inventor
蔺素珍
杨晓莉
李大威
王丽芳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
North University of China
Original Assignee
North University of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by North University of China filed Critical North University of China
Priority to CN201810962126.5A priority Critical patent/CN109325931A/en
Publication of CN109325931A publication Critical patent/CN109325931A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10088—Magnetic resonance imaging [MRI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Abstract

The present invention relates to image interfusion methods, more particularly to multi-modality images fusion method, specially based on the multi-modality images fusion method for generating confrontation network and super-resolution, this method carries out as follows: designing and constructs generation confrontation network, network structure obtains a generation model by the dynamic balance training of generator and arbiter using the depth residual error neural network of design；Construct the super-resolution network based on convolutional layer；Multiband/multi-modal source images input is generated into model, obtains preliminary blending image；Trained super-resolution network is input an image into again, obtains the final blending image of high quality.This method realizes multiband/multi-modality images neural network fusion end to end, avoids the difficulty that Image Multiscale multi-direction decomposition and fusion rule design is carried out according to priori knowledge, realizes network self-adapting fusion.

Description

Based on the multi-modality images fusion method for generating confrontation network and super-resolution network
Technical field
The present invention relates to image interfusion method more particularly to multi-modality images fusion methods, specially based on generation confrontation The multi-modality images fusion method of network and super-resolution network.
Background technique
Multiband/multi-mode imaging system has been widely used in various fields such as military affairs, medical treatment, industrial detections, image Fusion is one of the key technology that these systems realize high-precision intelligent detection.Current image fusion technology is broadly divided into sky Domain and frequency domain two major classes.The former algorithm is simple, speed is fast, has more utilization in hardware system, but syncretizing effect is limited；The latter More targetedly fusion rule is respectively adopted by the multiple dimensioned multi-direction decomposition result to original image, fusion effect can be improved Fruit is current research hotspot, but often algorithm is complicated, poor dependent on artificial parameter or priori knowledge, generalization ability, cannot be certainly Adapt to fusion, it is difficult to meet the detection needs of complex scene.
Recently there is image interfusion method of the research and probe based on deep learning, deep learning effectively solves to a certain extent Conventional method of having determined problems faced, such as document " Multi-focus image fusion with a deep Convolutional neural network " (Information Fusion 36 (2017) 191-207) convolutional Neural net Activity level measurement and fusion rule are used network implementations by network extraction source characteristics of image together；Document " stacks convolution based on depth Point that the image co-registration of neural network " (Chinese journal of computers Vol.40No.11) is decomposed using neural network is stacked as low-and high-frequency Class device merges the low-and high-frequency image of network output by different fusion rules.Although these methods improve computation rate, Reduce the complicated sex work that rule and parameter is manually set, but all only realize a certain local algorithm in fusion process from It adapts to, obtains characteristic pattern or decomposed and reconstituted filter such as adaptive, there is no realization really adaptive fusions, and with Synchronous fusion two images are object, are not able to satisfy multiband/multi-modal detection needs.
For this reason, it may be necessary to a kind of method neural network based, to solve the problems, such as the adaptive fusion of multi-modality images.
Summary of the invention
The present invention is proposed based on generating confrontation network and oversubscription to solve the problems, such as the adaptive fusions of multi-modality images The multi-modality images fusion method of resolution network.
The present invention adopts the following technical scheme that realization: based on the multimode for generating confrontation network and super-resolution network State image interfusion method, comprising the following steps:
It designs and constructs generation confrontation network: generating confrontation network and be divided into two models of generator and arbiter；Generator It is made of three convolutional layers and seven residual blocks, wherein each residual block includes two layers of convolution, is modified after each convolutional layer Linear unit (Rectified Linear Unit, ReLU) non-linearization；Arbiter by six convolutional layers, standard three Layer residual unit block and one connect and compose entirely, each convolutional layer carries out BatchNorm (BN) data other than input layer and returns One change operates, and after BN, LeakyReLU (LReLU) activation primitive is used to promote network nonlinear degree；
Building super-resolution network: super-resolution network is made of three-layer coil product, first layer convolution extraction characteristic pattern, and second Layer convolution realizes that the Nonlinear Mapping of characteristic dimension, third layer convolution realize reconstruct；
Network is fought using generating, inputs multi-modal source images, is merged by network self-adapting feature extraction with adaptive, Generate preliminary blending image；
Preliminary blending image is inputted into super-resolution network, the final blending image of outputting high quality.
The above-mentioned multi-modality images fusion method based on generation confrontation network and super-resolution, generates confrontation net The associated losses function of network includes Pixel-level MSE loss, gradient loss and confrontation loss；Pixel-level MSE lossW, H respectively indicate the width and height of image in formula,It is to generate picture point (x, y) Pixel value,Indicate true picture point (x, y) pixel value；Gradient loss is expressed as Wherein,It indicates to generate the gradient that image (x, y) is put,Indicate the gradient of true picture (x, y) point；Confrontation lossIpredIndicate input data, N indicates image pixel number, finally applies to combined results L2 regularization is to prevent over-fitting and reduce spike pseudomorphism, i.e. associated losses function λ1、λ2、λ3For weight.
The above-mentioned multi-modality images fusion method based on generation confrontation network and super-resolution, λ1=0.001, λ2= 0.5, λ3=1, balance the weight of three loss functions, joint training neural network.
The above-mentioned multi-modality images fusion method based on generation confrontation network and super-resolution network, generates confrontation network Training set and test set include visible light (390-700nm), near-infrared (700-1000nm) and infrared long wave (8-12 μm) three Band image and CT, MRI_T2 and SPECT-Tc multi-modality medical image；The training set and test set of super-resolution network by The high-definition picture of same type forms.
The above-mentioned multi-modality images fusion method based on generation confrontation network and super-resolution, the activation primitive of convolutional layer Y=max (0, x) orWherein i indicates different channels, aiFor fixed negative semiaxis slope.
Confrontation network (generative adversarial network, GAN) is generated by it to the strong of distribution modeling Big function can encourage to generate image towards true picture flowing, to generate the true picture with likelihood high frequency detail. Min_max two-player game provides the simple and powerful method of one kind to estimate target distribution and generate new image pattern.GAN It is divided into two models of generator and arbiter.Generator indicates the mapping of input variable to data space with G (z), and differentiates Device with D (x) indicate from label rather than the probability of input data.Correct label is distributed in training D (x) maximization carrys out self-training The probability of sample and sample, while training G (z) is to minimize log (1-D (G (z))).The generation model powerful as one kind, The computer vision fields such as color, image repair, image super-resolution have successful application to GAN on the image.Various GAN variants It proposes in succession, GAN performance is mainly improved in terms of the network architecture and objective function two, and network inputs are also expanded to by random noise Single image, many GAN technologies generate close to true photo, or by grayscale image according to contour images and are converted to cromogram. Multiband/multi-modality images fusion can regard the process that input different images generate a width ideal image as, so, logically Say that GAN can be used for image co-registration, multiband/multi-modality images are inputted GAN by the present invention in the form of the multichannel of piece image, GAN network structure using present inventive concept do not bring parameter to increase with neural network depth down, calculation amount becomes larger even The depth residual error neural network of the problems such as network over-fitting, self-adaptive feature extraction and adaptive fusion rule by network, Output includes the initial fusion image of multiple image information, avoids and carries out the multi-direction decomposition of Image Multiscale according to priori knowledge With the difficulty of fusion rule design.Image Super Resolution Processing technology can break through the resolution ratio limitation of image capture device, sufficiently Using the complementary information between multiple image, the fusion of pixel-level image information is realized.The present invention is in order to solve to meet in e-learning The training set arrived is few and the low problem low so as to cause network output image resolution ratio of resolution ratio, and super-resolution is tied with GAN phase It closes, reference super-resolution network optimizes output image, the final blending image of outputting high quality.The present invention realizes end and arrives Terminal nerve network integration image avoids human intervention, and fusion results are cleaner and do not have artifact, provide better vision matter Amount.
Detailed description of the invention
Fig. 1 is generator structure chart.
Fig. 2 is arbiter structure chart.
Fig. 3 is super-resolution network structure.
Fig. 4 is visible images.
Fig. 5 is near-infrared image.
Fig. 6 is infrared long wave image.
Fig. 7 is blending image of the invention.
Specific implementation method
A kind of multi-modality images fusion method based on generation confrontation network and super-resolution network, comprising the following steps:
1. designing and constructing generation confrontation network structure
Generator network structure is the convolutional neural networks based on residual error, by three convolutional layers and seven residual block structures At wherein each residual block includes two layers of convolution；Arbiter is by six convolutional layers, three layers of residual unit block of a standard and one Full articulamentum is constituted, specific as follows:
(1) multiband source images or multi-modal medicine source images are inputted into generator, carries out a convolution operation, convolution kernel Size is 3 × 3 × 64, then inputs 7 residual blocks, and each residual block is made of two convolutional layers, and the target for needing to learn is F (x)=H (x)-x, in formula, x indicates network inputs, and H (x) indicates that desired output, F (x) indicate residual error mapping；Two layers volume is carried out again Product operation first extracts deeper time feature, exports 256 dimensional characteristics figures, then drop to 1 dimension, and one width source images size of output is melted Conjunction image, i.e., 128 × 128 × 1.
(2) for arbiter as a classifier, task is try one's best differentiation generation data and truthful data, will be given birth to It grows up to be a useful person the image of generation and true picture inputs arbiter respectively, 6 long convolution operations that stride first are carried out to input picture, are first mentioned More high-dimensional feature is taken, then carries out dimensionality reduction, exports 16 × 16 × 128 images；Then three layers of residual block list of a standard are inputted Member finally connects an one-dimensional full articulamentum, realizes two classification tasks, deeper between network structure, in addition to first layer convolution, Data normalization all is given using BatchNorm (BN) after every layer of convolution, after BN, uses LeakyReLU (LReLU) activation primitive Promoted network nonlinear degree, activation primitive select y=max (0, x) orWherein i is indicated not Same channel, aiFor fixed negative semiaxis slope.
2. designing and constructing super-resolution network
Super-resolution network of the invention is made of simple three-layer coil product, and first layer convolution extracts characteristic pattern, the second layer Convolution realizes that the Nonlinear Mapping of characteristic dimension, third layer convolution realize reconstruct.It is specific as follows:
(1) first layer characteristic pattern is expressed with following formula: F1=max (0, W1*X+B1), in formula, W1And B1Respectively indicate filter Wave device and biasing, X are input pictures, and * indicates convolution operation, 64 filters for the use of convolution kernel being 9 × 9, this layer operation Equal to being extracted 64 dimensional features of source images；
(2) second layer is expressed as, F2=max (0, W2*F1+B2), define W2For 32 64 × 1 × 1 filters, B2It indicates Biasing, therefore, this layer realizes the Nonlinear Mapping from 64 dimensional features to 32 dimensional features；
(3) reconstruction of layer is also defined as convolution operation, Y=W3*F2+B3, W3And B3Respectively indicate filter and biasing, convolution kernel Size is 5 × 5.
3. neural metwork training
It is as follows to generation confrontation network and each self-training of super-resolution network, specific training process:
(1) generator and arbiter interval training, i.e., first train a generator, arbiter of retraining, then according to Secondary circulation, until the two reaches dynamic equilibrium；Super-resolution network is individually trained；
(2) associated losses function is designed, backpropagation mode training neural network is passed through.Associated losses function includes pixel Grade MSE loss:W, H respectively indicate the width and height of image in formula,It is The pixel value of picture point (x, y) is generated,Indicate true picture point (x, y) pixel value；Gradient loss description image border letter Breath introduces this loss to optimize image border, indicates are as follows:Wherein,It indicates to generate the gradient that image (x, y) is put,Indicate the gradient of true picture (x, y) point；Confrontation loss:IpredIndicate input data, N indicates image pixel number.Finally to combined results Apply L2 regularization to prevent over-fitting and reduce spike pseudomorphism, i.e.,Weight Respectively λ1=0.001, λ2=0.5, λ3=1, balance the weight of three loss functions, joint training neural network.One loss Function may make the picture quality generated not so good, and associated losses function can preferably enrich details, optimize edge.
4. based on the multi-modality images fusion for generating confrontation network and super-resolution
(1) multi-modal source images are merged into multichannel to be loaded into neural network, certainly by convolution, dimensionality reduction etc. in network Adaptive method approaches generator with arbiter, by optimizing the associated losses function of generator and arbiter, increasingly generates it Likelihood data, training obtain generating model；
(2) multi-modal source images input to be fused is generated into model, obtains initial fusion image；
(3) initial fusion image is inputted into trained super-resolution network, obtains final high quality blending image.
The training set and test set for generating confrontation network include visible light (390-700nm), near-infrared (700-1000nm) With three wave band equal proportion mixed image of infrared long wave (8-12 μm) and CT, MRI_T2 and SPECT-Tc multi-modality medical image； The training set and test set of super-resolution network are made of the high-definition picture of same type.
Generator and arbiter have been all made of depth residual error neural network, and generator structure is by three-layer coil product and seven residual errors Block is constituted, and arbiter uses six layers of convolution, a residual unit block and a full articulamentum, and super-resolution network is using simple Three-layer coil accumulates network.Residual block is more, and transmitting image information is more, but excessive, and consuming time is long influences network efficiency and effect Increase slowly, generator residual block value range is between 7-9.It generates confrontation network and uses depth residual error neural network, avoid Neural network depth down and bring parameter to increase, calculation amount become larger even network over-fitting the problems such as；Super-resolution network Three layers of feature convolution operation have only been used, it is simple and high-efficient.
Batch size value is excessive to occupy more memory, too small consuming time between 28-44 when neural metwork training； Learning rate selects 0.002, and learning rate size determines network convergence rate, excessive to lead to network oscillation, restrains unstable, too small consumption Take the more time, influences network efficiency, therefore choose between 0.2-0.02.

Claims (5)

1. based on the multi-modality images fusion method for generating confrontation network and super-resolution network, it is characterised in that including following step It is rapid:
It designs and constructs generation confrontation network: generating confrontation network and be divided into two models of generator and arbiter, generator is by three A convolutional layer and seven residual blocks are constituted, wherein each residual block includes two layers of convolution；Modified line is carried out after each convolutional layer Unit non-linearization；Arbiter is connected and composed entirely by six convolutional layers, three layers of residual unit block of standard and one, each Convolutional layer all carries out the operation of BatchNorm data normalization other than input layer, is activated after normalization operation using LeakyReLU Function promotes network nonlinear degree；
Building super-resolution network: super-resolution network is made of three-layer coil product, and first layer convolution extracts characteristic pattern, second layer volume Product realizes that the Nonlinear Mapping of characteristic dimension, third layer convolution realize reconstruct；
Network is fought using generating, inputs multi-modal source images, by network self-adapting feature extraction and adaptive fusion, is generated Preliminary blending image；
Preliminary blending image is inputted into super-resolution network, the final blending image of outputting high quality.
2. it is according to claim 1 based on the multi-modality images fusion method for generating confrontation network and super-resolution, it is special Sign is that the associated losses function for generating confrontation network includes Pixel-level MSE loss, gradient loss and confrontation loss；Pixel-level MSE lossW, H respectively indicate the width and height of image in formula,It is raw At the pixel value of picture point (x, y),Indicate true picture point (x, y) pixel value；Gradient loss is expressed asWherein,It indicates to generate the gradient that image (x, y) is put,Table Show the gradient of true picture (x, y) point；Confrontation lossIpredIndicate input data, N It indicates image pixel number, L2 regularization finally is applied to prevent over-fitting and reduce spike pseudomorphism to combined results, that is, is combined Loss functionλ1、λ2、λ3For weight.
3. it is according to claim 2 based on the multi-modality images fusion method for generating confrontation network and super-resolution, it is special Sign is λ1=0.001, λ2=0.5, λ3=1, balance the weight of three loss functions, joint training neural network.
4. the multi-modality images fusion method according to claim 1 or 2 based on generation confrontation network and super-resolution, Be characterized in that generate confrontation network training collection and test set include visible light, near-infrared and three band image of infrared long wave and CT, MRI_T2 and SPECT-Tc multi-modality medical image, the training set and test set of super-resolution network by same type height Image in different resolution composition.
5. the multi-modality images fusion method according to claim 1 or 2 based on generation confrontation network and super-resolution, Be characterized in that convolutional layer activation primitive y=max (0, x) or
CN201810962126.5A 2018-08-22 2018-08-22 Based on the multi-modality images fusion method for generating confrontation network and super-resolution network Pending CN109325931A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810962126.5A CN109325931A (en) 2018-08-22 2018-08-22 Based on the multi-modality images fusion method for generating confrontation network and super-resolution network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810962126.5A CN109325931A (en) 2018-08-22 2018-08-22 Based on the multi-modality images fusion method for generating confrontation network and super-resolution network

Publications (1)

Publication Number Publication Date
CN109325931A true CN109325931A (en) 2019-02-12

Family

ID=65264260

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810962126.5A Pending CN109325931A (en) 2018-08-22 2018-08-22 Based on the multi-modality images fusion method for generating confrontation network and super-resolution network

Country Status (1)

Country Link
CN (1) CN109325931A (en)

Cited By (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109978785A (en) * 2019-03-22 2019-07-05 中南民族大学 The image super-resolution reconfiguration system and its method of multiple recurrence Fusion Features
CN109993702A (en) * 2019-04-10 2019-07-09 大连民族大学 Based on the language of the Manchus image super-resolution rebuilding method for generating confrontation network
CN110084751A (en) * 2019-04-24 2019-08-02 复旦大学 Image re-construction system and method
CN110120024A (en) * 2019-05-20 2019-08-13 百度在线网络技术（北京）有限公司 Method, apparatus, equipment and the storage medium of image procossing
CN110211046A (en) * 2019-06-03 2019-09-06 重庆邮电大学 A kind of remote sensing image fusion method, system and terminal based on generation confrontation network
CN110222794A (en) * 2019-06-21 2019-09-10 福州大学 The self-adaptive features fusion method of multi-modality images
CN110288609A (en) * 2019-05-30 2019-09-27 南京师范大学 A kind of multi-modal whole-heartedly dirty image partition method of attention mechanism guidance
CN110428370A (en) * 2019-07-01 2019-11-08 北京理工大学 A kind of method that utilization is eccentrically rotated raising pencil-beam SPECT imaging resolution
CN110503187A (en) * 2019-07-26 2019-11-26 江苏大学 A kind of implementation method of the generation confrontation network model generated for functional magnetic resonance imaging data
CN110544275A (en) * 2019-08-19 2019-12-06 中山大学 Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags
CN110580509A (en) * 2019-09-12 2019-12-17 杭州海睿博研科技有限公司 multimodal data processing system and method for generating countermeasure model based on hidden representation and depth
CN110610464A (en) * 2019-08-15 2019-12-24 天津中科智能识别产业技术研究院有限公司 Face image super-resolution method based on dense residual error neural network
CN110738099A (en) * 2019-08-30 2020-01-31 中山大学 low-resolution pedestrian re-identification method based on self-adaptive double-branch network
CN110796080A (en) * 2019-10-29 2020-02-14 重庆大学 Multi-pose pedestrian image synthesis algorithm based on generation of countermeasure network
CN111046900A (en) * 2019-10-25 2020-04-21 重庆邮电大学 Semi-supervised generation confrontation network image classification method based on local manifold regularization
CN111080528A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution and model training method, device, electronic equipment and medium
CN111080527A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution method and device, electronic equipment and storage medium
CN111260652A (en) * 2020-01-09 2020-06-09 浙江传媒学院 Image generation system and method based on MIMO-GAN
CN111260594A (en) * 2019-12-22 2020-06-09 天津大学 Unsupervised multi-modal image fusion method
CN111401292A (en) * 2020-03-25 2020-07-10 成都东方天呈智能科技有限公司 Face recognition network construction method fusing infrared image training
CN111696066A (en) * 2020-06-13 2020-09-22 中北大学 Multi-band image synchronous fusion and enhancement method based on improved WGAN-GP
CN111696168A (en) * 2020-06-13 2020-09-22 中北大学 High-speed MRI reconstruction method based on residual self-attention image enhancement
CN111709903A (en) * 2020-05-26 2020-09-25 中国科学院长春光学精密机械与物理研究所 Infrared and visible light image fusion method
CN111754446A (en) * 2020-06-22 2020-10-09 怀光智能科技（武汉）有限公司 Image fusion method, system and storage medium based on generation countermeasure network
CN111861949A (en) * 2020-04-21 2020-10-30 北京联合大学 Multi-exposure image fusion method and system based on generation countermeasure network
CN111915545A (en) * 2020-08-06 2020-11-10 中北大学 Self-supervision learning fusion method of multiband images
CN111951199A (en) * 2019-05-16 2020-11-17 武汉Tcl集团工业研究院有限公司 Image fusion method and device
CN112085687A (en) * 2020-09-10 2020-12-15 浙江大学 Method for converting T1 to STIR image based on detail enhancement
CN112116527A (en) * 2020-09-09 2020-12-22 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network framework and cascade network
CN112215788A (en) * 2020-09-15 2021-01-12 湖北工业大学 Multi-focus image fusion algorithm based on improved generation countermeasure network
CN112446828A (en) * 2021-01-29 2021-03-05 成都东方天呈智能科技有限公司 Thermal imaging super-resolution reconstruction method fusing visible image gradient information
CN112508775A (en) * 2020-12-10 2021-03-16 深圳先进技术研究院 MRI-PET image mode conversion method and system based on loop generation countermeasure network
CN113012045A (en) * 2021-02-23 2021-06-22 西南交通大学 Generation countermeasure network for synthesizing medical image
CN113012086A (en) * 2021-03-22 2021-06-22 上海应用技术大学 Cross-modal image synthesis method
CN113096169A (en) * 2021-03-31 2021-07-09 华中科技大学 Non-rigid multimode medical image registration model establishing method and application thereof
CN113112441A (en) * 2021-04-30 2021-07-13 中北大学 Multi-band low-resolution image synchronous fusion method based on dense network and local brightness traversal operator
CN113256500A (en) * 2021-07-02 2021-08-13 北京大学第三医院（北京大学第三临床医学院） Deep learning neural network model system for multi-modal image synthesis
CN113506222A (en) * 2021-07-30 2021-10-15 合肥工业大学 Multi-mode image super-resolution method based on convolutional neural network
CN113627504A (en) * 2021-08-02 2021-11-09 南京邮电大学 Multi-mode multi-scale feature fusion target detection method based on generation of countermeasure network
CN113723470A (en) * 2021-08-09 2021-11-30 北京工业大学 Pollen image synthesis method and device fusing multilayer information and electronic equipment
CN114782590A (en) * 2022-03-17 2022-07-22 山东大学 Multi-object content joint image generation method and system
CN114881864A (en) * 2021-10-12 2022-08-09 北京九章云极科技有限公司 Training method and device for seal restoration network model
CN115272261A (en) * 2022-08-05 2022-11-01 广州大学 Multi-modal medical image fusion method based on deep learning

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023605A (en) * 2016-07-15 2016-10-12 姹ゅ钩 Traffic signal lamp control method based on deep convolution neural network
CN107464216A (en) * 2017-08-03 2017-12-12 济南大学 A kind of medical image ultra-resolution ratio reconstructing method based on multilayer convolutional neural networks
CN107563493A (en) * 2017-07-17 2018-01-09 华南理工大学 A kind of confrontation network algorithm of more maker convolution composographs
CN107590774A (en) * 2017-09-18 2018-01-16 北京邮电大学 A kind of car plate clarification method and device based on generation confrontation network
CN107968962A (en) * 2017-12-12 2018-04-27 华中科技大学 A kind of video generation method of the non-conterminous image of two frames based on deep learning
CN108319932A (en) * 2018-03-12 2018-07-24 中山大学 A kind of method and device for the more image faces alignment fighting network based on production

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023605A (en) * 2016-07-15 2016-10-12 姹ゅ钩 Traffic signal lamp control method based on deep convolution neural network
CN107563493A (en) * 2017-07-17 2018-01-09 华南理工大学 A kind of confrontation network algorithm of more maker convolution composographs
CN107464216A (en) * 2017-08-03 2017-12-12 济南大学 A kind of medical image ultra-resolution ratio reconstructing method based on multilayer convolutional neural networks
CN107590774A (en) * 2017-09-18 2018-01-16 北京邮电大学 A kind of car plate clarification method and device based on generation confrontation network
CN107968962A (en) * 2017-12-12 2018-04-27 华中科技大学 A kind of video generation method of the non-conterminous image of two frames based on deep learning
CN108319932A (en) * 2018-03-12 2018-07-24 中山大学 A kind of method and device for the more image faces alignment fighting network based on production

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
N. MERKLE, AT EL.: ""On the possibility of conditional adversarial networks for multi-sensor image matching"", 《2017 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS)》 *
XIANGYU LIU, AT EL.: ""PSGAN: A GENERATIVE ADVERSARIAL NETWORK FOR REMOTE SENSING IMAGE PAN-SHARPENING"", 《ARXIV》 *
杨卫华等: "《眼科人工智能》", 28 February 2018, 湖北科学技术出版社 *

Cited By (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109978785A (en) * 2019-03-22 2019-07-05 中南民族大学 The image super-resolution reconfiguration system and its method of multiple recurrence Fusion Features
CN109978785B (en) * 2019-03-22 2020-11-13 中南民族大学 Image super-resolution reconstruction system and method based on multi-level recursive feature fusion
CN109993702A (en) * 2019-04-10 2019-07-09 大连民族大学 Based on the language of the Manchus image super-resolution rebuilding method for generating confrontation network
CN109993702B (en) * 2019-04-10 2023-09-26 大连民族大学 Full-text image super-resolution reconstruction method based on generation countermeasure network
CN110084751A (en) * 2019-04-24 2019-08-02 复旦大学 Image re-construction system and method
CN111951199A (en) * 2019-05-16 2020-11-17 武汉Tcl集团工业研究院有限公司 Image fusion method and device
CN110120024A (en) * 2019-05-20 2019-08-13 百度在线网络技术（北京）有限公司 Method, apparatus, equipment and the storage medium of image procossing
CN110120024B (en) * 2019-05-20 2021-08-17 百度在线网络技术（北京）有限公司 Image processing method, device, equipment and storage medium
US11645735B2 (en) 2019-05-20 2023-05-09 Baidu Online Network Technology (Beijing) Co., Ltd. Method and apparatus for processing image, device and computer readable storage medium
CN110288609A (en) * 2019-05-30 2019-09-27 南京师范大学 A kind of multi-modal whole-heartedly dirty image partition method of attention mechanism guidance
CN110211046B (en) * 2019-06-03 2023-07-14 重庆邮电大学 Remote sensing image fusion method, system and terminal based on generation countermeasure network
CN110211046A (en) * 2019-06-03 2019-09-06 重庆邮电大学 A kind of remote sensing image fusion method, system and terminal based on generation confrontation network
CN110222794A (en) * 2019-06-21 2019-09-10 福州大学 The self-adaptive features fusion method of multi-modality images
CN110222794B (en) * 2019-06-21 2023-02-07 福州大学 Self-adaptive feature fusion method of multi-modal image
CN110428370B (en) * 2019-07-01 2021-11-23 北京理工大学 Method for improving conical beam SPECT imaging resolution by utilizing eccentric rotation
CN110428370A (en) * 2019-07-01 2019-11-08 北京理工大学 A kind of method that utilization is eccentrically rotated raising pencil-beam SPECT imaging resolution
CN110503187A (en) * 2019-07-26 2019-11-26 江苏大学 A kind of implementation method of the generation confrontation network model generated for functional magnetic resonance imaging data
CN110610464A (en) * 2019-08-15 2019-12-24 天津中科智能识别产业技术研究院有限公司 Face image super-resolution method based on dense residual error neural network
CN110544275A (en) * 2019-08-19 2019-12-06 中山大学 Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags
CN110544275B (en) * 2019-08-19 2022-04-26 中山大学 Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags
CN110738099B (en) * 2019-08-30 2022-06-07 中山大学 Low-resolution pedestrian re-identification method based on self-adaptive double-branch network
CN110738099A (en) * 2019-08-30 2020-01-31 中山大学 low-resolution pedestrian re-identification method based on self-adaptive double-branch network
CN110580509A (en) * 2019-09-12 2019-12-17 杭州海睿博研科技有限公司 multimodal data processing system and method for generating countermeasure model based on hidden representation and depth
CN111046900A (en) * 2019-10-25 2020-04-21 重庆邮电大学 Semi-supervised generation confrontation network image classification method based on local manifold regularization
CN111046900B (en) * 2019-10-25 2022-10-18 重庆邮电大学 Semi-supervised generation confrontation network image classification method based on local manifold regularization
CN110796080A (en) * 2019-10-29 2020-02-14 重庆大学 Multi-pose pedestrian image synthesis algorithm based on generation of countermeasure network
CN110796080B (en) * 2019-10-29 2023-06-16 重庆大学 Multi-pose pedestrian image synthesis algorithm based on generation countermeasure network
CN111080527B (en) * 2019-12-20 2023-12-05 北京金山云网络技术有限公司 Image super-resolution method and device, electronic equipment and storage medium
CN111080528A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution and model training method, device, electronic equipment and medium
CN111080527A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution method and device, electronic equipment and storage medium
CN111080528B (en) * 2019-12-20 2023-11-07 北京金山云网络技术有限公司 Image super-resolution and model training method and device, electronic equipment and medium
CN111260594A (en) * 2019-12-22 2020-06-09 天津大学 Unsupervised multi-modal image fusion method
CN111260594B (en) * 2019-12-22 2023-10-31 天津大学 Unsupervised multi-mode image fusion method
CN111260652A (en) * 2020-01-09 2020-06-09 浙江传媒学院 Image generation system and method based on MIMO-GAN
CN111260652B (en) * 2020-01-09 2023-09-08 浙江传媒学院 MIMO-GAN-based image generation system and method
CN111401292A (en) * 2020-03-25 2020-07-10 成都东方天呈智能科技有限公司 Face recognition network construction method fusing infrared image training
CN111861949B (en) * 2020-04-21 2023-07-04 北京联合大学 Multi-exposure image fusion method and system based on generation countermeasure network
CN111861949A (en) * 2020-04-21 2020-10-30 北京联合大学 Multi-exposure image fusion method and system based on generation countermeasure network
CN111709903A (en) * 2020-05-26 2020-09-25 中国科学院长春光学精密机械与物理研究所 Infrared and visible light image fusion method
CN111696066A (en) * 2020-06-13 2020-09-22 中北大学 Multi-band image synchronous fusion and enhancement method based on improved WGAN-GP
CN111696168A (en) * 2020-06-13 2020-09-22 中北大学 High-speed MRI reconstruction method based on residual self-attention image enhancement
CN111696066B (en) * 2020-06-13 2022-04-19 中北大学 Multi-band image synchronous fusion and enhancement method based on improved WGAN-GP
CN111696168B (en) * 2020-06-13 2022-08-23 中北大学 High-speed MRI reconstruction method based on residual self-attention image enhancement
CN111754446A (en) * 2020-06-22 2020-10-09 怀光智能科技（武汉）有限公司 Image fusion method, system and storage medium based on generation countermeasure network
CN111915545B (en) * 2020-08-06 2022-07-05 中北大学 Self-supervision learning fusion method of multiband images
CN111915545A (en) * 2020-08-06 2020-11-10 中北大学 Self-supervision learning fusion method of multiband images
CN112116527B (en) * 2020-09-09 2024-02-23 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network frame and cascade network
CN112116527A (en) * 2020-09-09 2020-12-22 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network framework and cascade network
CN112085687A (en) * 2020-09-10 2020-12-15 浙江大学 Method for converting T1 to STIR image based on detail enhancement
CN112085687B (en) * 2020-09-10 2023-12-01 浙江大学 Method for converting T1 to STIR image based on detail enhancement
CN112215788A (en) * 2020-09-15 2021-01-12 湖北工业大学 Multi-focus image fusion algorithm based on improved generation countermeasure network
CN112508775A (en) * 2020-12-10 2021-03-16 深圳先进技术研究院 MRI-PET image mode conversion method and system based on loop generation countermeasure network
CN112446828B (en) * 2021-01-29 2021-04-13 成都东方天呈智能科技有限公司 Thermal imaging super-resolution reconstruction method fusing visible image gradient information
CN112446828A (en) * 2021-01-29 2021-03-05 成都东方天呈智能科技有限公司 Thermal imaging super-resolution reconstruction method fusing visible image gradient information
CN113012045A (en) * 2021-02-23 2021-06-22 西南交通大学 Generation countermeasure network for synthesizing medical image
CN113012045B (en) * 2021-02-23 2022-07-15 西南交通大学 Generation countermeasure network for synthesizing medical image
CN113012086B (en) * 2021-03-22 2024-04-16 上海应用技术大学 Cross-modal image synthesis method
CN113012086A (en) * 2021-03-22 2021-06-22 上海应用技术大学 Cross-modal image synthesis method
CN113096169B (en) * 2021-03-31 2022-05-20 华中科技大学 Non-rigid multimode medical image registration model establishing method and application thereof
CN113096169A (en) * 2021-03-31 2021-07-09 华中科技大学 Non-rigid multimode medical image registration model establishing method and application thereof
CN113112441B (en) * 2021-04-30 2022-04-26 中北大学 Multi-band low-resolution image synchronous fusion method based on dense network and local brightness traversal operator
CN113112441A (en) * 2021-04-30 2021-07-13 中北大学 Multi-band low-resolution image synchronous fusion method based on dense network and local brightness traversal operator
CN113256500A (en) * 2021-07-02 2021-08-13 北京大学第三医院（北京大学第三临床医学院） Deep learning neural network model system for multi-modal image synthesis
CN113506222B (en) * 2021-07-30 2024-03-01 合肥工业大学 Multi-mode image super-resolution method based on convolutional neural network
CN113506222A (en) * 2021-07-30 2021-10-15 合肥工业大学 Multi-mode image super-resolution method based on convolutional neural network
CN113627504B (en) * 2021-08-02 2022-06-14 南京邮电大学 Multi-mode multi-scale feature fusion target detection method based on generation of countermeasure network
CN113627504A (en) * 2021-08-02 2021-11-09 南京邮电大学 Multi-mode multi-scale feature fusion target detection method based on generation of countermeasure network
CN113723470A (en) * 2021-08-09 2021-11-30 北京工业大学 Pollen image synthesis method and device fusing multilayer information and electronic equipment
CN114881864A (en) * 2021-10-12 2022-08-09 北京九章云极科技有限公司 Training method and device for seal restoration network model
CN114782590A (en) * 2022-03-17 2022-07-22 山东大学 Multi-object content joint image generation method and system
CN115272261A (en) * 2022-08-05 2022-11-01 广州大学 Multi-modal medical image fusion method based on deep learning

Similar Documents

Publication Publication Date Title
CN109325931A (en) Based on the multi-modality images fusion method for generating confrontation network and super-resolution network
CN109196526B (en) Method and system for generating multi-modal digital images
CN107577985B (en) The implementation method of the face head portrait cartooning of confrontation network is generated based on circulation
Vu et al. Illumination-robust face recognition using retina modeling
Hong et al. End-to-end unpaired image denoising with conditional adversarial networks
DE102020125197A1 (en) FINE GRAIN OBJECT SEGMENTATION IN VIDEO WITH DEEP FEATURES AND GRAPHICAL MULTI-LEVEL MODELS
WO2020015330A1 (en) Enhanced neural network-based image restoration method, storage medium, and system
CN110322416A (en) Image processing method, device and computer readable storage medium
CN112967178B (en) Image conversion method, device, equipment and storage medium
CN103020933B (en) A kind of multisource image anastomosing method based on bionic visual mechanism
CN111292262B (en) Image processing method, device, electronic equipment and storage medium
CN109064437A (en) Image fusion method based on guided filtering and online dictionary learning
CN111047543A (en) Image enhancement method, device and storage medium
CN110349085A (en) A kind of single image super-resolution feature Enhancement Method based on generation confrontation network
WO2023000895A1 (en) Image style conversion method and apparatus, electronic device and storage medium
CN113516601A (en) Image restoration technology based on deep convolutional neural network and compressed sensing
CN108734677A (en) A kind of blind deblurring method and system based on deep learning
CN116188912A (en) Training method, device, medium and equipment for image synthesis model of theme image
Wang et al. JPEG artifacts removal via compression quality ranker-guided networks
CN113298744B (en) End-to-end infrared and visible light image fusion method
Zhu et al. Multiscale channel attention network for infrared and visible image fusion
El-Sayed et al. Efficient fusion of medical images based on CNN
Wang et al. Qsfm: Model pruning based on quantified similarity between feature maps for ai on edge
Gao et al. Learning to Incorporate Texture Saliency Adaptive Attention to Image Cartoonization.
Rao et al. UMFA: a photorealistic style transfer method based on U-Net and multi-layer feature aggregation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20190212

RJ01 Rejection of invention patent application after publication