CN114419328B - Image fusion method and system for generating countermeasure network based on self-adaptive enhancement - Google Patents

Image fusion method and system for generating countermeasure network based on self-adaptive enhancement Download PDF

Info

Publication number
CN114419328B
CN114419328B CN202210071844.XA CN202210071844A CN114419328B CN 114419328 B CN114419328 B CN 114419328B CN 202210071844 A CN202210071844 A CN 202210071844A CN 114419328 B CN114419328 B CN 114419328B
Authority
CN
China
Prior art keywords
image
fusion
source
adaptive
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210071844.XA
Other languages
Chinese (zh)
Other versions
CN114419328A (en
Inventor
张聪炫
单长鲁
陈震
卢锋
葛利跃
陈昊
秦文健
李凌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanchang Hangkong University
Original Assignee
Nanchang Hangkong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanchang Hangkong University filed Critical Nanchang Hangkong University
Priority to CN202210071844.XA priority Critical patent/CN114419328B/en
Publication of CN114419328A publication Critical patent/CN114419328A/en
Application granted granted Critical
Publication of CN114419328B publication Critical patent/CN114419328B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G06T5/90—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to an image fusion method and system for generating an countermeasure network based on self-adaptive enhancement. The method comprises the following steps: constructing a dense detail feature extraction network, and respectively carrying out feature extraction on the source image by combining a dense convolution mode with a detail information compensation mechanism; constructing a double-channel self-adaptive fusion network fusion source characteristic diagram and a detail information characteristic diagram; after the feature graphs are spliced, a 1*1 convolution network is constructed to realize cross-channel interaction and information fusion; decoding the fused feature images to obtain fused images; and adding an adaptive structural similarity loss function when the whole network model is trained. According to the method, a detail information compensation mechanism is introduced to enhance details of the fusion image and reduce information loss, a dual-channel adaptive fusion network is used for balancing infrared information and visible light information in the fusion image in a channel dimension, an adaptive structure similarity loss function is added to adaptively enhance similarity between the fusion image and a source image in a space dimension, and fusion image quality is improved.

Description

Image fusion method and system for generating countermeasure network based on self-adaptive enhancement
Technical Field
The invention relates to the technical field of image fusion, in particular to an image fusion method and system for generating an countermeasure network based on self-adaption enhancement.
Background
Image fusion is an important technique in image processing, the main goal of which is to integrate salient features extracted from each source image, as a single infrared or visible light sensor cannot capture complete scene information. In general, the visible image has abundant perceived details with high spatial resolution, and the infrared image contains thermal radiation of the target, so that it is of great importance to fuse complementary information of the infrared image and the visible image into a new composite image to process different tasks. The most advanced fusion algorithms are currently widely used in many applications such as automatic driving automobiles, visual tracking and video surveillance. Fusion algorithms can be broadly divided into two categories: conventional methods and methods based on deep learning. In recent years, deep learning-based methods have shown tremendous potential in image fusion tasks and are believed to have the potential to provide better performance than traditional algorithms.
At present, a great obstacle to image fusion based on a deep learning method is that no truth value exists, so that only unsupervised training or supervised training with two source images as truth values can be adopted for an end-to-end network during training. However, when using unsupervised training, the end-to-end network loses much information when extracting features; when two truth images are adopted for supervised training, the problem of unbalanced information distribution can be generated in the fused images.
Disclosure of Invention
The invention aims to provide an image fusion method and system based on self-adaptive enhancement generation countermeasure network, which are used for solving the problems of information loss and imbalance of effective information distribution in the existing image fusion method based on deep learning.
In order to achieve the above object, the present invention provides the following solutions:
an image fusion method for generating an countermeasure network based on adaptive enhancement, comprising:
acquiring a source image; the source image comprises an infrared image and a visible light image;
combining the dense convolution network with a detail information compensation mechanism to construct a dense detail feature extraction network;
performing feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; the source characteristic map of the source image comprises a source characteristic map of an infrared image and a source characteristic map of a visible light image; the detail information feature map comprises a detail information feature map of an infrared image and a detail information feature map of a visible light image;
constructing a two-channel self-adaptive fusion network based on the two-channel maximum pooling self-adaptive fusion mechanism and the two-channel average pooling self-adaptive fusion mechanism;
Performing two-channel maximum pooling self-adaptive fusion on the source feature images of the source images based on the two-channel self-adaptive fusion network to obtain a fusion source feature image;
performing double-channel average pooling self-adaptive fusion on the detail information feature images of the source images based on the double-channel self-adaptive fusion network to obtain fused detail information feature images;
the two-channel self-adaptive fusion network is adopted to splice the fusion source characteristic diagram and the fusion detail information characteristic diagram, and a spliced characteristic diagram is obtained;
inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels, so as to obtain fused feature images;
decoding the fused feature images by adopting a decoding network to obtain fused images;
sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network;
constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fusion image and the source image;
training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function, and generating a trained self-adaptive enhancement generation countermeasure network;
And adopting the trained self-adaptive enhancement generation countermeasure network to perform image fusion of the infrared image and the visible light image.
Optionally, the feature extraction is performed on the source image based on the dense detail feature extraction network to obtain a source feature map and a detail information feature map of the source image, which specifically includes:
based on the dense detail feature extraction network, adopting a dense detail feature extraction network formula p i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i An i-th layer feature map representing the dense detail feature extraction network; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; broadcast (x) -p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
Optionally, the performing dual-channel maximum pooling adaptive fusion on the source feature map of the source image based on the dual-channel adaptive fusion network to obtain a fused source feature map specifically includes:
Based on the two-channel self-adaptive fusion network, a formula is adopted
Figure BDA0003482350870000031
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure BDA0003482350870000032
Source signature representing infrared image, +.>
Figure BDA0003482350870000033
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure BDA0003482350870000034
representing a convolution operation on a source signature of an infrared image,/->
Figure BDA0003482350870000035
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; and X DEG represents a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
Optionally, the performing dual-channel average pooling adaptive fusion on the detail information feature map of the source image based on the dual-channel adaptive fusion network to obtain a fused detail information feature map specifically includes:
based on the two-channel self-adaptive fusion network, a formula is adopted
Figure BDA0003482350870000036
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure BDA0003482350870000041
Detail information feature map representing infrared image, < >>
Figure BDA0003482350870000042
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension; / >
Figure BDA0003482350870000043
The characteristic diagram of the detail information representing the infrared image is convolved,
Figure BDA0003482350870000044
performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; x is X d And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
Optionally, the constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fused image and the source image specifically includes:
using the formula
Figure BDA0003482350870000045
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
using the formula
Figure BDA0003482350870000046
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number;
using the formula
Figure BDA0003482350870000047
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
Calculating the structural similarity ssim (x, y) of the fusion image and the source image according to the brightness similarity l (x, y), the contrast similarity c (x, y) and the structural similarity s (x, y) of the fusion image and the source image by adopting a formula ssim (x, y) =l (x, y) ·c (x, y) ·s (x, y);
constructing an adaptive structural similarity loss function according to the structural similarity ssim (x, y) of the fusion image and the source image
Figure BDA0003482350870000048
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing an infrared image block in a sliding window, f w Representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; />
Figure BDA0003482350870000051
Pixel mean value representing visible light image block in sliding window,/>
Figure BDA0003482350870000052
Representing a pixel average value of the infrared image block in the sliding window; SSIM represents the value of the final structural similarity loss function in a sliding window.
An image fusion system for generating an countermeasure network based on adaptive enhancement, comprising:
the source image acquisition module is used for acquiring a source image; the source image comprises an infrared image and a visible light image;
The dense detail feature extraction network construction module is used for constructing a dense detail feature extraction network by combining the dense convolution network with a detail information compensation mechanism;
the feature extraction module is used for carrying out feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; the source characteristic map of the source image comprises a source characteristic map of an infrared image and a source characteristic map of a visible light image; the detail information feature map comprises a detail information feature map of an infrared image and a detail information feature map of a visible light image;
the double-channel self-adaptive fusion network construction module is used for constructing a double-channel self-adaptive fusion network based on a double-channel maximum pooling self-adaptive fusion mechanism and a double-channel average pooling self-adaptive fusion mechanism;
the source characteristic diagram fusion module is used for carrying out double-channel maximum pooling self-adaptive fusion on the source characteristic diagram of the source image based on the double-channel self-adaptive fusion network to obtain a fusion source characteristic diagram;
the detail information feature map fusion module is used for carrying out double-channel average pooling self-adaptive fusion on the detail information feature map of the source image based on the double-channel self-adaptive fusion network to obtain a fused detail information feature map;
The fusion feature map splicing module is used for splicing the fusion source feature map and the fusion detail information feature map by adopting the double-channel self-adaptive fusion network to obtain a spliced feature map;
the convolution network fusion module is used for inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels so as to obtain fused feature images;
the feature map code module is used for decoding the fused feature map by adopting a decoding network to obtain a fused image;
the self-adaptive enhancement generation countermeasure network construction module is used for sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network;
the self-adaptive structure similarity loss function construction module is used for constructing a self-adaptive structure similarity loss function according to the brightness similarity, the contrast similarity and the structure similarity of the fusion image and the source image;
the self-adaptive enhancement generation countermeasure network training module is used for training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function to generate a trained self-adaptive enhancement generation countermeasure network;
And the image fusion module is used for carrying out image fusion on the infrared image and the visible light image by adopting the trained self-adaptive enhancement generation countermeasure network.
Optionally, the feature extraction module specifically includes:
a feature extraction unit for extracting a network formula p by using dense detail features based on the dense detail feature extraction network i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i An i-th layer feature map representing the dense detail feature extraction network; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; broadcast (x) -p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
Optionally, the source feature map fusion module specifically includes:
the source characteristic diagram fusion unit is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure BDA0003482350870000061
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure BDA0003482350870000062
Source signature representing infrared image, +.>
Figure BDA0003482350870000063
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure BDA0003482350870000071
representing a convolution operation on a source signature of an infrared image,/->
Figure BDA0003482350870000072
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; and X DEG represents a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
Optionally, the detailed information feature map fusion module specifically includes:
the detail information feature map fusion unit is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure BDA0003482350870000073
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure BDA0003482350870000074
Detail information feature map representing infrared image, < >>
Figure BDA0003482350870000075
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension;/>
Figure BDA0003482350870000076
the characteristic diagram of the detail information representing the infrared image is convolved,
Figure BDA0003482350870000077
Performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; x is X d And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
Optionally, the adaptive structural similarity loss function building module specifically includes:
a brightness similarity calculation unit for using the formula
Figure BDA0003482350870000078
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
a contrast similarity calculation unit for using the formula
Figure BDA0003482350870000079
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number;
a structural similarity calculation unit for adopting the formula
Figure BDA00034823508700000710
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
a structural similarity calculating unit, configured to calculate structural similarity ssim (x, y) between the fused image and the source image according to brightness similarity l (x, y), contrast similarity c (x, y), and structural similarity s (x, y) of the fused image and the source image by using a formula ssim (x, y) =l (x, y) =c (x, y) ·s (x, y);
An adaptive structural similarity loss function construction unit for constructing an adaptive structural similarity loss function based on structural similarity ssim (x, y) of the fusion image and the source image
Figure BDA0003482350870000081
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing an infrared image block in a sliding window, f w Representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; />
Figure BDA0003482350870000082
Pixel mean value representing visible light image block in sliding window,/>
Figure BDA0003482350870000083
Representing a pixel average value of the infrared image block in the sliding window; SSIM represents the value of the final structural similarity loss function in a sliding window.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention provides an image fusion method and system for generating an countermeasure network based on self-adaptive enhancement, wherein the method comprises the following steps: acquiring a source image; the source image comprises an infrared image and a visible light image; combining the dense convolution network with a detail information compensation mechanism to construct a dense detail feature extraction network; performing feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; constructing a two-channel self-adaptive fusion network based on the two-channel maximum pooling self-adaptive fusion mechanism and the two-channel average pooling self-adaptive fusion mechanism; performing two-channel maximum pooling self-adaptive fusion on the source feature images of the source images based on the two-channel self-adaptive fusion network to obtain a fusion source feature image; performing double-channel average pooling self-adaptive fusion on the detail information feature images of the source images based on the double-channel self-adaptive fusion network to obtain fused detail information feature images; the two-channel self-adaptive fusion network is adopted to splice the fusion source characteristic diagram and the fusion detail information characteristic diagram, and a spliced characteristic diagram is obtained; inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels, so as to obtain fused feature images; decoding the fused feature images by adopting a decoding network to obtain fused images; sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network; constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fusion image and the source image; training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function, and generating a trained self-adaptive enhancement generation countermeasure network; and adopting the trained self-adaptive enhancement generation countermeasure network to perform image fusion of the infrared image and the visible light image. According to the method, a detail information compensation mechanism is introduced to enhance details of the fusion image and reduce information loss, a dual-channel adaptive fusion network is used for balancing infrared information and visible light information in the fusion image in a channel dimension, an adaptive structure similarity loss function is added to adaptively enhance brightness similarity, contrast similarity and structure similarity of the fusion image and two source images in a space dimension, the problems of information loss and imbalance of effective information distribution in the existing image fusion method based on deep learning are solved, and the quality of the fusion image is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an image fusion method for generating an countermeasure network based on adaptive enhancement according to the present invention;
FIG. 2 is a schematic diagram of an infrared image in a public dataset according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a visible light image in a public dataset according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a dense detail feature extraction network according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a dual-channel adaptive fusion network according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an infrared and visible light image fusion result provided by an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide an image fusion method and system based on self-adaptive enhancement generation countermeasure network, which are used for solving the problems of information loss and imbalance of effective information distribution in the existing image fusion method based on deep learning.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
FIG. 1 is a flow chart of an image fusion method for generating an countermeasure network based on adaptive enhancement according to the present invention. Referring to fig. 1, an image fusion method for generating an countermeasure network based on adaptive enhancement according to the present invention includes:
step 101: a source image is acquired.
FIG. 2 is a schematic diagram of an infrared image in a public dataset according to an embodiment of the present invention; fig. 3 is a schematic diagram of a visible light image in a public data set according to an embodiment of the present invention. Referring to fig. 2 and 3, the adaptive enhancement of the present invention generates source images including infrared images and visible images against network inputs.
Step 102: and combining the dense convolution network with a detail information compensation mechanism to construct a dense detail feature extraction network.
Fig. 4 is a schematic structural diagram of a dense detail feature extraction network according to an embodiment of the present invention. Referring to fig. 4, the present invention combines a dense convolution network with a detailed information compensation mechanism to define a new feature extraction network: dense detail feature extraction network. The dense convolution network extracts deep information and shallow information through jump connection, and the detail information compensation mechanism acquires additional detail compensation information corresponding to the deep and shallow information. The dense detail feature extraction network formula constructed by the invention is as follows:
p i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i )))))) (1)
y i ＝cat(p i ,broadcast(x)-p i ) (2)
In the formulas (1) and (2), x represents an infrared image or a visible light image, y i And (3) representing all feature graphs of an ith layer of the dense detail feature extraction network, wherein i is more than 0 and less than or equal to n, n is the number of layers of the dense detail feature extraction network, and n is more than 2.P is p i Represents the extracted i-th layer source characteristic diagram, broadcast (x) -p i An i-th layer detail information feature map (also called detail compensation feature map) corresponding to the source feature map is represented, x i An i-th layer feature map representing a dense detail feature extraction network. cat indicates that the feature graphs are spliced on the feature channels, and broadcast indicates that a broadcasting mechanism can automatically performExpansion dimension conv i And (5) representing the ith layer convolution operation of feature extraction after feature graph splicing.
Step 103: and extracting the features of the source image based on the dense detail feature extraction network to obtain a source feature map and a detail information feature map of the source image.
The constructed dense detail feature extraction network is used as a coding network of the self-adaptive enhancement generation countermeasure network, and features extraction is carried out on two input source images (an infrared image and a visible light image) by combining a detail information compensation mechanism in a dense convolution mode, so that a source feature image and a detail information feature image of the two source images are obtained. The source characteristic images of the source images comprise two groups of source characteristic images, namely a source characteristic image of an infrared image and a source characteristic image of a visible light image; the detail information feature map comprises two groups of detail information feature maps, namely, the detail information feature map of the infrared image and the detail information feature map of the visible light image.
The step 103 specifically includes:
based on the dense detail feature extraction network, adopting a dense detail feature extraction network formula p i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i An i-th layer feature map representing the dense detail feature extraction network; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; broadcast (x) -p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
Step 104: and constructing a two-channel self-adaptive fusion network based on the two-channel maximum pooling self-adaptive fusion mechanism and the two-channel average pooling self-adaptive fusion mechanism.
Fig. 5 is a schematic structural diagram of a dual-channel adaptive fusion network according to an embodiment of the present invention. Referring to fig. 5, the present invention constructs the two-channel adaptive fusion network based on a two-channel maximum pooling adaptive fusion mechanism and a two-channel average pooling adaptive fusion mechanism. Step 103, after two groups of source feature images and two other groups of detail information feature images are obtained, a two-channel maximum pooling self-adaptive fusion mechanism is constructed to fuse the two groups of source feature images, and a group of fused source feature images is obtained; and constructing a double-channel average pooling self-adaptive fusion mechanism to fuse the two sets of detail information feature images to obtain a set of fused detail information feature images.
Step 105: and carrying out double-channel maximum pooling self-adaptive fusion on the source characteristic images of the source images based on the double-channel self-adaptive fusion network to obtain a fusion source characteristic image.
Acquiring four groups of feature images extracted by a dense detail feature extraction network, carrying out double-channel maximum pooling self-adaptive fusion on the two groups of obtained source feature images, and uniformly distributing pixel intensity information of the two groups of source feature images in the fused feature images in a channel dimension; carrying out double-channel average pooling self-adaptive fusion on the two groups of detail compensation feature images, and uniformly distributing detail texture information of the two groups of detail compensation feature images in the fused feature images in the channel dimension; and finally, adaptively enhancing the effective information of the two source images in the fusion image. The calculation formula of the two-channel self-adaptive fusion network fusion source characteristic diagram is as follows:
Figure BDA0003482350870000121
in the formula (3), the amino acid sequence of the compound,
Figure BDA0003482350870000122
source feature maps respectively representing an input infrared image and a visible light image, max representing a maximum value of the input feature map in a channel dimension, +.>
Figure BDA0003482350870000123
Representing a convolution operation on a source signature of an infrared image,/->
Figure BDA0003482350870000124
And (3) carrying out convolution operation on the source characteristic diagram of the visible light image, and obtaining a fusion source characteristic diagram after weighting by a double-channel maximum pooling self-adaptive fusion mechanism by X degrees.
Thus, the step 105 specifically includes:
based on the two-channel self-adaptive fusion network, a formula is adopted
Figure BDA0003482350870000125
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure BDA0003482350870000126
Source signature representing infrared image, +.>
Figure BDA0003482350870000127
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure BDA0003482350870000128
representing a convolution operation on a source signature of an infrared image,/->
Figure BDA0003482350870000129
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; and X DEG represents a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
Step 106: and carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images based on the double-channel self-adaptive fusion network to obtain fused detail information feature images.
And carrying out double-channel average pooling self-adaptive fusion on the two groups of detail compensation feature images, and uniformly distributing detail texture information of the two groups of detail compensation feature images in the fused feature images in the channel dimension, so as to finally self-adaptively enhance effective information of two source images in the fused image. The calculation formula of the two-channel self-adaptive fusion network fusion detail information feature map is as follows:
Figure BDA0003482350870000131
In the formula (4), the amino acid sequence of the compound,
Figure BDA0003482350870000132
detail compensation feature map representing an input infrared image,/->
Figure BDA0003482350870000133
Detail-compensated feature map representing visible light image, mean represents averaging of the input feature map in channel dimension,/->
Figure BDA0003482350870000134
Representing a convolution operation of a detail-compensated feature map of an infrared image,/>
Figure BDA0003482350870000135
The convolution operation is performed on the detail compensation characteristic diagram of the visible light image, sigma represents the sigmoid operation and X represents d And (5) representing a detail fusion characteristic diagram obtained after weighting by a double-channel average pooling self-adaptive fusion mechanism.
Thus, the step 106 specifically includes:
based on the two-channel self-adaptive fusion network, a formula is adopted
Figure BDA0003482350870000136
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure BDA0003482350870000137
Detail information feature map representing infrared image, < >>
Figure BDA0003482350870000138
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension; />
Figure BDA0003482350870000139
The characteristic diagram of the detail information representing the infrared image is convolved,
Figure BDA00034823508700001310
performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; x is X d And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
Step 107: and splicing the fusion source characteristic diagram and the fusion detail information characteristic diagram by adopting the double-channel self-adaptive fusion network to obtain a spliced characteristic diagram.
Two groups of fusion feature images (fusion source feature images and fusion detail information feature images) are spliced in the channel dimension to serve as input, and a 1*1 convolution network is constructed to achieve information interaction and information fusion of feature images across channels.
Step 108: and inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels, so as to obtain the fused feature images.
The invention adopts the two-channel self-adaptive fusion network to splice the fusion source characteristic diagram and the fusion detail information characteristic diagram, and inputs the spliced characteristic diagram into a 1*1 convolution network to realize the information interaction and information fusion of the characteristic diagram crossing channels, so as to obtain the fused characteristic diagram. Using 1*1 convolution may preserve accumulating more pixel intensity information and detail texture information than 3*3 convolution; and then decoding the fused feature images to obtain fused images, so that the final fused result contains rich source image information.
Step 109: and decoding the fused feature images by adopting a decoding network to obtain fused images.
And finally, decoding the feature images after fusion to obtain a fusion image.
Step 110: and sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network.
The self-adaptive enhancement generation countermeasure network takes the dense detail feature extraction network as a coding network, and performs feature extraction on two input source images by combining a dense convolution mode with a detail information compensation mechanism; the infrared information and the visible light information in the fused image are balanced in the channel dimension by using a two-channel self-adaptive fusion network; splicing the two groups of fusion feature graphs, and constructing a 1*1 convolution network to realize cross-channel interaction and information fusion; and finally, decoding the feature images after fusion to obtain the fused image.
Step 111: and constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fusion image and the source image.
The invention adopts the self-adaptive structure similarity loss function to calculate the structure similarity, contrast similarity and brightness similarity of the fusion image and two source images obtained by the network model (namely the self-adaptive enhancement generation countermeasure network); using a 3*3 sliding window to obtain the average value of the pixel intensities of the two source image sliding windows; obtaining the weight corresponding to each window according to the average sigmoid; the structural similarity of all windows is calculated, an average value is calculated and is used as a final self-adaptive structural similarity loss function value, and the network parameters of the network model are adjusted through back propagation, so that effective image information can be self-adaptively enhanced in the space dimension. The adaptive structural similarity loss function calculation formulas are shown in formulas (5) to (9):
Brightness similarity:
Figure BDA0003482350870000151
contrast similarity:
Figure BDA0003482350870000152
structural similarity:
Figure BDA0003482350870000153
structural similarity:
Figure BDA0003482350870000154
wherein x and y represent the input source image and the fusion image, mu x 、μ y Representing the average value, σ, of two images (source and fusion images) x 、σ y Representing standard deviation, sigma, of two images xy Representing covariance of the two images; c (C) 1 、C 2 、C 3 Is a minimum number in order to ensure the stability of the algorithm; ssim (x, y) represents the structural similarity of two pictures.
The adaptive structural similarity loss function is expressed as follows:
Figure BDA0003482350870000155
in the formula (9), vi w Representing visible light image blocks in a sliding window, f w Representing a fused image block in a sliding window, ir w Representing an infrared image block in a sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window;
Figure BDA0003482350870000156
pixel mean value representing visible light image block in sliding window,/>
Figure BDA0003482350870000157
Indicating slidingPixel average value of infrared image block in window; SSIM represents the value of the final structural similarity loss function in a sliding window.
Thus, the step 111 specifically includes:
using the formula
Figure BDA0003482350870000158
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
using the formula
Figure BDA0003482350870000161
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number; />
Using the formula
Figure BDA0003482350870000162
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
calculating the structural similarity ssim (x, y) of the fusion image and the source image according to the brightness similarity l (x, y), the contrast similarity c (x, y) and the structural similarity s (x, y) of the fusion image and the source image by adopting a formula ssim (x, y) =l (x, y) ·c (x, y) ·s (x, y);
constructing an adaptive structural similarity loss function according to the structural similarity ssim (x, y) of the fusion image and the source image
Figure BDA0003482350870000163
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing the infrared image blocks in the sliding window,f w representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; />
Figure BDA0003482350870000164
Pixel mean value representing visible light image block in sliding window,/>
Figure BDA0003482350870000165
Representing a pixel average value of the infrared image block in the sliding window; SSIM represents the value of the final structural similarity loss function in a sliding window.
Step 112: and training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function, and generating a trained self-adaptive enhancement generation countermeasure network.
Based on the self-adaptive structure similarity loss function, the whole network model is trained for multiple times through back propagation, and the network after parameter training optimization can self-adaptively enhance infrared and visible light image information.
Step 113: and adopting the trained self-adaptive enhancement generation countermeasure network to perform image fusion of the infrared image and the visible light image.
The invention adaptively enhances the generation of a fusion image of an infrared image and a visible light image which are input into an countermeasure network and output into the infrared image and the visible light image. The trained self-adaptive enhancement generation countermeasure network is adopted to carry out image fusion on the infrared image shown in the figure 1 and the visible light image shown in the figure 2, and the generated fusion image is shown in the figure 6. As can be seen from fig. 6, the adaptive enhancement generation of the fusion image generated by the countermeasure network enhances the detail information of the image, balances the infrared information and the visible light information in the fusion image, and greatly improves the image quality of the fusion image.
The method solves the problem of information loss of the end-to-end generation countermeasure network when extracting the features by utilizing the dense detail feature extraction network; the effective information distribution problem of the infrared image and the visible light image in the fusion image is solved in the channel dimension and the space dimension respectively by using the double-channel attention mechanism and the self-adaptive structure similarity loss function, and the quality of the fusion image is effectively improved.
Based on the image fusion method for generating the countermeasure network based on the self-adaptive enhancement, the invention also provides an image fusion system for generating the countermeasure network based on the self-adaptive enhancement, which comprises the following steps:
the source image acquisition module is used for acquiring a source image; the source image comprises an infrared image and a visible light image;
the dense detail feature extraction network construction module is used for constructing a dense detail feature extraction network by combining the dense convolution network with a detail information compensation mechanism;
the feature extraction module is used for carrying out feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; the source characteristic map of the source image comprises a source characteristic map of an infrared image and a source characteristic map of a visible light image; the detail information feature map comprises a detail information feature map of an infrared image and a detail information feature map of a visible light image;
The double-channel self-adaptive fusion network construction module is used for constructing a double-channel self-adaptive fusion network based on a double-channel maximum pooling self-adaptive fusion mechanism and a double-channel average pooling self-adaptive fusion mechanism;
the source characteristic diagram fusion module is used for carrying out double-channel maximum pooling self-adaptive fusion on the source characteristic diagram of the source image based on the double-channel self-adaptive fusion network to obtain a fusion source characteristic diagram;
the detail information feature map fusion module is used for carrying out double-channel average pooling self-adaptive fusion on the detail information feature map of the source image based on the double-channel self-adaptive fusion network to obtain a fused detail information feature map;
the fusion feature map splicing module is used for splicing the fusion source feature map and the fusion detail information feature map by adopting the double-channel self-adaptive fusion network to obtain a spliced feature map;
the convolution network fusion module is used for inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels so as to obtain fused feature images;
the feature map code module is used for decoding the fused feature map by adopting a decoding network to obtain a fused image;
The self-adaptive enhancement generation countermeasure network construction module is used for sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network;
the self-adaptive structure similarity loss function construction module is used for constructing a self-adaptive structure similarity loss function according to the brightness similarity, the contrast similarity and the structure similarity of the fusion image and the source image;
the self-adaptive enhancement generation countermeasure network training module is used for training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function to generate a trained self-adaptive enhancement generation countermeasure network;
and the image fusion module is used for carrying out image fusion on the infrared image and the visible light image by adopting the trained self-adaptive enhancement generation countermeasure network.
The feature extraction module specifically comprises:
a feature extraction unit for extracting a network formula p by using dense detail features based on the dense detail feature extraction network i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i Representing the density ofExtracting an i-th layer feature map of the network by the detail features; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; broadcast (x) -p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
The source characteristic diagram fusion module specifically comprises:
the source characteristic diagram fusion unit is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure BDA0003482350870000191
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure BDA0003482350870000192
Source signature representing infrared image, +.>
Figure BDA0003482350870000193
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure BDA0003482350870000194
representing a convolution operation on a source signature of an infrared image,/->
Figure BDA0003482350870000195
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; and X DEG represents a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
The detailed information feature map fusion module specifically comprises:
detail information feature map fusion unitThe method is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure BDA0003482350870000196
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure BDA0003482350870000197
Detail information feature map representing infrared image, < >>
Figure BDA0003482350870000198
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension; />
Figure BDA0003482350870000199
The characteristic diagram of the detail information representing the infrared image is convolved,
Figure BDA00034823508700001910
performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; x is X d And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
The self-adaptive structure similarity loss function construction module specifically comprises:
a brightness similarity calculation unit for using the formula
Figure BDA00034823508700001911
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
a contrast similarity calculation unit for using the formula
Figure BDA0003482350870000201
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number;
a structural similarity calculation unit for adopting the formula
Figure BDA0003482350870000202
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
a structural similarity calculating unit, configured to calculate structural similarity ssim (x, y) between the fused image and the source image according to brightness similarity l (x, y), contrast similarity c (x, y), and structural similarity s (x, y) of the fused image and the source image by using a formula ssim (x, y) =l (x, y) =c (x, y) ·s (x, y);
an adaptive structural similarity loss function construction unit for constructing an adaptive structural similarity loss function based on structural similarity ssim (x, y) of the fusion image and the source image
Figure BDA0003482350870000203
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing an infrared image block in a sliding window, f w Representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; />
Figure BDA0003482350870000204
Pixel mean value representing visible light image block in sliding window,/>
Figure BDA0003482350870000205
Representing in a sliding windowPixel average value of the infrared image block; SSIM represents the value of the final structural similarity loss function in a sliding window.
The invention discloses a method and a system for generating infrared and visible light image fusion of an countermeasure network based on self-adaptive enhancement, which are characterized in that firstly, an infrared image and a visible light image are respectively input; constructing a dense detail feature extraction network as a coding network, and respectively carrying out feature extraction on two input source images by combining a dense convolution mode with a detail information compensation mechanism to obtain two groups of source feature images and two other groups of detail information feature images; constructing a two-channel maximum pooling self-adaptive fusion mechanism to fuse two groups of source feature images to obtain a group of fusion source feature images; constructing a double-channel average pooling self-adaptive fusion mechanism to fuse two groups of detail information feature images to obtain a group of fused detail information feature images; splicing the two groups of fusion feature graphs, and constructing a 1*1 convolution network to realize cross-channel interaction and information fusion; finally, decoding the feature images after fusion to obtain a fusion image; and adding an adaptive structural similarity loss function when the whole network model is trained. According to the invention, a detail compensation mechanism is introduced into a generator coding network to enhance details of a fusion image and reduce information loss, a dual-channel adaptive fusion network is used for balancing infrared information and visible light information in the fusion image in a channel dimension, an adaptive structure similarity loss function is added for adaptively enhancing brightness similarity, contrast similarity and structure similarity of the fusion image and two source images in a space dimension, and a dense detail feature extraction network, a dual-channel attention mechanism and an adaptive structure similarity loss function are used for optimizing an infrared and visible light image fusion network model, so that the information loss problem and the effective information distribution problem existing in the prior art are solved, and the image fusion quality is effectively improved.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims (10)

1. An image fusion method for generating an countermeasure network based on adaptive enhancement, comprising:
acquiring a source image; the source image comprises an infrared image and a visible light image;
combining the dense convolution network with a detail information compensation mechanism to construct a dense detail feature extraction network;
performing feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; the source characteristic map of the source image comprises a source characteristic map of an infrared image and a source characteristic map of a visible light image; the detail information feature map comprises a detail information feature map of an infrared image and a detail information feature map of a visible light image;
Constructing a two-channel self-adaptive fusion network based on the two-channel maximum pooling self-adaptive fusion mechanism and the two-channel average pooling self-adaptive fusion mechanism;
performing two-channel maximum pooling self-adaptive fusion on the source feature images of the source images based on the two-channel self-adaptive fusion network to obtain a fusion source feature image;
performing double-channel average pooling self-adaptive fusion on the detail information feature images of the source images based on the double-channel self-adaptive fusion network to obtain fused detail information feature images;
the two-channel self-adaptive fusion network is adopted to splice the fusion source characteristic diagram and the fusion detail information characteristic diagram, and a spliced characteristic diagram is obtained;
inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels, so as to obtain fused feature images;
decoding the fused feature images by adopting a decoding network to obtain fused images;
sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network;
constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fusion image and the source image;
Training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function, and generating a trained self-adaptive enhancement generation countermeasure network;
and adopting the trained self-adaptive enhancement generation countermeasure network to perform image fusion of the infrared image and the visible light image.
2. The method according to claim 1, wherein the feature extraction of the source image based on the dense detail feature extraction network obtains a source feature map and a detail information feature map of the source image, specifically comprising:
based on the dense detail feature extraction network, adopting a dense detail feature extraction network formula p i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i An i-th layer feature map representing the dense detail feature extraction network; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; br (br)oadcast(x)-p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
3. The method according to claim 1, wherein the performing, based on the two-channel adaptive fusion network, two-channel maximum pooling adaptive fusion on the source feature map of the source image to obtain a fused source feature map specifically includes:
based on the two-channel self-adaptive fusion network, a formula is adopted
Figure FDA0003482350860000021
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure FDA0003482350860000022
Source signature representing infrared image, +.>
Figure FDA0003482350860000023
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure FDA0003482350860000024
representing a convolution operation on a source signature of an infrared image,/->
Figure FDA0003482350860000025
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; x is X o And (5) representing a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
4. The method according to claim 1, wherein the performing, based on the two-channel adaptive fusion network, two-channel average pooling adaptive fusion on the detail information feature map of the source image to obtain a fused detail information feature map specifically includes:
Based on the two-channel self-adaptive fusion network, a formula is adopted
Figure FDA0003482350860000031
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure FDA0003482350860000032
Detail information feature map representing infrared image, < >>
Figure FDA0003482350860000033
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension; />
Figure FDA0003482350860000034
The characteristic diagram of the detail information representing the infrared image is convolved,
Figure FDA0003482350860000035
performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; x is X d And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
5. The method according to claim 1, wherein the constructing an adaptive structural similarity loss function according to the brightness similarity, the contrast similarity and the structural similarity of the fused image and the source image specifically comprises:
using the formula
Figure FDA0003482350860000036
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
using the formula
Figure FDA0003482350860000037
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number; />
Using the formula
Figure FDA0003482350860000038
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
calculating the structural similarity ssim (x, y) of the fusion image and the source image according to the brightness similarity l (x, y), the contrast similarity c (x, y) and the structural similarity s (x, y) of the fusion image and the source image by adopting a formula ssim (x, y) =l (x, y) ·c (x, y) ·s (x, y);
constructing an adaptive structural similarity loss function according to the structural similarity ssim (x, y) of the fusion image and the source image
Figure FDA0003482350860000041
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing an infrared image block in a sliding window, f w Representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; / >
Figure FDA0003482350860000042
Pixel mean value representing visible light image block in sliding window,/>
Figure FDA0003482350860000043
Representing a pixel average value of the infrared image block in the sliding window; SSIM represents the value of the final structural similarity loss function in a sliding window.
6. An image fusion system for generating an countermeasure network based on adaptive enhancement, comprising:
the source image acquisition module is used for acquiring a source image; the source image comprises an infrared image and a visible light image;
the dense detail feature extraction network construction module is used for constructing a dense detail feature extraction network by combining the dense convolution network with a detail information compensation mechanism;
the feature extraction module is used for carrying out feature extraction on the source image based on the dense detail feature extraction network to obtain a source feature image and a detail information feature image of the source image; the source characteristic map of the source image comprises a source characteristic map of an infrared image and a source characteristic map of a visible light image; the detail information feature map comprises a detail information feature map of an infrared image and a detail information feature map of a visible light image;
the double-channel self-adaptive fusion network construction module is used for constructing a double-channel self-adaptive fusion network based on a double-channel maximum pooling self-adaptive fusion mechanism and a double-channel average pooling self-adaptive fusion mechanism;
The source characteristic diagram fusion module is used for carrying out double-channel maximum pooling self-adaptive fusion on the source characteristic diagram of the source image based on the double-channel self-adaptive fusion network to obtain a fusion source characteristic diagram;
the detail information feature map fusion module is used for carrying out double-channel average pooling self-adaptive fusion on the detail information feature map of the source image based on the double-channel self-adaptive fusion network to obtain a fused detail information feature map;
the fusion feature map splicing module is used for splicing the fusion source feature map and the fusion detail information feature map by adopting the double-channel self-adaptive fusion network to obtain a spliced feature map;
the convolution network fusion module is used for inputting the spliced feature images into a 1*1 convolution network to realize information interaction and information fusion of the feature images across channels so as to obtain fused feature images;
the feature map code module is used for decoding the fused feature map by adopting a decoding network to obtain a fused image;
the self-adaptive enhancement generation countermeasure network construction module is used for sequentially connecting the dense detail feature extraction network, the dual-channel self-adaptive fusion network, the 1*1 convolution network and the decoding network to form a self-adaptive enhancement generation countermeasure network;
The self-adaptive structure similarity loss function construction module is used for constructing a self-adaptive structure similarity loss function according to the brightness similarity, the contrast similarity and the structure similarity of the fusion image and the source image;
the self-adaptive enhancement generation countermeasure network training module is used for training the network parameters of the self-adaptive enhancement generation countermeasure network through back propagation based on the self-adaptive structure similarity loss function to generate a trained self-adaptive enhancement generation countermeasure network;
and the image fusion module is used for carrying out image fusion on the infrared image and the visible light image by adopting the trained self-adaptive enhancement generation countermeasure network.
7. The system of claim 6, wherein the feature extraction module specifically comprises:
a feature extraction unit for extracting a network formula p by using dense detail features based on the dense detail feature extraction network i ＝conv i (x 1 ,cat(…,conv 2 (cat(x i-2 ,conv 1 (cat(x i-1 ,x i ) -a) and y) i ＝cat(p i ,broadcast(x)-p i ) Extracting the characteristics of the source image to obtain a source characteristic image and a detail information characteristic image of the source image; wherein x represents the source image, x i An i-th layer feature map representing the dense detail feature extraction network; cat () means that the feature map in brackets is spliced on the feature channel; conv i () The ith layer convolution operation of extracting the characteristics of the spliced characteristic graphs in brackets is shown; p is p i Representing the extracted i-th layer source feature map; broadcast (x) -p i Representing a source signature p i Corresponding i-th layer detail information feature map, wherein broadcast (x) represents a broadcasting mechanism to automatically expand the dimension of the source image x; y is i Representing the i-th layer full feature map of the dense detail feature extraction network.
8. The system of claim 6, wherein the source signature fusion module specifically comprises:
the source characteristic diagram fusion unit is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure FDA0003482350860000061
Performing double-channel maximum pooling self-adaptive fusion on the source feature images of the source images to obtain fusion source feature images; wherein (1)>
Figure FDA0003482350860000062
Source signature representing infrared image, +.>
Figure FDA0003482350860000063
A source signature representing a visible light image; max () represents maximizing the channel dimension for the feature map in brackets;
Figure FDA0003482350860000064
representing a convolution operation on a source signature of an infrared image,/->
Figure FDA0003482350860000065
Performing convolution operation on a source characteristic diagram of the visible light image; sigma () represents a sigmoid operation; x is X o And (5) representing a fusion source characteristic diagram obtained after weighting by a double-channel maximum pooling self-adaptive fusion mechanism.
9. The system of claim 6, wherein the detailed information feature map fusion module specifically comprises:
the detail information feature map fusion unit is used for adopting a formula based on the two-channel self-adaptive fusion network
Figure FDA0003482350860000066
Carrying out double-channel average pooling self-adaptive fusion on the detail information feature images of the source images to obtain fused detail information feature images; wherein->
Figure FDA0003482350860000067
Detail information feature map representing infrared image, < >>
Figure FDA0003482350860000068
A detailed information feature map representing a visible light image; mean () represents averaging the feature map in brackets over the channel dimension; />
Figure FDA0003482350860000069
The characteristic diagram of the detail information representing the infrared image is convolved,
Figure FDA00034823508600000610
performing convolution operation on the detail information feature map of the visible light image; sigma () represents a sigmoid operation; />
Figure FDA00034823508600000611
And (5) representing a fusion detail information characteristic diagram obtained after weighting through a double-channel average pooling self-adaptive fusion mechanism.
10. The system of claim 6, wherein the adaptive structural similarity loss function building block comprises:
a brightness similarity calculation unit for using the formula
Figure FDA0003482350860000071
Calculating the brightness similarity l (x, y) of the fusion image and the source image; wherein mu x Mean value of pixel intensity, mu, representing source image x sliding window y Representing the average value of pixel intensities of a sliding window of the fused image y; c (C) 1 Is a minimum number;
a contrast similarity calculation unit for using the formula
Figure FDA0003482350860000072
Calculating contrast similarity c (x, y) of the fused image and the source image; wherein sigma x Representing standard deviation, sigma, of source image x y Representing the standard deviation of the fused image y; c (C) 2 Is a minimum number;
a structural similarity calculation unit for adopting the formula
Figure FDA0003482350860000073
Calculating the structural similarity s (x, y) between the fusion image and the source image; wherein sigma xy Representing the covariance of the source image x and the fused image y; c (C) 3 Is a minimum number;
a structural similarity calculating unit, configured to calculate structural similarity ssim (x, y) between the fused image and the source image according to brightness similarity l (x, y), contrast similarity c (x, y), and structural similarity s (x, y) of the fused image and the source image by using a formula ssim (x, y) =l (x, y) =c (x, y) ·s (x, y);
an adaptive structural similarity loss function construction unit for constructing an adaptive structural similarity loss function based on structural similarity ssim (x, y) of the fusion image and the source image
Figure FDA0003482350860000074
Wherein vi w Representing visible light image blocks in a sliding window, ir w Representing an infrared image block in a sliding window, f w Representing the fused image block in the sliding window; ssim (vi) w +f w ) Representing the structural similarity of the fused image block and the visible light image block in the sliding window, ssim (ir) w +f w ) Representing the structural similarity of the fusion image block and the infrared image block in the sliding window; />
Figure FDA0003482350860000075
Pixel mean value representing visible light image block in sliding window,/>
Figure FDA0003482350860000076
Representing a pixel average value of the infrared image block in the sliding window; SSIM represents the value of the final structural similarity loss function in a sliding window. />
CN202210071844.XA 2022-01-21 2022-01-21 Image fusion method and system for generating countermeasure network based on self-adaptive enhancement Active CN114419328B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210071844.XA CN114419328B (en) 2022-01-21 2022-01-21 Image fusion method and system for generating countermeasure network based on self-adaptive enhancement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210071844.XA CN114419328B (en) 2022-01-21 2022-01-21 Image fusion method and system for generating countermeasure network based on self-adaptive enhancement

Publications (2)

Publication Number Publication Date
CN114419328A CN114419328A (en) 2022-04-29
CN114419328B true CN114419328B (en) 2023-05-05

Family

ID=81275736

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210071844.XA Active CN114419328B (en) 2022-01-21 2022-01-21 Image fusion method and system for generating countermeasure network based on self-adaptive enhancement

Country Status (1)

Country Link
CN (1) CN114419328B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709902A (en) * 2020-05-21 2020-09-25 江南大学 Infrared and visible light image fusion method based on self-attention mechanism
CN111915545A (en) * 2020-08-06 2020-11-10 中北大学 Self-supervision learning fusion method of multiband images
CN112733950A (en) * 2021-01-18 2021-04-30 湖北工业大学 Power equipment fault diagnosis method based on combination of image fusion and target detection
CN113935935A (en) * 2021-10-19 2022-01-14 天翼数字生活科技有限公司 Dark light image enhancement method based on fusion of visible light and near infrared light

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8144927B2 (en) * 2008-01-31 2012-03-27 Max-Viz, Inc. Video image processing and fusion

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709902A (en) * 2020-05-21 2020-09-25 江南大学 Infrared and visible light image fusion method based on self-attention mechanism
CN111915545A (en) * 2020-08-06 2020-11-10 中北大学 Self-supervision learning fusion method of multiband images
CN112733950A (en) * 2021-01-18 2021-04-30 湖北工业大学 Power equipment fault diagnosis method based on combination of image fusion and target detection
CN113935935A (en) * 2021-10-19 2022-01-14 天翼数字生活科技有限公司 Dark light image enhancement method based on fusion of visible light and near infrared light

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Infrared and Visible Image Fusion through Details Preservation;Yaochen Liu等;《Sensors 2019》;第19卷(第20期);1-16 *
任意分辨率红外与可见光图像融合算法研究;岑悦亮;《中国优秀硕士学位论文全文数据库 (信息科技辑)》(第01期);I135-229 *

Also Published As

Publication number Publication date
CN114419328A (en) 2022-04-29

Similar Documents

Publication Publication Date Title
CN110363716B (en) High-quality reconstruction method for generating confrontation network composite degraded image based on conditions
CN111028150B (en) Rapid space-time residual attention video super-resolution reconstruction method
CN115713679A (en) Target detection method based on multi-source information fusion, thermal infrared and three-dimensional depth map
CN110351548B (en) Stereo image quality evaluation method guided by deep learning and disparity map weighting
CN113077505A (en) Optimization method of monocular depth estimation network based on contrast learning
CN114972748A (en) Infrared semantic segmentation method capable of explaining edge attention and gray level quantization network
CN116485867A (en) Structured scene depth estimation method for automatic driving
CN116012255A (en) Low-light image enhancement method for generating countermeasure network based on cyclic consistency
CN114926337A (en) Single image super-resolution reconstruction method and system based on CNN and Transformer hybrid network
CN112348033B (en) Collaborative saliency target detection method
CN114419328B (en) Image fusion method and system for generating countermeasure network based on self-adaptive enhancement
CN110503049B (en) Satellite video vehicle number estimation method based on generation countermeasure network
CN111275751A (en) Unsupervised absolute scale calculation method and system
CN116823647A (en) Image complement method based on fast Fourier transform and selective attention mechanism
CN113362251B (en) Anti-network image defogging method based on double discriminators and improved loss function
CN114092827A (en) Image data set generation method
Vismaya et al. Fuzzy c-means algorithm based depth estimation in single image
CN117078516B (en) Mine image super-resolution reconstruction method based on residual mixed attention
CN111914852B (en) Multi-mode saliency object detection method based on coding and decoding structure
CN116863320B (en) Underwater image enhancement method and system based on physical model
CN115456903B (en) Deep learning-based full-color night vision enhancement method and system
CN117474801B (en) Non-uniform remote sensing video image defogging method integrating space-time frequency information
Yang et al. Self-taught recovery of depth data
CN117115058A (en) Low-light image fusion method based on light weight feature extraction and color recovery
CN117115021A (en) Dynamic underwater image enhancement method and device based on priori knowledge

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant