CN115587955A

CN115587955A - Image fusion method and apparatus, storage medium, and electronic apparatus

Info

Publication number: CN115587955A
Application number: CN202211336876.4A
Authority: CN
Inventors: 余彦; 殷俊
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2022-10-28
Filing date: 2022-10-28
Publication date: 2023-01-10

Abstract

The application discloses an image fusion method and device, a storage medium and an electronic device, wherein the method comprises the following steps: respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group comprises low-frequency images respectively corresponding to the images, and the high-frequency image group comprises high-frequency images respectively corresponding to the images; respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; carrying out image fusion on images in the low-frequency image group to obtain a low-frequency fusion image; and carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image.

Description

Image fusion method and apparatus, storage medium, and electronic apparatus

Technical Field

The present application relates to the field of computers, and in particular, to an image fusion method and apparatus, a storage medium, and an electronic apparatus.

Background

In the related art, multi-sensor image (e.g., visible light image, infrared image, etc.) fusion can be widely applied to the aspects of target recognition, machine vision, remote sensing, medical image processing, etc. In order to solve the problems that a multi-scale transformation fusion method needs to process a large amount of data and is low in efficiency, a compressed sensing mode can be adopted for multi-sensor image fusion, and the compressed sensing-based method can be used for compressing while sampling so as to reduce the data amount and improve the data processing efficiency.

For example, a single layer wavelet decomposition may be performed on the visible light image and the infrared image; fusing the low-frequency sub-band coefficients (the sub-band coefficients are images) after wavelet decomposition to obtain fused low-frequency sub-band coefficients; carrying out global weighted fusion on the high-frequency sub-band coefficients to obtain fused high-frequency sub-band coefficients; and performing wavelet inverse transformation on the fused low-frequency sub-band coefficient and the fused high-frequency sub-band coefficient to obtain a fused image.

However, global weighting fusion is performed on the high-frequency subband coefficients, so that the complexity of fusion calculation is high, the calculation amount is large, and the efficiency of image fusion is low. Therefore, the image fusion method in the related art has the problem of low efficiency of image fusion due to high complexity of fusion calculation.

Disclosure of Invention

The embodiment of the application provides an image fusion method and device, a storage medium and an electronic device, which are used for solving the problem that the image fusion method in the related art is low in efficiency due to high complexity of fusion calculation.

According to an aspect of an embodiment of the present application, there is provided an image fusion method, including: respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group comprises low-frequency images respectively corresponding to the images, and the high-frequency image group comprises high-frequency images respectively corresponding to the images; respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; performing image fusion on images in the low-frequency image group to obtain a low-frequency fusion image; and carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of the at least one image.

According to another aspect of embodiments of the present application, there is provided an image fusion apparatus including: the image fusion device comprises a transformation unit, a fusion unit and a fusion unit, wherein the transformation unit is used for respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, the low-frequency image group comprises low-frequency images respectively corresponding to the images, and the high-frequency image group comprises high-frequency images respectively corresponding to the images; the processing unit is used for respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; the execution unit is used for respectively carrying out image block fusion on the image blocks in each image block pair in the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; the fusion unit is used for carrying out image fusion on the images in the low-frequency image group to obtain a low-frequency fusion image; and the inverse transformation unit is used for carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of the at least one image.

In one exemplary embodiment, the transformation unit includes: and the transformation module is used for respectively carrying out layer-of-four-direction non-downsampling contourlet transformation on the at least one image to be fused to obtain the low-frequency image group and the high-frequency image group.

In one exemplary embodiment, the execution unit includes: the first execution module is used for multiplying a measurement matrix with the image blocks in each image block pair respectively to obtain a plurality of observation image block pairs, wherein the column number of the measurement matrix is the same as the row number of the image blocks in each image block pair; the fusion module is used for respectively fusing the pixel points of each observation image block pair in the plurality of observation image block pairs based on the matching degree between the regional energies of the pixel points to obtain a plurality of fusion measurement image blocks; and the reconstruction module is used for reconstructing each fused measurement image block in the plurality of fused measurement image blocks by using a sensing matrix to obtain the plurality of fused image blocks, wherein the sensing matrix is obtained by multiplying the measurement matrix by a redundant dictionary matched with the measurement matrix.

In one exemplary embodiment, the fusion module includes: the first execution sub-module is configured to respectively use each observation image block pair as a current observation image block pair to execute the following operations to obtain the multiple fusion measurement image blocks: taking each pixel point in the current observation image block pair as a current pixel point to respectively execute the following fusion operations: determining the matching degree of the current observation image block to the regional energy of the current pixel point to obtain the current matching degree; under the condition that the current matching degree is greater than or equal to a matching degree threshold value, carrying out weighted summation on the pixel value of the current pixel point of the current observation image block pair to obtain the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair; and under the condition that the current matching degree is smaller than a matching degree threshold, determining the pixel value of a target observation image block at the current pixel point as the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest area energy at the current pixel point in the current observation image block pair.

In one exemplary embodiment, the apparatus further comprises: a determining unit, configured to determine, before performing weighted summation on the pixel value of the current pixel in the current observation image block pair, a product of a ratio of a first difference to a second difference and a preset coefficient as a first weight, and determine, as a second weight, a difference between 1 and the first weight, where the first difference is a difference between the current matching degree and the matching degree threshold, the second difference is a difference between 1 and the matching degree threshold, the first weight is a weighting coefficient corresponding to an observation image block corresponding to the current pixel in the current observation image block pair and having a largest pixel value, and the second weight is a weighting coefficient corresponding to an observation image block corresponding to the current pixel in the current observation image block pair and having a smallest pixel value.

In one exemplary embodiment, the reconstruction module includes: a second execution sub-module, configured to perform the following reconstruction operations on each column of each fused measured image block using the sensing matrix, to obtain the multiple fused image blocks, where each column is a current column in a process of performing the following reconstruction operations: constructing sparse coefficients matching the current column using the perceptual matrix; and multiplying the sparse coefficient by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

In one exemplary embodiment, the fusion unit includes: a second execution module, configured to respectively use each pixel point of an image in the low-frequency image group as a current pixel point to execute the following fusion operation, so as to obtain a pixel value of the current pixel point of the low-frequency fusion image: respectively determining the pixel value variance of a group of pixel points in a preset field with the current pixel point as the center in each image of the low-frequency image group to obtain the field variance corresponding to each image; and determining the maximum domain variance in the domain variances corresponding to the images as the pixel value of the current pixel point in the low-frequency fusion image.

According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the image fusion method when running.

According to another aspect of the embodiments of the present application, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the image fusion method through the computer program.

In the embodiment of the application, a mode of fusing the obtained high-frequency image blocks after the high-frequency images are partitioned is adopted, and a low-frequency image group and a high-frequency image group corresponding to at least one image are obtained by respectively carrying out image transformation in multiple directions on at least one image to be fused; the low-frequency image group comprises low-frequency images corresponding to the images respectively, and the high-frequency image group comprises high-frequency images corresponding to the images respectively; respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; respectively carrying out image block fusion on image blocks in each image block pair in the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; performing image fusion on images in the low-frequency image group to obtain a low-frequency fusion image; and carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image. The high-frequency image is partitioned, the high-frequency image blocks obtained by partitioning are fused firstly, and the high-frequency fused image is obtained by splicing the fused image blocks, so that the grid effect between the blocks during partitioning solving can be reduced, the purpose of reducing the complexity of image fusion calculation is realized, the technical effect of improving the efficiency of image fusion is achieved, and the problem of low efficiency of image fusion caused by high complexity of fusion calculation in the image fusion method in the related art is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

FIG. 1 is a schematic diagram of a hardware environment for an alternative image fusion method according to an embodiment of the present application;

FIG. 2 is a schematic flow chart diagram illustrating an alternative image fusion method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an alternative image fusion method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an alternative image fusion method according to an embodiment of the application;

FIG. 5 is a schematic flow chart diagram of an alternative image fusion method according to an embodiment of the present application;

FIG. 6 is a block diagram of an alternative image fusion apparatus according to an embodiment of the present application;

fig. 7 is a block diagram of an alternative electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of an embodiment of the present application, there is provided an image fusion method. Optionally, in this embodiment, the image fusion method may be applied to a hardware environment including the terminal device 102 and the server 104 as shown in fig. 1. As shown in fig. 1, the server 104 is connected to the terminal device 102 through a network, and may be configured to provide services (e.g., application services, etc.) for the terminal device or a client installed on the terminal device, and a database may be provided on the server or separately from the server, and is configured to provide data storage services for the server 104.

The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. The terminal device 102 may be, but is not limited to, a smart phone, a smart computer, a smart tablet, and the like.

The image fusion method according to the embodiment of the present application may be executed by the server 104, or executed by the terminal device 102, or executed by both the server 104 and the terminal device 102. Taking the terminal device 102 to execute the image fusion method in this embodiment as an example, fig. 2 is a schematic flowchart of an optional image fusion method according to this embodiment, and as shown in fig. 2, the flowchart of the method may include the following steps:

step S202, image transformation in multiple directions is respectively carried out on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group comprises low-frequency images corresponding to the images respectively, and the high-frequency image group comprises high-frequency images corresponding to the images respectively.

The image fusion method in this embodiment may be applied to at least one scene in which image fusion is performed, where one or more fused images may be multi-sensor images, which may be images of the same scene in different bands, or images of other types that need to be subjected to image fusion, and multiple images may be images acquired by the same image acquisition component or images acquired by different image acquisition components. The at least one image may include, but is not limited to, an image of at least one of: visible light image, infrared image, correspondingly, the image acquisition component can be a camera, an infrared sensor and the like.

In the related art, the method for fusing images of multiple sensors mainly comprises the following steps: the method for multi-scale transformation fusion is based on a compressed sensing method, wherein the multi-scale transformation fusion method needs to process a large amount of data and is not high in efficiency, the method based on compressed sensing is used for compressing while sampling, the data volume is reduced, the efficiency is improved, but image information is not fully decomposed, the reconstruction complexity is high, and the high-frequency information fusion effect is poor.

For example, for an image fusion mode based on compressed sensing, single-layer wavelet decomposition may be performed on an input visible light image and an input infrared image to obtain a decomposed low-frequency subband coefficient and a decomposed high-frequency subband coefficient; fusing the low-frequency subband coefficients, and fusing the high-frequency subband coefficients based on the observation coefficients; and performing wavelet inverse transformation on the fused low-frequency sub-band coefficient and the fused high-frequency sub-band coefficient to obtain a fused image. However, the global weighting fusion method for the observation coefficients of the high-frequency components has a large amount of calculation, and the selection of the fusion weight cannot adapt to each region, and the accuracy cannot be guaranteed.

In order to at least partially solve the above problem, in this embodiment, a high-frequency subband coefficient obtained by image transformation may be subjected to blocking processing, and corresponding image blocks obtained by blocking may be fused, and since the high-frequency subband coefficient is subjected to blocking and global fusion of an image is split into fusion of image blocks, the amount of calculation required for image fusion may be reduced, and meanwhile, parameters (for example, fusion weights) used in the fusion may be applied to each region, and accuracy of image fusion may be improved.

For at least one image to be fused, for example, a visible light image (image a) and an infrared image (image B), image transformation in multiple directions can be respectively performed on each image, so that a low-frequency image corresponding to each image and a high-frequency image corresponding to each image are obtained, and thus a low-frequency image group and a high-frequency image group are obtained, wherein the sizes of the respective images in the low-frequency image group can be the same, and the sizes of the respective images in the high-frequency image group can be the same. The number of low frequency image groups may be one or more, for example, 1, and the number of high frequency image groups may be one or more, for example, 4. The processing procedures of different low-frequency image groups are similar, and the processing procedures of different high-frequency image groups are also similar, and in the case of no conflict, the method for fusing the low-frequency image groups is applicable to each low-frequency image group, and the method for fusing the high-frequency image groups is applicable to each high-frequency image group.

The image transformation may include image decomposition, which may be a plurality of differently scaled transformations for each image, and the low-frequency image (i.e., low-frequency subband coefficients) and the high-frequency image (i.e., high-frequency subband coefficients) resulting from the image transformation for each image may be a plurality of differently scaled sub-images for each image decomposition.

Optionally, before performing image fusion, in order to improve the quality of image fusion, the two images may be preprocessed to align scenes in different images (i.e., performing scene alignment), and the preprocessing operation may include: angle change operations, etc. In this embodiment, the preprocessing process of the image is not limited.

And step S204, respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group.

For a high frequency image set, each high frequency image set may contain two high frequency images. For each high-frequency image, the high-frequency image can be subjected to blocking processing according to a preset size, and a plurality of image block pairs corresponding to the high-frequency image group are obtained. For example, a sliding window may be used to perform a blocking process of a preset size on an image in the high-frequency image group, and there is a partial overlap between adjacent blocks, thereby obtaining a plurality of image block pairs corresponding to the high-frequency image group.

For example, a sliding window is used to block the high frequency image n × n, keeping a partial overlap between adjacent blocks. For different high-frequency images in one high-frequency image group, the image blocks at the same position can be used as one image block pair, and since each high-frequency image can be blocked into a plurality of image blocks, at the end of blocking, a plurality of image block pairs can be obtained.

And step S206, respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image.

After obtaining the plurality of image block pairs, the terminal device may perform image block fusion on the image block pairs in the plurality of image block pairs, and a mode of fusing each image block pair may be similar to a mode of fusing the entire high-frequency image. For example, each image block pair is fused based on compressed sensing, resulting in a fused image block corresponding to each image block pair, thereby resulting in a plurality of fused image blocks, where the plurality of image block pairs correspond to the plurality of fused image blocks one-to-one, i.e., one image block pair is fused into one fused image block. Optionally, in order to improve the efficiency of image block fusion, different pairs of image blocks may be image block fused by multiple threads, each thread may be used to image block fuse at least one pair of image blocks.

After the plurality of fused image blocks are obtained, the plurality of fused image blocks can be subjected to image block splicing according to the positions of the image blocks in the image block pairs in the high-frequency image to obtain the high-frequency fused image, or the plurality of fused image blocks can be subjected to image block splicing according to the position relation among the fused image blocks to obtain the high-frequency fused image. In the process of splicing the fused image blocks, for the pixels with overlaps between the adjacent image blocks, the pixel value of the pixel in any image block or the average value of the pixel values of all the pixels can be taken as the pixel value of the overlapping pixel in the high-frequency fused image.

For example, after obtaining the high-frequency fused image blocks (i.e., fused image blocks), the image blocks may be stitched, and the overlapping portions averaged to obtain the high-frequency fused image.

And step S208, carrying out image fusion on the images in the low-frequency image group to obtain a low-frequency fusion image.

The terminal equipment can perform image fusion on the low-frequency image group to obtain a low-frequency fusion image. The low-frequency image group can be fused in the following modes: the image fusion is performed on the low-frequency image group by using an averaging method (i.e., a global average), and other image fusion methods may also be used.

And step S210, carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image.

After obtaining the high-frequency fused image and the low-frequency fused image, the terminal device may perform inverse image transformation corresponding to the aforementioned image transformation on the high-frequency fused image and the low-frequency fused image, thereby obtaining a fused image of at least one image.

For example, the obtained low-frequency fused image and the high-frequency fused image are inversely transformed to obtain a fused image of the visible light image and the infrared image.

Through the steps S202 to S210, performing image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, where the low-frequency image group includes low-frequency images corresponding to the respective images, and the high-frequency image group includes high-frequency images corresponding to the respective images; respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; respectively carrying out image block fusion on image blocks in each image block pair in the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; carrying out image fusion on images in the low-frequency image group to obtain a low-frequency fusion image; the image inverse transformation is carried out on the high-frequency fusion image and the low-frequency fusion image to obtain the fusion image of at least one image, the problem that the image fusion efficiency is low due to the high complexity of fusion calculation in the image fusion method in the related technology is solved, and the image fusion efficiency is improved.

In an exemplary embodiment, the performing image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image includes:

s11, performing non-downsampling contourlet transformation of a layer of four directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group.

The image transformation to be performed on the image to be fused may be various, for example, wavelet transformation. In order to improve the defect that the Contourlet is easy to change during the Contourlet translation, and facilitate the subsequent image processing, each image to be fused may be subjected to image transformation by using NSCT (non Sampled Contourlet Transform) or a similar image transformation manner with translation invariance (e.g., curvelet Transform, contourlet Transform, etc.).

In this embodiment, two images to be fused may be subjected to NSCT transformation in multiple directions, respectively, to obtain a low-frequency image group and a high-frequency image group. The number of layers of NSCT transformation is different, and the number of the obtained high-frequency image groups is different. Compared with other image transformation, the NSCT transformation carries out upsampling operation, so that the defect that the contour wave is easy to change during translation can be overcome, detailed information can be effectively highlighted, and the quality of a fused image can be improved.

For example, the visible light image and the infrared image may be decomposed into a plurality of sub-images with different directions and different scales by NSCT, and the visible light image and the infrared image may be decomposed into a low frequency image and a plurality of high frequency images, respectively.

In order to reduce the complexity of image fusion and improve the efficiency of image fusion, a layer of four-direction NSCT transformation may be performed on at least one image to be fused, and each image is subjected to a layer of four-direction NSCT transformation and is respectively a low-frequency image and four high-frequency images, so as to obtain a low-frequency image group and four high-frequency image groups, where each high-frequency image group includes a high-frequency image obtained by performing the same-direction NSCT transformation on each image of the at least one image.

For example, as shown in fig. 3, a layer-four-direction NSCT transform may be performed on the visible light image and the infrared image to be fused, respectively, and after each image is subjected to the image transform, 5 sub-images including 1 low-frequency image and 4 high-frequency images may be obtained, that is, 1 low-frequency image group and 4 high-frequency image group are obtained in total. In the decomposition reconstruction process of NSCT, a down-sampling process is not available, the multi-directionality and the anisotropy of the traditional contourlet transform to the image representation are inherited, and the translation invariance which is not possessed by the traditional contourlet transform is provided.

By the embodiment, the complexity of image fusion can be reduced and the efficiency of image fusion can be improved by performing one-layer four-direction NSCT on the image to be fused.

In an exemplary embodiment, respectively performing image block fusion on an image block in each of a plurality of image block pairs to obtain a plurality of fused image blocks includes:

s21, multiplying the measurement matrix with the image blocks in each image block pair respectively to obtain a plurality of observation image block pairs, wherein the column number of the measurement matrix is the same as the row number of the image blocks in each image block pair;

s22, respectively fusing the pixel points of each observation image block pair in the plurality of observation image block pairs based on the matching degree between the regional energies of the pixel points to obtain a plurality of fused measurement image blocks;

and S23, reconstructing each fused measurement image block in the multiple fused measurement image blocks by using a sensing matrix to obtain the multiple fused image blocks, wherein the sensing matrix is obtained by multiplying the measurement matrix by a redundant dictionary matched with the measurement matrix.

In this embodiment, image block fusion may be performed on each image block pair based on the observation coefficients. Correspondingly, the parameters used for image block fusion may include, but are not limited to: measurement matrix, redundant dictionary (matching measurement matrix), sensing matrix. The measurement matrix and the redundant dictionary can be randomly constructed, or can be obtained by training with training samples based on a set target, and the perception matrix is obtained by multiplying the measurement matrix and the redundant dictionary. In this embodiment, the construction manner of the measurement matrix and the redundant dictionary is not limited.

For example, a random bernoulli measurement matrix Φ and a DCT (Discrete Cosine Transform) redundant dictionary Ψ may be constructed, where the measurement matrix dimension m × n, m < n, and the redundant dictionary dimension n × 4n are multiplied to obtain a sensing matrix D.

For each image block pair, the column number of the measurement matrix is the same as the row number of the image block in each image block pair, and the measurement matrix and the image block in each image block pair can be multiplied respectively to obtain a corresponding observation image block pair, thereby obtaining a plurality of observation image block pairs. For example, the high-frequency image blocks may be multiplied by the measurement matrix respectively to obtain high-frequency observation image blocks of the visible light image and the infrared image.

In this embodiment, no matter whether the image is a high-frequency image (image block) or a low-frequency image, the image size (the number of pixels included) of the fused image obtained by image fusion is the same as the image size of the image before fusion, that is, the image obtained by fusing two images having the image size of C × D is still the image of C × D. For a current observation image block pair (which may be any observation image block pair), each pixel point in the observation image block pair may be respectively fused based on a matching degree between regional energies of the pixel points to obtain a corresponding fused measured image block, and a plurality of observation image block pairs may obtain a plurality of fused measured image blocks.

When a current pixel point (any pixel point in the current observation image block pair) of the current observation image block pair is carried out, the regional energy in a preset region corresponding to each observation image block of the current observation image block pair and the current pixel point can be respectively determined; and fusing the pixel values of the current pixel points in the current observation image block pair based on the determined matching degree between the regional energies (namely, the regional energy matching degree) to obtain the pixel values of the current pixel points in the fused measurement image block.

Here, when the pixel value of the current pixel point is fused based on the local energy matching degree, the weighting coefficient of each observation image block may be determined based on the local energy matching degree, and the pixel value of the current pixel point of each observation image block is weighted and fused based on the determined weighting coefficient, so as to obtain the pixel value of the current pixel point in the fusion measurement image block.

For example, the high-frequency observation image blocks may be fused using a high-frequency coefficient fusion rule. When the high-frequency observation image blocks are fused, the regional energy E of the high-frequency observation image blocks HA and HB at the pixel points (x, y) can be calculated based on the formula (1) _A (x,y)、E _B (x,y)：

Where H is the image block, lxw is the energy region size, which can be set to 3 × 3, ω is the weighting matrix, which can be

And reconstructing each fused measurement image block by using a sensing matrix based on compressed sensing to obtain a plurality of fused image blocks. The manner of reconstructing the image block by using the perceptual matrix may refer to related technologies, which is not limited in this embodiment.

In this embodiment, an observation image block is obtained by multiplying an observation matrix by a high-frequency image block, fusion is performed by using a region energy matching method, a pixel value of a pixel point is determined by using energy characteristics of an image in a surrounding region with the pixel point as a center, and the determined pixel value includes high-frequency characteristics of the image in the region, so that a fusion effect can be improved, more details can be retained, and noise can be reduced.

In an exemplary embodiment, respectively fusing the pixel points of each observation image block pair in the plurality of observation image block pairs based on the matching degree between the regional energies of the pixel points to obtain a plurality of fused measurement image blocks, including:

s31, taking each observation image block pair as a current observation image block pair, and respectively executing the following operations to obtain a plurality of fusion measurement image blocks:

taking each pixel point in the current observation image block pair as a current pixel point to respectively execute the following fusion operations:

determining the matching degree of the current observation image block pair between the regional energy of the current pixel point to obtain the current matching degree;

under the condition that the current matching degree is greater than or equal to the matching degree threshold, carrying out weighted summation on the pixel value of the current pixel point of the current observation image block pair to obtain the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair;

and under the condition that the current matching degree is smaller than the matching degree threshold, determining the pixel value of the current pixel point of the target observation image block as the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest regional energy of the current pixel point in the current observation image block pair.

In this embodiment, the same processing operation may be adopted to perform observation image block fusion on each of the plurality of observation image block pairs. Meanwhile, when image block fusion is carried out on an observation image block pair, the same fusion operation can be adopted to determine the pixel value after fusion of each pixel point. Here, the following fusion operations are respectively performed with the currently processed pixel point as the current pixel point and the observation image block pair to which the current pixel point belongs as the current observation image block pair, so as to obtain the pixel value of the current pixel point in the fused measurement image block after the fusion of the current observation image block pair:

step 1, determining the matching degree of the current observation image block to the regional energy of the current pixel point to obtain the current matching degree.

For example, the local energy matching degree S (x, y) for each pair of high-frequency observation image blocks HA, HB may be calculated based on equation (2):

based on the formula (2), the regional energy matching degree of each pair of high-frequency observation image blocks HA and HB at each pixel point can be determined.

And 2, under the condition that the current matching degree is greater than or equal to the matching degree threshold, carrying out weighted summation on the pixel values of the current pixel points of the current observation image block pair to obtain the pixel values of the current pixel points in the fusion measurement image block corresponding to the current observation image block pair.

If the current matching degree is greater than or equal to the matching degree threshold, the pixel values of the current pixel points in the current observation image block pair can be subjected to weighted summation to obtain the pixel values of the current pixel points in the fusion measurement image block corresponding to the current observation image block pair. Here, the weighted summation of the pixel values of the current pixel point for the current observation image block may be: and carrying out weighted summation on the pixel values of the current pixel points of all the observation image blocks in the current observation image block pair. The weighting coefficients corresponding to different observed image blocks may be determined according to the current matching degree, or may be preset, which is not limited in this embodiment.

For example, when the observed image block is fused, an energy matching degree threshold value delta of 0.5 ≦ delta ≦ 1 may be set, and when the energy matching degree is greater than or equal to the threshold value, it indicates that the similarity of the block is high, and weighted fusion may be performed.

And 3, under the condition that the current matching degree is smaller than the matching degree threshold, determining the pixel value of the current pixel point of the target observation image block as the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest regional energy of the current pixel point in the current observation image block pair.

If the current matching degree is smaller than the matching degree threshold, the observation image block with the largest area energy in the current pixel point in the current observation image block pair can be determined to obtain the target observation image block, for example, the area energies of all the observation image blocks in the current observation image block pair in the current pixel point can be compared, the observation image block corresponding to the largest area energy is determined, and the target observation image block is obtained; and determining the pixel value of the current pixel point of the target observation image block as the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair. In this case, the weighting coefficient corresponding to the target observation image block may be 1, and the weighting coefficients corresponding to the remaining observation image blocks may be 0.

For example, if the energy matching degree is lower than the threshold, it indicates that the similarity of the region is low, and the region with high region energy is used as the fusion result, and the calculation formula is as follows:

and HC (x, y) is the pixel value of the pixel point (x, y) in the fusion measurement image block.

By the embodiment, the fusion of the measurement image blocks is performed based on the threshold of the energy similarity and the energy matching degree of the region, so that the fusion effect can be ensured, and the complexity of fusion calculation can be reduced.

In an exemplary embodiment, before performing the weighted summation on the pixel values of the current pixel point of the current observation image block, the method further includes:

s41, determining the product of the ratio of the first difference to the second difference and a preset coefficient as a first weight, and determining the difference between 1 and the first weight as a second weight, wherein the first difference is the difference between the current matching degree and a matching degree threshold, the second difference is the difference between 1 and the matching degree threshold, the first weight is a weighting coefficient corresponding to an observation image block with the largest pixel value of a current pixel point corresponding to the current observation image block pair, and the second weight is a weighting coefficient corresponding to an observation image block with the smallest pixel value of a current pixel point corresponding to the current observation image block pair.

Before the weighted summation of the current observation image block pair on the pixel values of the current pixel point, the weighting coefficients corresponding to the respective observation image blocks in the current observation image block pair may be determined, and the weighting coefficients corresponding to different observation image blocks may be preset, for example, the weighting coefficient corresponding to the observation image block with high area energy is a larger value, and the weighting coefficient corresponding to the observation image block with low area energy is a smaller value. In order to ensure the image fusion effect, the weighting coefficient corresponding to each observed image block can be determined based on the current matching degree and the matching degree threshold.

In this embodiment, the product of the ratio of the difference between the current matching degree and the matching degree threshold to the difference between 1 and the matching degree threshold and the preset coefficient may be determined as the weighting coefficient corresponding to the largest observed image block in the pixel values of the corresponding current pixel points in the current observed image block pair; and determining the difference value between 1 and the first weight as a weighting coefficient corresponding to the other observation image block in the current observation image block pair. The predetermined coefficient may be a predetermined value, for example, 0.5.

For example, when the observation image blocks are weighted and fused based on formula (4):

wherein HC (x, y) is a fused observed image, w _max And w _min To merge weight values, w _max And w _min The calculation formula (2) is shown in formula (5):

according to the embodiment, the weighting coefficient corresponding to each observation image block is determined based on the current matching degree and the matching degree threshold, so that the image fusion effect can be ensured, and the quality of the fused image is improved.

In an exemplary embodiment, reconstructing each of the plurality of fused measured image blocks using the perceptual matrix to obtain a plurality of fused image blocks comprises:

s51, using the sensing matrix to respectively execute the following reconstruction operations on each column in each fusion measurement image block to obtain a plurality of fusion image blocks, wherein each column is a current column in the process of executing the following reconstruction operations:

constructing sparse coefficients matched with the current column by using a perception matrix;

and multiplying the sparse coefficient by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

In this embodiment, when reconstructing each fused measured image block, a perceptual matrix may be used to perform a reconstruction operation on each column of each fused measured image block, so as to reconstruct a corresponding column of the corresponding fused image block, and further obtain a fused image block corresponding to each fused measured image block. The multiple fused measurement image blocks can be correspondingly reconstructed into multiple fused image blocks, and the multiple fused image blocks have a one-to-one correspondence relationship.

When a column to be currently reconstructed (which may be any column in any fused measurement image block) is reconstructed, the following reconstruction operations may be performed on the column as the current column:

step 1, constructing a sparse coefficient matched with a current column by using a sensing matrix.

For the current column, a sparse coefficient matched with the current column can be constructed by using a sensing matrix through a reconstruction algorithm in compressed sensing, wherein the used reconstruction algorithm can be an OMP (Orthogonal Matching Pursuit) algorithm or other reconstruction algorithms; when the column reconstruction is carried out, a group of atoms which are most matched with the current column can be searched from the sensing matrix, a sparse approximation is constructed, and a sparse coefficient is obtained through iteration.

For example, the OMP algorithm may be used to reconstruct the fused high-frequency image block from the fused high-frequency measured image block HC. When the fused high-frequency image block is reconstructed, a group of atoms which are most matched with the fused high-frequency observation image block can be searched from the sensing matrix for each column of the fused high-frequency observation image block, a sparse approximation is constructed, and a sparse coefficient is obtained through iteration. The process of obtaining the sparse coefficient may include:

1) Note the first column of the image block as x, firstInitialized residual r ₀ Is x, index set a is empty.

2) Finding out perception matrix D and residual r _k The foot mark with the largest inner product row is recorded as

d _i Is an atom of the sensing matrix and N is the sensing matrix dimension.

3) The most relevant element index found is added to the index set, i.e., A _k+1 ＝A _k ∪{λ _k+1 }。

4) The residual error is updated, i.e.,

5) Ending the iteration after the residual error is less than the threshold value to obtain a sparse coefficient of

Multiplying the sparse coefficient by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block

And 2, multiplying the sparse coefficient by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

For example, after obtaining the sparse coefficients, the sparse coefficients may be multiplied by a redundant dictionary (e.g., the DWT dictionary Ψ) to obtain a column of the reconstructed image, and the above operations may be repeated for each column of the image block to obtain the reconstructed high-frequency fusion image block.

Through the embodiment, a column of matched sparse coefficients in the measurement image block are constructed and fused by using the perception matrix, and then column reconstruction is performed based on the coefficient coefficients and the redundant dictionary, so that the convenience of image block reconstruction can be improved.

In an exemplary embodiment, image fusion is performed on images in the low-frequency image group to obtain a low-frequency fusion image, and the method includes:

s61, respectively taking each pixel point of the image in the low-frequency image group as a current pixel point to execute the following fusion operation, and obtaining the pixel value of the current pixel point of the low-frequency fusion image:

s62, respectively determining the pixel value variance of a group of pixel points in a preset field with the current pixel point as the center in each image of the low-frequency image group to obtain the field variance corresponding to each image;

and S63, determining the maximum domain variance in the domain variances corresponding to the images as the pixel value of the current pixel point in the low-frequency fusion image.

Although the low-frequency image group can be fused simply and conveniently by adopting the global average value, partial information is lost, and the outline information of the fused image is inaccurate. In this embodiment, a mode in which the domain variance takes the maximum value may be used to fuse low-frequency components (i.e., low-frequency images) to obtain fused low-frequency components, the image information amount in the region is represented by the variance (i.e., the variance reflects the dispersion of the gray value of the image, and the larger the variance, the larger the information amount is), and the pixel values of each point are affected by the image features of the adjacent regions, so that the main content and the contour of the image can be retained to a greater extent.

For the low-frequency image, the value obtained by fusing the pixel points of each pixel position can be respectively determined, and the value is used as the pixel value of the pixel point of the pixel position corresponding to the fused low-frequency fused image. For each pixel point (for example, a pixel point at a position a × b) in the low-frequency image group, the fusion operation can be performed by taking the maximum value of the domain variance as the current pixel point, so as to obtain the pixel value of the current pixel point (for example, a pixel point at a position a × b) in the low-frequency fusion image. Performing the fusing operation on the current pixel point may include the following operations:

operation 1, respectively determining a pixel value variance of a group of pixel points in a preset domain centered on a current pixel point in each low-frequency image, and obtaining a domain variance corresponding to each low-frequency image.

For example, the low-frequency image is fused by using a rule that the domain variance takes the maximum value, and for any one of the visible light image and the infrared image, a pixel point with the coordinate (x, y) in the image can be taken, and the variance in the neighborhood range with the size of M is calculated through the formula (6):

wherein L (x, y) is a low frequency image,

is the mean of the low frequency images in the neighborhood M.

And 2, determining the maximum domain variance in the domain variances corresponding to each low-frequency image as the pixel value of the current pixel point in the low-frequency fusion image.

After the domain variance (corresponding to the current pixel point) corresponding to each low-frequency image is obtained, the maximum domain variance among the domain variances corresponding to each low-frequency image may be selected, and the maximum domain variance is determined as the pixel value of the current pixel point in the low-frequency fusion image, or the pixel value of the low-frequency image corresponding to the maximum domain variance at the current pixel point may be determined as the pixel value of the current pixel point in the low-frequency fusion image.

For example, for a pixel point, an image pixel value of the visible light image and the infrared image with a large variance in the field of the pixel point may be selected as a pixel value of the low-frequency fusion image.

Through the embodiment, the low-frequency image is fused in a mode that the domain variance takes the maximum value, so that the main content and the outline of the image can be retained to a greater extent, and the accuracy of the fused image is improved.

The following explains an image fusion method in the embodiment of the present application with reference to an alternative example. In this optional example, the two images are a visible light image and an infrared image, respectively, the image transform is NSCT transform, the inverse transform is NSCT inverse transform, and the image block reconstruction uses an OMP algorithm.

The optional example provides an image fusion method based on NSCT transform and compressed sensing, as shown in fig. 4 and 5, the flow of the image fusion method in the optional example may include the following steps:

step S502, respectively carrying out layer-four-direction NSCT conversion on the visible light image and the infrared image to respectively obtain 5 sub-images comprising 1 low-frequency image and 4 high-frequency images.

In step S504, the high-frequency image is blocked (pixel overlap between adjacent image blocks is maintained, and the overlapping portion is averaged during reconstruction) to obtain a high-frequency image block, and the observation matrix is multiplied by the high-frequency image block to obtain an observation image block (i.e., a high-frequency observation image block).

The image segmentation is actually an overall constraint by blocking the high-frequency components, and reduces the grid effect between blocks when the blocks are solved.

Step S506, fusing the observation image blocks based on the regional energy matching degree to obtain fused observation image blocks, reconstructing the fused observation image blocks based on compressed sensing to obtain reconstructed fused image blocks (high-frequency fused image blocks), and splicing the fused image blocks to obtain a high-frequency fused image.

Here, the high-frequency image is fused by using the rule of region energy matching, and the principle is as follows: and carrying out blocking, sparse expression and observation on the high-frequency components to obtain observation coefficients on a compressed domain, fusing the observation coefficients of each block by using a rule of region energy matching, and obtaining a fused high-frequency image through a reconstruction algorithm.

And step S508, fusing the low-frequency images by taking the maximum value based on the domain variance to obtain low-frequency fused images.

Step S510, performing NSCT inverse transformation on the obtained fused low-frequency image and fused high-frequency image (i.e., fused components), to obtain an image in which the visible light image and the infrared image are fused.

By the optional example, information of the image in different scales and different directions can be fully mined, more details are reserved after the high-frequency part is fused, the calculation complexity is reduced by blocking processing, and meanwhile, the overlapping part is averaged, so that the grid effect is avoided; the main content and contours of the image are preserved to a greater extent for the low frequency part.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art will recognize that the embodiments described in this specification are preferred embodiments and that acts or modules referred to are not necessarily required for this application.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., a ROM (Read-Only Memory)/RAM (Random Access Memory), a magnetic disk, an optical disk) and includes several instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the methods according to the embodiments of the present application.

According to another aspect of the embodiment of the present application, an image fusion apparatus for implementing the image fusion method is also provided. Fig. 6 is a block diagram of an alternative image fusion apparatus according to an embodiment of the present application, and as shown in fig. 6, the apparatus may include:

a transforming unit 602, configured to perform image transformation in multiple directions on at least one image to be fused, respectively, to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image; the low-frequency image group comprises low-frequency images corresponding to the images respectively, and the high-frequency image group comprises high-frequency images corresponding to the images respectively;

a processing unit 604, connected to the transforming unit 602, configured to perform block processing on the images in the high-frequency image group according to a preset size, so as to obtain a plurality of image block pairs corresponding to the high-frequency image group;

the execution unit 606 is connected to the processing unit 604, and is configured to perform image block fusion on the image blocks in each of the plurality of image block pairs to obtain a plurality of fused image blocks, and perform image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image;

a fusion unit 608 connected to the execution unit 606, configured to perform image fusion on the images in the low-frequency image group to obtain a low-frequency fusion image;

and an inverse transformation unit 610, connected to the fusion unit 608, configured to perform image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image.

It should be noted that the transforming unit 602 in this embodiment may be configured to execute the step S202, the processing unit 604 in this embodiment may be configured to execute the step S204, the executing unit 606 in this embodiment may be configured to execute the step S206, the fusing unit 608 may be configured to execute the step S208, and the inverse transforming unit 610 may be configured to execute the step S210.

Respectively carrying out image transformation in multiple directions on at least one image to be fused through the module to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image; the low-frequency image group comprises low-frequency images corresponding to the images respectively, and the high-frequency image group comprises high-frequency images corresponding to the images respectively; respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group; respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image; carrying out image fusion on images in the low-frequency image group to obtain a low-frequency fusion image; the image inverse transformation is carried out on the high-frequency fusion image and the low-frequency fusion image to obtain the fusion image of at least one image, the problem that the image fusion efficiency is low due to the high complexity of fusion calculation in the image fusion method in the related technology is solved, and the image fusion efficiency is improved.

In one exemplary embodiment, the transformation unit includes:

and the transformation module is used for respectively carrying out layer-of-four-direction non-downsampling contourlet transformation on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group.

In one exemplary embodiment, the execution unit includes:

the first execution module is used for multiplying the measurement matrix with the image blocks in each image block pair respectively to obtain a plurality of observation image block pairs, wherein the column number of the measurement matrix is the same as the row number of the image blocks in each image block pair;

the fusion module is used for respectively fusing the pixel points of each observation image block pair in the plurality of observation image block pairs based on the matching degree between the regional energies of the pixel points to obtain a plurality of fusion measurement image blocks;

and the reconstruction module is used for reconstructing each fused measurement image block in the plurality of fused measurement image blocks by using the sensing matrix to obtain the plurality of fused image blocks, wherein the sensing matrix is obtained by multiplying the measurement matrix by a redundant dictionary matched with the measurement matrix.

In one exemplary embodiment, the fusion module includes:

the first execution sub-module is used for respectively taking each observation image block pair as a current observation image block pair to execute the following operations to obtain a plurality of fusion measurement image blocks: taking each pixel point in the current observation image block pair as a current pixel point to respectively execute the following fusion operations:

In an exemplary embodiment, the apparatus further includes:

the determining unit is used for determining a product of a ratio of a first difference value and a second difference value and a preset coefficient as a first weight value before performing weighted summation on the pixel value of the current pixel point of the current observation image block pair, and determining a difference value between 1 and the first weight value as a second weight value, wherein the first difference value is a difference value between the current matching degree and a matching degree threshold value, the second difference value is a difference value between 1 and the matching degree threshold value, the first weight value is a weighting coefficient corresponding to an observation image block with the largest pixel value of the current pixel point corresponding to the current observation image block pair, and the second weight value is a weighting coefficient corresponding to an observation image block with the smallest pixel value of the current pixel point corresponding to the current observation image block pair.

In one exemplary embodiment, the reconstruction module includes:

the second execution sub-module is configured to perform the following reconstruction operations on each column of each fused measurement image block using the sensing matrix, so as to obtain a plurality of fused image blocks, where each column is a current column in a process of performing the following reconstruction operations:

constructing sparse coefficients matched with the current column by using a perception matrix; the sparse coefficients are multiplied by the redundant dictionary,

and obtaining the current column in the fused image block corresponding to each fused measurement image block.

In one exemplary embodiment, the fusion unit includes:

the second execution module is used for respectively taking each pixel point of the image in the low-frequency image group as a current pixel point to execute the following fusion operation so as to obtain the pixel value of the current pixel point of the low-frequency fusion image:

respectively determining the pixel value variance of a group of pixel points in a preset field with the current pixel point as the center in each image of the low-frequency image group to obtain the field variance corresponding to each image;

and determining the maximum domain variance in the domain variances corresponding to the images as the pixel value of the current pixel point in the low-frequency fusion image.

It should be noted that the modules described above are the same as examples and application scenarios realized by corresponding steps, but are not limited to what is disclosed in the foregoing embodiments. It should be noted that the modules described above as a part of the apparatus may be operated in a hardware environment as shown in fig. 1, and may be implemented by software, or may be implemented by hardware, where the hardware environment includes a network environment.

According to still another aspect of an embodiment of the present application, there is also provided a storage medium. Optionally, in this embodiment, the storage medium may be configured to execute a program code of any one of the image fusion methods described in the embodiments of the present application.

Optionally, in this embodiment, the storage medium may be located on at least one of a plurality of network devices in a network shown in the above embodiment.

Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

the method comprises the steps of S1, respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group comprises low-frequency images corresponding to the images respectively, and the high-frequency image group comprises high-frequency images corresponding to the images respectively;

s2, respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group;

s3, respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image;

s4, carrying out image fusion on the images in the low-frequency image group to obtain a low-frequency fusion image;

and S5, carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image.

Optionally, the specific example in this embodiment may refer to the example described in the above embodiment, which is not described again in this embodiment.

Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a U disk, a ROM, a RAM, a removable hard disk, a magnetic disk, or an optical disk.

According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the image fusion method, where the electronic device may be a server, a terminal, or a combination thereof.

Fig. 7 is a block diagram of an alternative electronic device according to an embodiment of the present application, as shown in fig. 7, including a processor 702, a communication interface 704, a memory 706 and a communication bus 708, where the processor 702, the communication interface 704 and the memory 706 communicate with each other via the communication bus 708, where,

a memory 706 for storing computer programs;

the processor 702, when executing the computer program stored in the memory 706, performs the following steps:

Alternatively, the communication bus may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 7, but that does not indicate only one bus or one type of bus. The communication interface is used for communication between the electronic device and other equipment.

The memory may include RAM, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Alternatively, the memory may be at least one memory device located remotely from the aforementioned processor.

As an example, the memory 706 may include, but is not limited to, the transformation unit 602, the processing unit 604, the execution unit 606, the fusion unit 608, and the inverse transformation unit 610 in the image fusion apparatus. In addition, other module units in the image fusion device may also be included, but are not limited to, and are not described in detail in this example.

The processor may be a general-purpose processor, and may include but is not limited to: a CPU (Central Processing Unit), an NP (Network Processor), and the like; but also a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.

Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.

It can be understood by those skilled in the art that the structure shown in fig. 7 is only an illustration, and the device implementing the image fusion method may be a terminal device, and the terminal device may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 7 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 7, or have a different configuration than shown in FIG. 7.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disk, ROM, RAM, magnetic or optical disk, and the like.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application, which are essential or part of the technical solutions contributing to the prior art, or all or part of the technical solutions, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the methods described in the embodiments of the present application.

In the embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be implemented in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, and may also be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or at least two units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present application and it should be noted that, as will be apparent to those skilled in the art, numerous modifications and adaptations can be made without departing from the principles of the present application and such modifications and adaptations are intended to be considered within the scope of the present application.

Claims

1. An image fusion method, comprising:

respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group comprises low-frequency images respectively corresponding to the images, and the high-frequency image group comprises high-frequency images respectively corresponding to the images;

respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group;

respectively carrying out image block fusion on the image blocks in each image block pair of the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image;

carrying out image fusion on images in the low-frequency image group to obtain a low-frequency fusion image;

and carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of the at least one image.

2. The method according to claim 1, wherein the performing image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image comprises:

and respectively carrying out non-downsampling contourlet transformation in four directions on one layer of the at least one image to be fused to obtain the low-frequency image group and the high-frequency image group.

3. The method according to claim 1, wherein the separately performing tile fusion on the tiles in each of the plurality of tile pairs to obtain a plurality of fused tiles comprises:

multiplying a measurement matrix with the image blocks in each image block pair respectively to obtain a plurality of observation image block pairs, wherein the column number of the measurement matrix is the same as the row number of the image blocks in each image block pair; respectively fusing the pixel points of each observation image block pair in the plurality of observation image block pairs based on the matching degree between the regional energies of the pixel points to obtain a plurality of fusion measurement image blocks;

and reconstructing each fused measurement image block in the plurality of fused measurement image blocks by using a sensing matrix to obtain the plurality of fused image blocks, wherein the sensing matrix is obtained by multiplying the measurement matrix by a redundant dictionary matched with the measurement matrix.

4. The method of claim 3, wherein the fusing the pixels of each of the plurality of observation image block pairs based on the matching degree between the local energies of the pixels to obtain a plurality of fused measured image blocks comprises:

and respectively taking each observation image block pair as a current observation image block pair to execute the following operations to obtain a plurality of fusion measurement image blocks:

under the condition that the current matching degree is greater than or equal to a matching degree threshold value, carrying out weighted summation on the pixel values of the current pixel points of the current observation image block pair to obtain the pixel values of the current pixel points in the fusion measurement image block corresponding to the current observation image block pair;

and under the condition that the current matching degree is smaller than a matching degree threshold, determining the pixel value of a target observation image block at the current pixel point as the pixel value of the current pixel point in the fusion measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest area energy at the current pixel point in the current observation image block pair.

5. The method of claim 4, wherein prior to said weighted summing of pixel values of said current pair of observed image blocks at said current pixel point, said method further comprises:

determining a product of a ratio of a first difference value and a second difference value and a preset coefficient as a first weight value, and determining a difference value between 1 and the first weight value as a second weight value, wherein the first difference value is a difference value between the current matching degree and the matching degree threshold value, the second difference value is a difference value between 1 and the matching degree threshold value, the first weight value is a weighting coefficient corresponding to an observation image block with the largest pixel value of the current pixel point corresponding to the current observation image block pair, and the second weight value is a weighting coefficient corresponding to an observation image block with the smallest pixel value of the current pixel point corresponding to the current observation image block pair.

6. The method of claim 3, wherein the reconstructing each of the plurality of fused measured image blocks using the perceptual matrix to obtain the plurality of fused image blocks comprises:

using the sensing matrix to respectively execute the following reconstruction operations on each column in each fused measurement image block to obtain a plurality of fused image blocks, wherein each column is a current column in the process of executing the following reconstruction operations:

constructing sparse coefficients matched with the current column by using the perception matrix;

and multiplying the sparse coefficient by the redundant dictionary to obtain the current column in the fusion image block corresponding to each fusion measurement image block.

7. The method according to any one of claims 1 to 6, wherein the image fusion of the images in the low-frequency image group to obtain a low-frequency fusion image comprises:

respectively taking each pixel point of the image in the low-frequency image group as a current pixel point to execute the following fusion operation to obtain the pixel value of the current pixel point of the low-frequency fusion image:

8. An image fusion apparatus, comprising:

the image fusion device comprises a transformation unit, a fusion unit and a fusion unit, wherein the transformation unit is used for respectively carrying out image transformation in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, the low-frequency image group comprises low-frequency images respectively corresponding to the images, and the high-frequency image group comprises high-frequency images respectively corresponding to the images;

the processing unit is used for respectively carrying out blocking processing on the images in the high-frequency image group according to a preset size to obtain a plurality of image block pairs corresponding to the high-frequency image group;

the execution unit is used for respectively carrying out image block fusion on the image blocks in each image block pair in the plurality of image block pairs to obtain a plurality of fused image blocks, and carrying out image block splicing on the plurality of fused image blocks to obtain a high-frequency fused image;

the fusion unit is used for carrying out image fusion on the images in the low-frequency image group to obtain a low-frequency fusion image;

and the inverse transformation unit is used for carrying out image inverse transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of the at least one image.

9. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 7 by means of the computer program.