CN104008533B - Multisensor Image Fusion Scheme based on block adaptive signature tracking - Google Patents
Multisensor Image Fusion Scheme based on block adaptive signature tracking Download PDFInfo
- Publication number
- CN104008533B CN104008533B CN201410271445.3A CN201410271445A CN104008533B CN 104008533 B CN104008533 B CN 104008533B CN 201410271445 A CN201410271445 A CN 201410271445A CN 104008533 B CN104008533 B CN 104008533B
- Authority
- CN
- China
- Prior art keywords
- image
- image block
- iteration
- block
- fused
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000003044 adaptive effect Effects 0.000 title claims abstract description 13
- 230000004927 fusion Effects 0.000 title abstract description 41
- 239000011159 matrix material Substances 0.000 claims abstract description 83
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 41
- 239000013598 vector Substances 0.000 claims abstract description 24
- 238000007500 overflow downdraw method Methods 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 15
- 230000003595 spectral effect Effects 0.000 claims description 7
- 238000012549 training Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 abstract description 6
- 230000000052 comparative effect Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 239000002131 composite material Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000011217 control strategy Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000002329 infrared spectrum Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
The present invention provides a kind of Multisensor Image Fusion Scheme based on block adaptive signature tracking, and this method is made up of following steps：Two width source images to be fused are divided into image block by 1, and the pixel of every piece of image block is expressed as into column vector, and the block picture element matrix for obtaining source images is represented；2 remove picture element matrix to carry out Its Sparse Decomposition after DC component, obtain the sparse decomposition coefficients of every piece of image block；The sparse decomposition coefficients of 3 pairs of image blocks are merged, and obtain fused images block coefficient matrix；The fused images block coefficient matrix of 4 pairs of image blocks is reconstructed, along with DC component, obtains fused images block picture element matrix；The fused images block picture element matrix of the image block of 5 all positions of reduction, obtains final fused images；Wherein, strategy is terminated using a kind of complex iteration in Its Sparse Decomposition, Comprehensive Control is carried out to system residual error, system residual error iteration convergence speed and system degree of rarefication, so as to more quickly obtain the fused images of better quality.
Description
Technical Field
The invention relates to the technical field of image processing, including image fusion, matching tracking, signal sparse decomposition and the like, in particular to a multi-sensor image fusion method based on block adaptive feature tracking.
Background
Generally, the purpose of image fusion is to extract useful information from image data information about the same object obtained from a multi-channel device or a multi-sensor through image processing and fusion, and finally obtain a high-quality composite image. Therefore, the utilization rate of image information is improved, the spatial resolution and the spectral resolution of the original image are improved, and convenience is provided for subsequent analysis and processing. The image fusion technology is widely applied in different fields, and can be roughly divided into three types: multi-focus image fusion, remote sensing multi-spectral image fusion and multi-modal medical imaging image fusion.
For multi-focus images, it is generally desirable to keep the information on the focal plane as detailed as possible by taking one shot in order to obtain a sharp image of a three-dimensional scene. However, such a single shot is difficult to achieve due to the physical limitations of the depth of field of the camera. But a set of differently focused images for the same scene is readily available, and by fusing the focused images at different focus positions by a specific fusion rule, a "fully focused" image can be produced. Such image fusion is commonly applied in surveillance systems, military analysis, machine vision, and vision system enhancements.
In addition, in the fields of aerospace, meteorological and topographic mapping, water conservancy construction, urban planning, real-time monitoring and the like, it is generally necessary to fuse an infrared spectrum image obtained from an infrared camera and a visible light image obtained from a digital camera. In practice, infrared images contain target detection and location information, and visible light images often provide background information for the area under investigation. Further data integrity analysis and accurate representation of the scene can be obtained through multispectral image fusion.
Various fusion rules have been proposed by researchers for image fusion. The simplest method is to directly perform weighted average on an input image in a spatial domain, but the problems of detail feature loss, contrast reduction and the like are inevitably caused. Among many signal processing methods, it is desirable to find a sparse signal representation method, and the use of sparse approximation instead of original data representation can substantially reduce the cost of signal processing and improve the compression efficiency. In recent years, the ultra-complete signal sparse representation theory is developed and applied, in 1993, Mallat and Zhang firstly propose the idea of applying an ultra-complete redundant dictionary to carry out sparse decomposition on signals, and introduce a Matching Pursuit (MP) algorithm. Compared with the traditional multi-scale transformation signal decomposition method, the signal decomposition based on the redundant dictionary maps the signal to a trained over-complete (redundant) dictionary, selects the best atoms from the dictionary through certain iteration, and then selects the linear combination of the atoms to describe the signal. The construction of the dictionary is not subject to any restrictions, the principle being that it conforms as well as possible to the structure of the signal being approximated. The number of reference atoms in the overcomplete representation dictionary exceeds the dimension of the signal, and the dictionary contains abundant transformation bases, so that the signal is represented more stably and meaningfully. The Orthogonal Matching Pursuit (OMP) algorithm is based on the MP algorithm, and all selected atoms are subjected to orthogonalization processing in each decomposition step, so that the convergence rate of the iterative algorithm is increased. However, in the process of processing complex images, both MP algorithm and OMP algorithm inevitably have the problems of excessive errors and calculation amount, too long operation time, low efficiency and the like, which affect the real-time performance and operability of the algorithm.
For each image block of a source image, the texture information contained in the image block is different, and some image blocks are smooth, have more low-frequency components and fewer high-frequency components, and have more possible details; it is not suitable if only one control mechanism is used for sparse decomposition of the image blocks. For example, for an error control mechanism, that is, for an image block with rich image details, to reach a given error, more and more atomic functions need to be allocated, and particularly, in a case that texture information of the image block and an adopted dictionary do not conform well, the problem is particularly prominent, and a sparse representation cannot be obtained, because even if a new atomic function is added, the contribution to the reduction of an approximation error is low. In addition, the control mechanism of the sparsity refers to sparse decomposition under the requirement of the given sparsity, and for the image blocks with rich image details, the sparsity can be well controlled, and excessive atomic functions are prevented from being used for representation. But is not suitable for smooth image blocks because for such image blocks, the sparsity is low, and only a small number of atomic functions are needed to complete sparse representation, but now too many atomic functions are wasted, which affects the execution efficiency of the algorithm. In summary, a single sparse decomposition control mechanism cannot provide an efficient image sparse representation scheme.
For the above reasons, the present inventors have conducted intensive studies on the existing image fusion techniques, particularly the OMP algorithm, in order to develop an image fusion method capable of overcoming the above problems.
Disclosure of Invention
In order to overcome the problems, the inventor of the present invention has conducted intensive research, and as a result, the inventor found that by changing an iteration termination condition of an OMP algorithm, the OMP algorithm is improved in a manner of comprehensively controlling a system residual, a system residual iteration convergence rate, and a system sparsity by using a composite iteration termination strategy, and at the same time, the improved OMP algorithm is used to perform sparsity decomposition on image blocks segmented from two source images to be fused, and then coefficient fusion, reconstruction of a coefficient matrix, and restoration of a pixel matrix are sequentially performed to obtain a final fused image, thereby completing the present invention.
Specifically, the present invention aims to provide the following:
(1) a multi-sensor image fusion method based on block self-adaptive feature tracking is characterized in that in the process of carrying out sparse decomposition on an image block pixel matrix, an iteration process is terminated under the following conditions:
the iteration times are greater than or equal to a preset system sparsity value;
alternatively, the iteration convergence rate of the system residual is less than or equal to a predetermined iteration convergence rate threshold, and the system residual norm is less than or equal to the square of a predetermined system residual threshold.
(2) The multi-sensor image fusion method based on block-adaptive feature tracking according to (1) above, characterized in that,
the predetermined system residual threshold is: the product of the variance of the error noise, the constant C and the arithmetic square root of the column vector dimension;
the predetermined system sparsity values are: the ratio of the dimension of a column vector generated by scanning the image block according to a line to the numerical value 3;
the predetermined iteration convergence rate threshold is a constant within a given range of 0-1.
(3) The multi-sensor image fusion method based on block-adaptive feature tracking according to (1) above, characterized in that,
the predetermined system residual threshold is
The predetermined system sparsity values are:
the predetermined iteration convergence rate threshold is cr0；
The method comprises the following steps of A, representing a residual error threshold of a preset system, C representing a constant with the value of 1.15, sigma representing the variance of error noise, sigma representing the value of 1, and N representing the dimension of a column vector generated by scanning an image block according to a line; k represents a predetermined system sparsity value, cr0The value is 0.15.
(4) The multi-sensor image fusion method based on block adaptive feature tracking according to the above (1), wherein the sparse decomposition of the image block pixel matrix is performed by using an OMP algorithm following an adaptive termination strategy, comprising the following sub-steps:
substep 1, initializing a system residual threshold, a predetermined iterative convergence rate threshold and a system sparsity value,namely, the system residual is equal to the initial value of the column vector, and an index set is set as an empty set when the iteration is not performed; wherein,represents the system residual after the 0 th iteration of the mth block image block, vmA column vector representing the mth block image block;
substep 2, calculating the subscript lambda corresponding to the maximum value in the inner product between the system residual error of the ith iteration and the columns of the dictionary matrixi(ii) a Wherein i is a natural number for representing the number of iterations;
substep 3, using the subscript λiUpdate index set, Λi＝Λi-1∪{λiAnd recording a set D of reconstructed atoms in the selected dictionary matrixi，Wherein, ΛiIndicating a new index set, Λ, i.e. the index set after the ith iterationi-1For the index set after the i-1 th iteration, { λiIs a collection of dictionary indices, Di-1Is the set of atoms after the i-1 st iteration,is an atomic function corresponding to the dictionary index;
substep 4, obtaining the estimation variable of the sparse matrix coefficient after the ith iteration by the least square methodEstimated variation of the sparse matrix coefficientsMeasurement ofI.e. using the selected set of atoms DiFor vmPerforming optimal representation to obtain coefficient values;
substep 5, updating the residual error of the ith iterationUpdating the iteration times i to i + 1; wherein r isiRefers to the system residual after the ith iteration, ri-1The system residual error after the i-1 iteration is referred to;
substep 6, judging whether the system residual error, the iterative convergence rate of the system residual error and the iteration times meet the termination iteration condition of the iteration process, if so, terminating the iteration to obtain a sparse decomposition coefficient; if not, returning to the substep 2 and continuing to the subsequent substeps.
(5) The multi-sensor image fusion method based on block-adaptive feature tracking according to the above (1), wherein the method comprises the following steps
Step 1, dividing two source images to be fused into image blocks, wherein the size of each image block is consistent, and any two adjacent image blocks have overlapping parts; expressing the pixels of each image block into a column vector to obtain a pixel matrix of each image block, namely obtaining the block pixel matrix expression of the source image;
step 2, removing direct current components from the pixel matrix of each image block, and then performing sparse decomposition to obtain a sparse decomposition coefficient of each image block;
step 3, fusing the sparse decomposition coefficients of each image block at the corresponding position according to the energy maximization principle to obtain a fused image block coefficient matrix of the image block at the position;
step 4, reconstructing the fused image block coefficient matrix of the image block at the position, and adding the reconstructed value of the direct current component to obtain a fused image block pixel matrix of the image block at the position;
step 5, restoring the fused image block pixel matrixes of the image blocks at all positions to obtain a final fused image;
the two source images to be fused are two images obtained at different focal length positions aiming at the same scene, or two spectral images under the spectral conditions of different illumination light wavelengths aiming at the same scene;
the step of removing the direct current component comprises calculating an average pixel value of each image block, and then subtracting the average value from the pixel value of each image block, namely the direct current component is the average pixel value of each image block;
the energy maximum principle is that the maximum absolute value of the sparse decomposition coefficient of the image blocks at the corresponding positions of the two source images is used as the sparse decomposition coefficient of the corresponding positions of the fused image blocks;
reconstruction is the reconstruction of the spatial pixel representation of each image block, i.e. the reduction of a one-dimensional representation of an image block to a two-dimensional representation.
(6) The multi-sensor image fusion method based on block adaptive feature tracking according to the above (5), wherein two adjacent image blocks having an overlapping portion are obtained by a sliding window technique, and two pixel points are translated each time, that is, the sliding step is 2.
(7) The multi-sensor image fusion method based on the block adaptive feature tracking according to the above (5), wherein the reconstruction method comprises: and recovering the spatial pixel representation of the image block from the fused sparse coefficient values, and calculating the average value of the pixel values at the same position of the overlapped part between the adjacent image blocks as the calculation result of the final pixel matrix.
(8) The multi-sensor image fusion method based on the block adaptive feature tracking according to the above (5), wherein the reduction fusion image block pixel matrix includes a pixel matrix of an overlapping portion of the reduction fusion image block and a pixel matrix of a direct current component portion; and for the pixel matrix of the direct current component part, the direct current component pixel matrix of the fused image is obtained by carrying out weighting operation on the direct current component of the source image block.
(9) The multi-sensor image fusion method based on block-adaptive feature tracking according to the above (5), wherein the dictionary used in the method is an overcomplete dictionary, which is a fixed training dictionary,
or the mixed dictionary is obtained by mixing the wavelet domain dictionary and the DCT redundant dictionary.
The invention combines the system residual, the convergence rate of the system residual and the sparsity of the system to carry out comprehensive control, the pixel matrix of each image block adaptively selects a sparse representation scheme, namely, whether iteration is stopped is judged according to an adaptive iteration termination condition, and if the system residual does not reach a specified error threshold or the convergence rate is higher than a specified threshold and the sparsity is in a controllable range, the iteration process is continuously executed if the pixel matrix of the image block is in the sparse decomposition process. For a smooth image block, even if the system residual reaches the preset requirement, the convergence rate is still fast, and the iterative process is allowed to continue, because the human visual system is often sensitive to the error of the smooth area; on the other hand, for an image block with rich texture, if the system residual is higher than the preset value and the convergence rate is lower than the preset threshold, the iteration process should stop at this time, because when the convergence rate is lower, it has been shown that the dictionary and the residual signal do not have similarity, and there is no meaning to continue the iteration, and at this time, the final approximation error of the texture image block is allowed to be relatively higher, which also conforms to the human visual system, i.e., it is not sensitive to the error of the texture-rich area.
Specifically, the beneficial effects of the invention include:
(1) the two sets of iteration termination strategies can automatically select the adaptive iteration termination strategy according to the characteristics of different image blocks, so that the image fusion time is short and the fusion efficiency is high;
(2) the iteration termination strategy is a composite iteration termination strategy, and the requirements of system residual error and sparsity are fully considered, so that the image fusion effect is better, the visual quality of the produced fusion image is higher, and no obvious artificial effect is introduced.
Drawings
FIG. 1 shows a flow diagram of an image fusion method according to a preferred embodiment of the present invention;
FIG. 2 illustrates an atomic function diagram of a training dictionary;
FIG. 3 illustrates an atomic function diagram of a hybrid dictionary;
FIG. 4 shows a right focused source image to be fused;
FIG. 5 shows a left focused source image to be fused;
FIG. 6 illustrates the resulting image from image fusion according to the method provided by the present invention;
FIG. 7 illustrates the resulting image from image fusion using an error control strategy to control the termination of the iteration;
fig. 8 shows the image finally obtained by image fusion using sparsity control strategy to control iteration termination.
Detailed Description
The invention is explained in more detail below with reference to the figures and examples. The features and advantages of the present invention will become more apparent from the description.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
In a multi-sensor image fusion method based on block adaptive feature tracking provided by the present invention, as shown in fig. 1, the method comprises the following steps:
step 1, dividing two source images to be fused into image blocks, wherein the size of each image block is consistent, and any two adjacent image blocks have overlapping parts; expressing the pixels of each image block into a column vector to obtain a pixel matrix of each image block, namely obtaining the block pixel matrix expression of the source image;
step 2, removing direct current components from the pixel matrix of each image block, and then performing sparse decomposition to obtain a sparse decomposition coefficient of each image block;
step 3, fusing the sparse decomposition coefficients of each image block at the corresponding position according to the energy maximization principle to obtain a fused image block coefficient matrix of the image block at the position;
step 4, reconstructing the fused image block coefficient matrix of the image block at the position, and adding the reconstructed value of the direct current component to obtain a fused image block pixel matrix of the image block at the position;
step 5, restoring the fused image block pixel matrixes of the image blocks at all positions to obtain a final fused image;
in a preferred embodiment, the two source images to be fused in step 1 are two images obtained at different focal positions for the same scene, or two spectral images, such as an infrared image and a visible light image, for the same scene under different spectral conditions of the wavelength of the illumination light; or medical images from different devices, such as images of bones and tissues, etc.
In a further preferred embodiment, step 1 is: firstly, inputting two source images I to be fused with the same sizeAAnd IBDividing the two source images into fixed-size and mutually overlapped image blocksExpressing the pixels of each image block into a column vector to obtain a block pixel matrix expression of the source image; n in the present invention refers to the dimension of the column vector generated by scanning the image blocks in rows, and as a preferred embodiment, N may be set equal to 64, i.e. each source image is divided into a set of 8 × 8 size image blocks; traversing from the upper left corner to the lower right corner of the source image pixel by adopting a sliding window technology, rearranging the corresponding pixels of each image block into a column of vectors of a matrix according to the dictionary sequence, and obtaining a block pixel matrix V of the imageAAnd VB，VAAnd VBCan be expressed by the following formula:
wherein, S represents the sliding step length, and the sliding step length is preferably set to be 2 in the invention; w represents the length of the source image to be fused, and H represents the width of the source image to be fused. The sliding window technology in the invention refers to: in the image with the length W and the width H, the image is moved by a certain rule with the length WWidth isConverting the pixel block corresponding to the window into a column in a block pixel matrix representation, and then moving the window one step to the right or downwards until the processing of the whole image is completed; the sliding step length in the invention refers to: and the number of the pixels which slide through in one step is moved.
In a preferred embodiment, step 2 is to perform sparse decomposition after removing a direct current component from a pixel matrix of each image block to obtain a sparse decomposition coefficient of each image block; the step of removing the direct current component comprises calculating an average pixel value of each image block, and then subtracting the average value from the pixel value of each image block, wherein the direct current component refers to the average pixel value of each image block; the sparse decomposition means that an OMP algorithm following an adaptive termination strategy is adopted for sparse representation of column vectors corresponding to each image block pixel matrix, and the sparse representation process comprises the following sub-steps:
substep 1, initializing a system residual error threshold, a predetermined iterative convergence rate threshold and a system sparsity value to orderNamely, the system residual is equal to the initial value of the column vector, and an index set is set as an empty set when the iteration is not performed; in the present invention, in the case of the present invention,represents the system residual after the 0 th iteration of the mth block image block, vmA column vector representing the mth block image block; the systematic residual is consistent with the approximation error meaning and refers to the error residue of each iteration.
Substep 2, calculating the subscript lambda corresponding to the maximum value in the inner product between the system residual error of the ith iteration and the columns of the dictionary matrixi(ii) a In the invention, i is a natural number and is used for representing the number of iteration, and λ is a subscript corresponding to the maximum value in an inner product between a system residual error and a column of a dictionary matrix;
substep 3, using the subscript λiUpdate index set, Λi＝Λi-1∪{λiAnd recording a set D of reconstructed atoms in the selected dictionary matrixi，In the present invention, ΛiIndicating a new index set, i.e. the index set after the ith iteration，Λi-1For the index set after the i-1 th iteration, { λiIs a collection of dictionary indices, Di-1Is the set of atoms after the i-1 st iteration,is an atomic function corresponding to the dictionary index;
substep 4, obtaining the estimation variable of the sparse matrix coefficient after the ith iteration by the least square methodEstimated variables of the sparse matrix coefficientsI.e. using the selected set of atoms DiFor vmPerforming optimal representation to obtain coefficient values;
substep 5, updating the residual error after the ith iterationUpdating the iteration times i to i + 1; in the present invention, riRefers to the system residual after the ith iteration, ri-1The system residual error after the i-1 iteration is referred to;
substep 6, judging whether the system residual error, the iterative convergence rate of the system residual error and the iteration times meet the termination iteration condition of the iteration process, if so, terminating the iteration to obtain a sparse decomposition coefficient; if not, returning to the substep 2 and continuing to the subsequent substeps.
In a further preferred embodiment, in sub-step 6, the condition for determining whether the iteration is terminated is: the iteration times are greater than or equal to a preset system sparsity value;
alternatively, the iteration convergence rate of the system residual is less than or equal to a predetermined iteration convergence rate threshold, and the system residual norm is less than or equal to the square of a predetermined system residual threshold.
If at least one of the two conditions is satisfied, the band falling termination is judged to be possible, otherwise, the iteration is continued, and the substep 2 is returned until the condition of the iteration termination is satisfied.
In a further preferred embodiment, the predetermined system residual threshold is a product of the variance of the error noise, a constant C and the arithmetic square root of the column vector dimension.
The preset system sparsity value K is the percentage of the number of selected atoms in the total number of atoms and is used for measuring the sparsity of signal sparse decomposition.
The predetermined iteration convergence rate threshold is a constant within a given range of 0-1.
In a further preferred embodiment, the predetermined system residual threshold is
The predetermined system sparsity values are:
the predetermined iteration convergence rate threshold is cr0；
The method comprises the steps of representing a preset system residual error threshold, C representing a constant with the value of 1.15, sigma representing the variance of error noise, sigma representing 1, N representing the dimension of a column vector generated by scanning an image block according to a line, and the value of 64 in the method, K representing a preset system sparsity value, cr0The value is 0.15.
The iterative convergence rate is represented by cr, and cr ∈ (0,1), and is defined as follows:
in further preferredIn an embodiment, a complete iteration is as follows: by usingRepresenting image blocks to be sparsely decomposed byRepresenting a sparse matrix obtained after sparse decomposition, wherein M represents the total number of image blocks obtained by decomposing a source image, and
(1) initializing system residualsOrder toSetting an index set Λ to phi and an iteration number i to 0;
(2)λi＝argmaxj＝1...M|<ri-1,dj>l where djRepresents the jth column of the dictionary matrix;
(3) updating index set Λi＝Λi-1∪{λiRecording the set of reconstructed atoms in the selected dictionary matrix
(4) Calculating coefficients
(5) Updating the residual error of the current iteration, namely the ith iterationUpdating the iteration times i to i + 1;
(6) and (3) judging: if i is greater than or equal to k,
orAnd cr is less than or equal to cr0If so, stopping iteration to obtain a sparse representation matrix; otherwise, returning to the step (2) to continue to execute the iterative operation.
The main difference between the OMP algorithm following the self-adaptive termination strategy and the OMP algorithm commonly used in the field is focused on the difference of iterative convergence conditions, so that the sparse decomposition coefficients obtained according to the algorithm are different.
In a preferred embodiment, step 3 is to fuse the sparse decomposition coefficients of each image block at the corresponding position according to the energy maximization principle to obtain a fused image block coefficient matrix of the image block at the position; the energy maximization principle is that the maximum absolute value of the sparse decomposition coefficients of the image blocks at the corresponding positions of the two source images is used as the sparse decomposition coefficient of the corresponding position of the fused image block, and the pixel corresponding to the coefficient with the maximum absolute value at the position contributes most to the fused image. The corresponding positions in the invention refer to areas with the same relative positions on the two source images to be fused, and are further understood as two image blocks divided at the same area on the two source images. The image content displayed on the two source images at two positions, which are mutually referred to as corresponding positions, is the same.
In a more preferred embodiment, the sparse decomposition coefficient is represented by XAAnd XBIs shown by
Wherein, XFAnd expressing the sparse decomposition coefficient of the corresponding position of the fused image block.
In a preferred embodiment, step 4 is to reconstruct the fused image block coefficient matrix of the image block at the position, and add the reconstructed value of the direct current component to obtain a fused image block pixel matrix; the reconstruction means reconstructing the spatial pixel representation of each image block, that is, restoring the one-dimensional representation of the image block to a two-dimensional representation, and the reconstruction method includes: and recovering the spatial pixel representation of the image block from the fused sparse coefficient values, and calculating the average value of the pixel values at the same position of the overlapped part between the adjacent image blocks as the calculation result of the final pixel matrix. In the invention, the image block is expressed as a certain column vector in the pixel matrix, and each column of the fused image block coefficient matrix is the decomposed sparse coefficient value.
In a further preferred embodiment, the fusion coefficient matrix is subjected to an inverse operation of sparse decomposition to obtain a pixel matrix of the fusion image block, and an operation formula thereof is as follows:
VF＝DXF
wherein, VFAnd D represents a dictionary adopted by sparse decomposition.
In a preferred embodiment, step 5 is to restore the fused image block pixel matrices of the image blocks at all positions to obtain a final fused image; the restoring of the fused image block pixel matrix comprises: restoring a pixel matrix of the overlapped part of the fused image block and a pixel matrix of the direct-current component part; and for the pixel matrix of the direct current component part, the direct current component pixel matrix of the fused image is obtained by carrying out weighting operation on the direct current component of the source image block.
In a further preferred embodiment, during the weighting operation, two source images are provided as a and B, the image block pixel matrix of which is denoted VAAnd VBThe DC components of the jth block of A and B are represented byAndindicating the DC component weighting factor omega of image AAComprises the following steps:
similarly, the DC component weighting factor ω of the image BBComprises the following steps:
fusing the DC component of the imageComprises the following steps:
wherein, N refers to the dimension of a column vector generated by scanning an image block according to a line; i. j is a natural number, i refers to a row of the pixel block matrix and j refers to a column of the pixel block matrix.
In a preferred embodiment, as shown in fig. 2 and 3, the dictionary used in the present invention is an overcomplete dictionary, which is a fixed training dictionary, or a hybrid dictionary obtained by mixing a wavelet domain dictionary and a DCT redundant dictionary, wherein the dictionary can be D ∈ RN×KIt is shown that in the present invention, the fixed training dictionary is obtained by training the K-SVD algorithm as shown in fig. 2; the hybrid dictionary is shown in fig. 3 and is derived from a hybrid of a wavelet domain dictionary (bior6.8) and a DCT redundant dictionary.
The image fusion method provided by the invention breaks through the limitation of a single iteration termination judgment condition of the traditional algorithm, has two conditions for judging the termination of iteration, and automatically selects a proper judgment condition according to the characteristics of different image blocks, so that the method provided by the invention has wider application range, can process images with rich details and smooth images, reduces the image fusion time, improves the fusion efficiency, has higher image fusion quality, and obtains better fusion effect.
Specific examples are as follows:
fusing the two source images in fig. 4 and fig. 5 to obtain a clear fused image, wherein the image given in fig. 4 is a right focused image, and the image given in fig. 5 is a left focused image; carrying out image fusion on the two images by using the same computer;
experimental example: the fusion method provided by the invention is used for image fusion, particularly, the iteration termination strategy provided by the invention is used for comprehensively controlling the system residual error, the system residual error convergence rate and the system sparsity, a picture shown in figure 6 is obtained within 114 seconds, and the quality indexes of the picture are shown in the following table (one):
watch 1
Comparative example one: the fusion method provided by the invention is used for fusion of fusion images, and is different from an experimental example in that only system residual is adopted to control the termination of iteration in the iteration process of an OMP algorithm, namely in the comparative example, the condition of termination of iteration is that the norm of the system residual is less than or equal to the square of a system residual threshold, and the system residual threshold is consistent with the system residual threshold in the experimental example; the picture shown in fig. 7 is obtained after 132 seconds, and the quality indexes of the picture are shown in the following table (two):
watch 2
Comparative example two: the fusion method provided by the invention is used for fusion of fusion images, and is different from an experimental example in that in the iteration process of an OMP algorithm, only a sparsity value is used for controlling the iteration termination, namely in the comparative example, the iteration termination condition is that the iteration frequency is more than or equal to a preset system sparsity value, and the sparsity value is consistent with that in the experimental example; it takes 199 seconds to obtain a picture as shown in fig. 8, and the quality index of the picture is shown in the following table (three):
watch (III)
The QABF measures the information quantity transferred from the two source images to the fusion image, the LABF measures the information lost in the fusion process, the value reflects the quantity of the lost information, the NABF measures the artificial distortion effect generated in the fusion process, and the smaller the value is, the smaller the distortion effect is.
According to the experimental comparison, the image fusion is shortest in time according to the method provided by the invention, which indicates that the efficiency of image fusion according to the method provided by the invention is highest, and then the quality index comparison of each image obtained after fusion indicates that the average gradient value of the image obtained by the fusion method provided by the invention is higher than that obtained in the comparative example, which indicates that the definition of the fused image obtained by the method provided by the invention is higher; and as can be seen from the comparison of image quality in the table, the image space frequency value obtained by the fusion method provided by the invention is lower than that obtained in the comparative example, and the image NABF value obtained by the fusion method provided by the invention is lower than that obtained in the comparative example.
The present invention has been described above in connection with preferred embodiments, but these embodiments are merely exemplary and merely illustrative. On the basis of the above, the invention can be subjected to various substitutions and modifications, and the substitutions and the modifications are all within the protection scope of the invention.
Claims (4)
1. A multi-sensor image fusion method based on block self-adaptive feature tracking is characterized in that in the process of carrying out sparse decomposition on an image block pixel matrix, an iteration process is terminated under the following conditions:
the iterative convergence rate of the system residual is less than or equal to a predetermined iterative convergence rate threshold and the system residual norm is less than or equal to the square of a predetermined system residual threshold;
the method comprises the following steps:
step 1, dividing two source images to be fused into image blocks, wherein the size of each image block is consistent, and any two adjacent image blocks have overlapping parts; expressing the pixels of each image block into a column vector to obtain a pixel matrix of each image block, namely obtaining the block pixel matrix expression of the source image;
step 2, removing direct current components from the pixel matrix of each image block, and then performing sparse decomposition to obtain a sparse decomposition coefficient of each image block;
step 3, fusing the sparse decomposition coefficients of each image block at the corresponding position according to the energy maximization principle to obtain a fused image block coefficient matrix of the image block at the position;
step 4, reconstructing the fused image block coefficient matrix of the image block at the position, and adding the reconstructed value of the direct current component to obtain a fused image block pixel matrix of the image block at the position;
step 5, restoring the fused image block pixel matrixes of the image blocks at all positions to obtain a final fused image;
the two source images to be fused are two images obtained at different focal length positions aiming at the same scene, or two spectral images under the spectral conditions of different illumination light wavelengths aiming at the same scene;
the step of removing the direct current component comprises calculating an average pixel value of each image block, and then subtracting the average value from the pixel value of each image block, namely the direct current component is the average pixel value of each image block;
the energy maximum principle is that the maximum absolute value of the sparse decomposition coefficient of the image blocks at the corresponding positions of the two source images is used as the sparse decomposition coefficient of the corresponding positions of the fused image blocks;
reconstructing is to reconstruct the spatial pixel representation of each image block, i.e. to restore the one-dimensional representation of the image block to a two-dimensional representation;
the predetermined system residual threshold is: the product of the variance of the error noise, the constant C and the arithmetic square root of the column vector dimension;
the predetermined system sparsity values are: the ratio of the dimension of a column vector generated by scanning the image block according to a line to the numerical value 3;
the preset iteration convergence rate threshold value is a given constant in a range of 0-1;
the predetermined system residual threshold is
The predetermined system sparsity values are:
the predetermined iteration convergence rate threshold is cr0；
The method comprises the following steps of A, representing a residual error threshold of a preset system, C representing a constant with the value of 1.15, sigma representing the variance of error noise, sigma representing the value of 1, and N representing the dimension of a column vector generated by scanning an image block according to a line; k represents a predetermined system sparsity value, cr0The value is 0.15;
sparse decomposition of the image block pixel matrix is performed by using an OMP algorithm following an adaptive termination strategy, comprising the following sub-steps:
substep 1, initializing a system residual error threshold, a predetermined iterative convergence rate threshold and a system sparsity value to orderNamely, the system residual is equal to the initial value of the column vector, and an index set is set as an empty set when the iteration is not performed; wherein,represents the system residual after the 0 th iteration of the mth block image block, vmA column vector representing the mth block image block;
substep 2, calculating the subscript lambda corresponding to the maximum value in the inner product between the system residual error of the ith iteration and the columns of the dictionary matrixi(ii) a Wherein i is a natural number for representing the number of iterations;
substep 3, using the subscript λiUpdate index set, Λi＝Λi-1∪{λiAnd recording a set D of reconstructed atoms in the selected dictionary matrixi，Wherein, ΛiIndicating a new index set, Λ, i.e. the index set after the ith iterationi-1For the index set after the i-1 th iteration, { λiIs a collection of dictionary indices, Di-1Is the set of atoms after the i-1 st iteration,is an atomic function corresponding to the dictionary index;
substep 4, obtaining the estimation variable of the sparse matrix coefficient after the ith iteration by the least square methodEstimated variables of the sparse matrix coefficientsI.e. using the selected set of atoms DiFor vmPerforming optimal representation to obtain coefficient values;
substep 5, updating the residual error of the ith iterationUpdating the iteration times i to i + 1; wherein r isiRefers to the system residual after the ith iteration, ri-1The system residual error after the i-1 iteration is referred to;
substep 6, judging whether the system residual error, the iterative convergence rate of the system residual error and the iteration times meet the termination iteration condition of the iteration process, if so, terminating the iteration to obtain a sparse decomposition coefficient; if not, returning to the substep 2 and continuing to perform the subsequent substeps;
two adjacent image blocks with overlapped parts are obtained by a sliding window technology, two pixel points are translated each time, namely the sliding step length is 2.
2. The multi-sensor image fusion method based on block-adaptive feature tracking according to claim 1, characterized in that the reconstruction method is as follows: and recovering the spatial pixel representation of the image block from the fused sparse coefficient values, and calculating the average value of the pixel values at the same position of the overlapped part between the adjacent image blocks as the calculation result of the final pixel matrix.
3. The multi-sensor image fusion method based on block-adaptive feature tracking of claim 1, wherein restoring the pixel matrix of the fused image block comprises restoring a pixel matrix of an overlapping portion of the fused image block and a pixel matrix of a direct current component portion; and for the pixel matrix of the direct current component part, the direct current component pixel matrix of the fused image is obtained by carrying out weighting operation on the direct current component of the source image block.
4. The multi-sensor image fusion method based on block-adaptive feature tracking of claim 1, wherein the dictionary used in the method is an overcomplete dictionary, the dictionary is a fixed training dictionary,
or the mixed dictionary is obtained by mixing the wavelet domain dictionary and the DCT redundant dictionary.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410271445.3A CN104008533B (en) | 2014-06-17 | 2014-06-17 | Multisensor Image Fusion Scheme based on block adaptive signature tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410271445.3A CN104008533B (en) | 2014-06-17 | 2014-06-17 | Multisensor Image Fusion Scheme based on block adaptive signature tracking |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104008533A CN104008533A (en) | 2014-08-27 |
| CN104008533B true CN104008533B (en) | 2017-09-29 |
Family
ID=51369174
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410271445.3A Expired - Fee Related CN104008533B (en) | 2014-06-17 | 2014-06-17 | Multisensor Image Fusion Scheme based on block adaptive signature tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104008533B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599259B (en) * | 2015-01-30 | 2017-04-05 | 华北电力大学 | Multi-modality image fusion method based on the orthogonal matched jamming of polyatom stage by stage |
| CN104867125B (en) * | 2015-06-04 | 2018-03-02 | 北京京东尚科信息技术有限公司 | Obtain the method and device of image |
| CN106612505A (en) * | 2015-10-23 | 2017-05-03 | 国网智能电网研究院 | Wireless sensor safety communication and anti-leakage positioning method based on region division |
| CN105825239A (en) * | 2016-03-31 | 2016-08-03 | 中国科学院电子学研究所 | Multi-sensor track fusion method based on sparse expression |
| CN106447640B (en) * | 2016-08-26 | 2019-07-16 | 西安电子科技大学 | Multi-focus image fusing method and device based on dictionary learning, rotation guiding filtering |
| CN106901941A (en) * | 2017-02-21 | 2017-06-30 | 哈尔滨医科大学 | A kind of joint of vertebral column depressurized system |
| CN108663677A (en) | 2018-03-29 | 2018-10-16 | 上海智瞳通科技有限公司 | A kind of method that multisensor depth integration improves target detection capabilities |
| CN109934794B (en) * | 2019-02-20 | 2020-10-27 | 常熟理工学院 | Multi-focus image fusion method based on significant sparse representation and neighborhood information |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2935822B1 (en) * | 2008-09-05 | 2010-09-17 | Commissariat Energie Atomique | METHOD FOR RECOGNIZING FORMS AND SYSTEM IMPLEMENTING THE METHOD |
-
2014
- 2014-06-17 CN CN201410271445.3A patent/CN104008533B/en not_active Expired - Fee Related
Non-Patent Citations (3)
| Title |
|---|
| "图像稀疏编码算法及应用研究";严春满;《中国博士学位论文全文数据库 信息科技辑》;20131115(第11期);论文第71-77页 * |
| "基于正交匹配跟踪的多传感器图像融合";黄影等;《图像图形技术研究与应用》;20101231;论文第99-100页 * |
| "基于残差比阈值的迭代终止条件匹配追踪稀疏分解方法";梁巍等;《上海交通大学学报》;20100228;第44卷(第22期);论文第172-173页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104008533A (en) | 2014-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104008533B (en) | Multisensor Image Fusion Scheme based on block adaptive signature tracking | |
| Zhao et al. | Discrete cosine transform network for guided depth map super-resolution | |
| Zhao et al. | Multisensor image fusion and enhancement in spectral total variation domain | |
| KR102215805B1 (en) | Method and system to obtain material separation image using deep learning technology | |
| CN104008538B (en) | Based on single image super-resolution method | |
| JP6415781B2 (en) | Method and system for fusing detected measurements | |
| Matakos et al. | Accelerated edge-preserving image restoration without boundary artifacts | |
| CN108399611B (en) | Multi-focus image fusion method based on gradient regularization | |
| CN107730482B (en) | Sparse fusion method based on regional energy and variance | |
| CN107194912B (en) | Brain CT/MR image fusion method based on sparse representation and improved coupled dictionary learning | |
| CN109636722B (en) | Method for reconstructing super-resolution of online dictionary learning based on sparse representation | |
| CN107301630B (en) | CS-MRI image reconstruction method based on ordering structure group non-convex constraint | |
| CN104599259B (en) | Multi-modality image fusion method based on the orthogonal matched jamming of polyatom stage by stage | |
| CN114862732B (en) | Synthetic aperture imaging method integrating event camera and traditional optical camera | |
| CN110060225B (en) | Medical image fusion method based on rapid finite shear wave transformation and sparse representation | |
| Qu et al. | TransFuse: A unified transformer-based image fusion framework using self-supervised learning | |
| CN107689038A (en) | A kind of image interfusion method based on rarefaction representation and circulation guiding filtering | |
| CN113298147A (en) | Image fusion method and device based on regional energy and intuitionistic fuzzy set | |
| CN107301631B (en) | SAR image speckle reduction method based on non-convex weighted sparse constraint | |
| Yang et al. | Reconstruction of structurally-incomplete matrices with reweighted low-rank and sparsity priors | |
| CN109741258B (en) | Image super-resolution method based on reconstruction | |
| Zhouping | Fusion algorithm of optical images and SAR with SVT and sparse representation | |
| Tang et al. | MFAGAN: A multiscale feature-attention generative adversarial network for infrared and visible image fusion | |
| Marivani et al. | Designing CNNs for multimodal image restoration and fusion via unfolding the method of multipliers | |
| Singh et al. | HoEnTOA: Holoentropy and Taylor Assisted Optimization based Novel Image Quality Enhancement Algorithm for Multi-Focus Image Fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170929 Termination date: 20190617 |