CN101527039A - Automatic image registration and rapid super-resolution fusion method based on edge feature - Google Patents

Abstract

The invention discloses an automatic image registration and rapid super-resolution fusion method based on edge feature, and relates to the field of image processing. The method is characterized in that the method comprises the following processing steps that: (1) a computer reads two kinds of target images to be processed to a buffer area from interfaces such as a USB, and the like, wherein one kind of images are low-resolution multispectral images, and the other kind of images are high-resolution grey images; (2) wavelet transformation and a least square fitting model are adopted to acquire registration parameters of the images; (3) the registered multispectral images are used to acquire strength component images through linear IHS transformation, and then histograms of the high-resolution grey images are matched to strength components; (4) the high-resolution grey images matched with the histograms are used to substitute the strength components, and the linear IHS inverse transformation is used to realize the super-resolution fusion of the images; and (5) super-resolution multispectral color images are output. The method has lower complexity and execution speed than the prior method, occupies less resources of a computer system, and is convenient to execute on embedded equipment.

Description

Automatic image registration and rapid super-resolution fusion method based on edge feature

One, technical field

The present invention relates to the Flame Image Process theory, realized a kind of automatic image registration and rapid super-resolution fusion method based on edge feature.

Two, background technology

In video monitoring, resource exploration, the online detection of surface of the work quality and military surveillance etc. were used, high-definition picture often can provide needed material particular information.In order to obtain high-resolution image, people at first expect it being the performance that improves hardware device, but can strengthen manufacturing cost, and existing manufacturing industry technology substantially can accomplish to bring into play the greatest physical characteristic of current material, the yardstick that resolution improves is also very limited for this reason.The method that adopts the raising hardware performance to obtain high-definition picture in sum can not satisfy the needs of future development.

At present, a kind of method that adopts signal processing technology to obtain high-resolution image has become the focus of this area research.The main thought of these class methods is to utilize original imaging device to adopt signal processing technology that the low-resolution image that collects is carried out software processes to obtain the higher image of resolution.

The existing method that adopts software algorithm to obtain high-definition picture all is to rely on the high performance universal PC to finish the registration and the reconstruction operation of image by adding certain handling procedure.Because in image reconstruction, what participate in computing is not piece image, and the calculated amount of reconstruction algorithm is very big, therefore, quick, real-time becomes a problem demanding prompt solution.

Different with resolution raising method commonly used at present, the present invention adopt fast linear IHS conversion joint and fuzzy integral to combine to realize image fast, super-resolution rebuilding in real time, and fusion method of the present invention is different and other fusion methods, can regulate spectral characteristic and spatial detail characteristic that fog-density is regulated and control fused images easily.The present invention has adopted fast linear IHS fusion method can accelerate image fusion speed in addition.Rebuilding pretreatment stage, invented a kind of high performance small echo in conjunction with the autoregistration that least square combines, make the precision of image registration greatly improve.Restructing algorithm is the higher occasion (as the surface of the work Quality Detection) of requirement of real time preferably, is convenient to realize this method on the low embedded platform of cost.

" integrated supervision monitoring technique and system integration theoretical research and application thereof " that the applicant presides over and bears calendar year 2001 obtains Jiangsu Province's scientific-technical progress second prize; " computational intelligence and the Image Information Processing research and the application system thereof " of presiding over and bearing in 2005 obtains Chinese Institute of Electronics's electronic information science and technology third prize; Publish 2 ones of monographs, calendar year 2001 " Intelligent Information Processing of digital picture " is published by National Defense Industry Press, solely work; 2004 " multimedia monitoring monitoring technique and systems " are published by National Defense Industry Press; 2006 " information and system integration technology and application " published by Science Press, and by years of researches and practice, the applicant is familiar with and grasps the research difficult point and the emphasis in digital picture Intelligent treatment and surveillance monitor field.

The applicant pays attention to research work theoretical and method especially in conjunction with doctoral candidate's cultivation.Carried out multi-source, multi-scale information blending theory, method in recent years, expanded the application of information processing based on 3S (GPS, GIS, RS) technology.The main achievement that obtains has " integrated supervision monitoring technique and system integration research and application thereof ", and 2001 obtain Jiangsu Province's scientific-technical progress second prize, national inventing patent: multi-source monitoring data information fusion disposal route.Researchs such as industrial video image compression encoding technology under the extremely low code check of having carried out " " the monitoring image preconditioning technique under the man-made noise background ", " ", " multimedia monitoring monitoring and Digital Video System integrated technology "; proposed the intelligent new method of more than ten kind of image processing, some is applied in practice.Calendar year 2001, " computational intelligence and Image Information Processing research and application system thereof " project was by the evaluation of Jiangsu Province Science and Technology Department.The applicant proposed the super-resolution fusion treatment algorithm of the multi-source based on fuzzy integral, multiple dimensioned remote sensing images, based on the super-resolution fusion treatment of the multi-source of PCA combined with wavelet transformed, multiple dimensioned remote sensing images, the wavelet transformation remote sensing image fusion algorithm and the improved remote sensing image fusion method of adaptive fuzzy density assignment based on wavelet transformation.Above-mentioned research is respectively to carry out under the subsidy of state natural sciences fund, Jiangsu Province's natural science fund, national 863 projects, publish monograph " Intelligent Information Processing of digital picture ", the achievement that has is published on the domestic and international academic journals, and the part application of result is in some horizontal problems.

Three, summary of the invention

The present invention is by having proposed a kind of automatic image registration and rapid super-resolution fusion method based on edge feature.This method has overcome that execution speed in the image co-registration in the past is slow, the shortcomings such as complexity height of algorithm.According to image information characteristics and application characteristic, the present invention proposes and realize that multispectral coloured image of one tunnel low resolution and one tunnel high resolving power gray level image by autoregistration and rapid super-resolution fusion method, obtain high-resolution colour picture.

Technical thought of the present invention is characterized as:

Note low resolution coloured image is M, and it has 3 wave bands, is respectively M ₁, M ₂, M ₃, note high resolving power gray level image is P.Adopt the super-resolution blending algorithm, make up the spectral information of M and the spatial detail information of P, can obtain the coloured image F of super-resolution.Concrete algorithm can be divided into three parts:

1. the registration of image

Adopt wavelet transformation to ask for the registration parameter of M and P in conjunction with the model of least square fitting.Wherein wavelet transformation is used for extracting the edge of image unique point, re-use affined transformation and set up the corresponding relation of M edge of image unique point and P edge of image unique point, ask for affine transformation parameter according to least square fitting at last, thereby to the P image, the image registration flow process is seen Fig. 3 with M image registration.

1) wavelet transformation extracts edge of image

The algorithm that utilizes wavelet transformation to extract the image border is summarized as follows: at first utilize dyadic wavelet transform that picture breakdown is the J layer.(high fdrequency component of decomposition layer of 1≤j≤J) is found the solution the very big W of little mode to be directed to j _j, and set a threshold value T, if W _j＞T then thinks marginal point.

Utilize above-mentioned algorithm, then to image M ₁(in this patent with M ₁Be example, M ₂, M ₃Similar) and P can obtain the edge feature point set respectively

( _j1(x′，y′)， _j2(x′，y′)，…， _jn(x′，y′))、( _j1(x，y)， _j2(x，y)，…， _jn(x，y)) ①

In the formula _Jk(x ', y ') presentation video M ₁Coordinate at k marginal point of wavelet decomposition j layer

_Jk(x, y) presentation video P is at the coordinate of k marginal point of wavelet decomposition j layer

k＝1，2，…，n

Total n edge feature point in the feature point set.

2) affined transformation

Because the image registration that this patent is considered can be thought rigid body translation, so can adopt affined transformation to come relation between the characteristic feature point, affined transformation as shown in the formula:

$\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{x}^{\′}\\ y^{\′}\end{array}\right)+\left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)$ ②

(reference picture in this patent refers to the P image for x, the y) coordinate of expression reference picture in the formula.

The coordinate of (x ', y ') expression image subject to registration, the image subject to registration in this patent refers to M ₁Image. $\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)$ The rotation of presentation video and convergent-divergent

(b ₁, b ₂) presentation video is at the translational movement of x, y direction.

3) least square fitting

Note A=(a ₁₁a ₁₂b ₁)

B＝(b ₂₁?b ₂₂?b ₂)

$Q=\left(\begin{array}{cccc}{}_{j1}\left(x^{\′}\right)& {}_{j2}\left(x^{\′}\right)& \·\·\·& {}_{\mathrm{jn}}\left(x^{\′}\right)\\ {}_{j1}\left(y^{\′}\right)& {}_{j2}\left(y^{\′}\right)& \·\·\·& {}_{\mathrm{jn}}\left(y^{\′}\right)\\ 1& 1& \·\·\·& 1\end{array}\right)$ ③

M _x＝( _j1(x)， _j2(x)，…， _jn(x))

M _y＝( _j1(y)， _j2(y)，…， _jn(y))

It is as follows to set up the least square expression formula thus:

AQ＝Mx ④

BQ＝My

Can solve A, B by above-mentioned least squares equation

In like manner can obtain 1,2 ..., each layer affine transformation parameter of J layer, final affine transformation parameter are the average of each layer affine transformation parameter.

2. the histogram of image mates

Note is M through the multispectral image of registration _R, it has three wave bands and is respectively M _R1, M _R2, M _R3

Suppose that the IHS conversion is linear transformation, then to image M _RCan carry out following IHS conversion:

$\left[\begin{array}{c}I\\ H\\ S\end{array}\right]=\left[\begin{array}{ccc}1/3& 1/3& 1/3\\ -\sqrt{2}/6& -\sqrt{2}/6& 2\sqrt{2}/6\\ 1/\sqrt{2}& -1/\sqrt{2}& 0\end{array}\right]\·\left[\begin{array}{c}{M}_{R1}\\ M_{R2}\\ M_{R3}\end{array}\right]$ ⑤

The histogram of image P is matched the strength component image I, and the image after the note process histogram coupling is respectively P ₁, IHS shift process such as Fig. 4.

3. the super-resolution of image merges

The super-resolution of image merges the linear IHS inverse transformation of employing, and this considers that mainly conventional IHS transform operation amount of linear IHS conversion is little, speed is fast.

Concrete super-resolution fusion process is: with P ₁Carry out fusion based on fuzzy integral with I, Fig. 5 is a fuzzy integral image co-registration process flow diagram, and concrete fusion process is as follows:

If I image, P image maximal value are respectively M ₁And M ₂, note M=max (M ₁, M ₂).Regard the pixel value of I image, P image as feature s, then (m, the belief function of n) locating is designated as h (s respectively at location of pixels _I, m, n), h (s _P, m, n).Make h (s _I, m, n)=I (m, n)/M, h (s _P, m, n)=P (m, n)/M

According to the constraint condition of the monotonicity of fuzzy integral, location of pixels is that (m n) locates, note s={s ₁(m, n), s ₂(m, n) }, { s wherein ₁(m, n), s ₂(m, n) } be defined as follows:

$\left\{\begin{array}{cc}{s}_1(m,n)=I(m,n),s_2(m,n)=P(m,n)& \mathrm{ifh}(s_I;m,n)\≤h(s_P;m,n)\\ s_1(m,n)=P(m,n),s_2(m,n)=I(m,n)& \mathrm{ifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$ ⑥

Therefore, fuzzy mearue

g(A ₁)＝g({s ₁(m，n)，s ₂(m，n)})＝1 ⑦

g(A ₂)＝g({s ₂(m，n)}) ⑧

As seen from the above analysis, as long as determine g ({ s ₂(m, n) }) just can obtain the pixel value after the fusion, and g ({ s ₂(m, n) }) is exactly image I or image P in that (m n) locates the fog-density of pixel value.In order to express easily, remember g ({ s herein ₂(m, n) }) be g (m, n).

The result of note fuzzy integral is that (m, n), the pixel value after the fusion is P ' (2 to FI ^jM, n)

$P^{\′}(m,n)=\left\{\begin{array}{cc}\mathrm{FI}(m,n)\×M& \mathrm{ifh}(s_I;m,n)\≤h(s_P;m,n)\\ \mathrm{FI}(m,n)\×M& \mathrm{ifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$

$=\left\{\begin{array}{cc}I(m,n)+(P(m,n)-I(m,n))\×g(m,n)& \mathrm{ifh}(s_I;m,n)\≤h(s_P;m,n)\\ P(m,n)+(I(m,n)-P(m,n))\×g(m,n)& \mathrm{ifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$ ⑨

G (m wherein, n) size can be decided according to the attribute specification to fused images, if g (m about image P, n) more and more littler, then to the g of I image (m, n) increasing, then the spectral information of fused images is become better and better, this shows that (m n) can be so that the spectrum of fused images and spatial detail information be best in the intermediate value scope for g.

P ' image is replaced the strength component image I, keep image M _RH component image, S component image after the IHS conversion are constant, utilize the inverse transformation formula (seeing 10. formula) of linear IHS conversion, and the super-resolution coloured image F that can get finally is:

$\left[\begin{array}{c}F\left(R\right)\\ F\left(G\right)\\ F\left(B\right)\end{array}\right]=\left[\begin{array}{ccc}1& -1/\sqrt{2}& 1/\sqrt{2}\\ 1& -1/\sqrt{2}& -1/\sqrt{2}\\ 1& \sqrt{2}& 0\end{array}\right]\·\begin{array}{}\left[\begin{array}{c}{P}_1\\ H\\ S\end{array}\right]\end{array}$ ⑩

IHS inverse transformation flow process as shown in Figure 6.

Four, description of drawings

Below in conjunction with accompanying drawing, describe embodiments of the invention in detail, wherein:

Accompanying drawing 1 is a workflow diagram of the present invention;

Accompanying drawing 2 is method flow diagrams of the present invention;

Accompanying drawing 3 is image registration process flow diagrams based on maximum mutual information of the present invention;

Accompanying drawing 4 is linear IHS direct transform process flow diagrams of multispectral image of the present invention;

Accompanying drawing 5 is fuzzy integral image co-registration process flow diagrams of the present invention;

Accompanying drawing 6 is linear HIS inverse transformation process flow diagrams of multispectral image of the present invention.

Five, embodiment

As shown in Figure 1, computing machine reads in multispectral coloured image of low resolution and high resolving power gray level image simultaneously from USB, interface such as infrared, and the coloured image of note low resolution is M, and it has 3 wave bands, is respectively M ₁, M ₂, M ₃Note high resolving power gray level image is P, resulting image M and P temporary storage buffer region, obtain the registration parameter of image M and P earlier, pass through the coloured image F that HIS conversion and IHS inverse transformation obtain super-resolution then successively, be stored in buffer zone, or directly be output in monitor, or by the network interface remote transmission.

As shown in Figure 2, it is method flow diagram of the present invention, algorithm is carried out, and at first gray level image P is registrated to I component, afterwards the gray level image behind the histogram registration and I component is done fusion based on fuzzy integral and is obtained new strength component degree image P ' as one of input quantity of IHS inverse transformation; Extract H component and S component input quantity, by P ' component, H component and S component being carried out directly obtain colored super-resolution image F after the IHS inverse transformation as the IHS inverse transformation.Described registration adopts wavelet transformation to ask for the registration parameter of M and P in conjunction with the model of least square fitting earlier.Wherein wavelet transformation is used for extracting the edge of image unique point, re-use affined transformation and set up the corresponding relation of M edge of image unique point and P edge of image unique point, ask for affine transformation parameter according to least square fitting at last, thereby the P image is arrived in M image registration.

Claims (2)

1. automatic image registration and rapid super-resolution fusion method based on an edge feature, be to finish two pending class images by all kinds of picture pick-up devices, one class is the multispectral coloured image of low resolution, another kind of is high-resolution gray level image, target image is digitized as is convenient to pass through USB, the infrared interface input Computer Processing that waits, result images after the processing exports impact damper to, can directly store in this locality, or directly output shows, or carry out remote transmission by the network storage equipment, it is characterized in that it also wraps expands following step: the registration parameter that 1) adopts wavelet transformation to ask for M and P in conjunction with the model of least square fitting is realized the registration of image.Wherein wavelet transformation is used for extracting the edge of image unique point, re-use affined transformation and set up the corresponding relation of M edge of image unique point and P edge of image unique point, ask for affine transformation parameter according to least square fitting at last, thereby the P image arrived in M image registration, it is characterized by:

A) wavelet transformation extracts edge of image

The algorithm that utilizes wavelet transformation to extract the image border is summarized as follows: at first utilize dyadic wavelet transform that picture breakdown is the J layer.(high fdrequency component of decomposition layer of 1≤j≤J) is found the solution the very big W of little mode to be directed to j _j, and set a threshold value T, if W _j＞T then thinks marginal point;

B) affined transformation

The image registration of considering can be thought rigid body translation, so can adopt affined transformation to come relation between the characteristic feature point, affined transformation as shown in the formula:

$\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{x}^{\′}\\ y^{\′}\end{array}\right)+\left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)$ ①

In the formula (x, the y) coordinate of expression reference picture,

The coordinate of (x ', y ') expression image subject to registration,

$\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)$ The rotation of presentation video and convergent-divergent

(b ₁, b ₂) presentation video is at the translational movement of x, y direction;

C) least square fitting

Note A=(a ₁₁a ₁₂b ₁)

B＝(b ₂₁?b ₂₂?b ₂)

$Q=\left(\begin{array}{cccc}{}_{j1}\left(x^{\′}\right)& {}_{j2}\left(x^{\′}\right)& ...& {}_{\mathrm{jn}}\left(x^{\′}\right)\\ {}_{j1}\left(y^{\′}\right)& {}_{j2}\left(y^{\′}\right)& ...& {}_{\mathrm{jn}}\left(y^{\′}\right)\\ 1& 1& ...& 1\end{array}\right)$ ②

M _x＝( _j1(x)， _j2(x)，…， _jn(x))

M _y＝( _j1(y)， _j2(y)，…， _jn(y))

It is as follows to set up the least square expression formula thus:

AQ＝Mx ③

BQ＝My ④

Can solve A, B by above-mentioned least squares equation;

2) histogram of image coupling

Note is M through the multispectral image of registration _R, it has three wave bands and is respectively M _R1, M _R2, M _R3

The histogram of image P is matched the strength component image I, and the image after the note process histogram coupling is respectively P ₁

3) super-resolution of image merges

The super-resolution of image merges the linear IHS inverse transformation of employing, and this considers that mainly conventional IHS transform operation amount of linear IHS conversion is little, speed is fast.

Concrete super-resolution fusion process is:

If I image, P image maximal value are respectively M ₁And M ₂, note M=max (M ₁, M ₂).Regard the pixel value of I image, P image as feature s, then (m, the belief function of n) locating is designated as h (s respectively at location of pixels _I, m, n), h (s _I, m, n).Make h (s _I, m, n)=I (m, n)/M, h (s _P, m, n)=P (m, n)/M

According to the constraint condition of the monotonicity of fuzzy integral, location of pixels is that (m n) locates, note s={s ₁(m, n), s ₂(m, n) }, { s wherein ₁(m, n), s ₂(m, n) } be defined as follows:

$\left\{\begin{array}{c}{s}_1(m,n)=I(m,n),s_2(m,n)=P(m,n)\mathrm{ifh}(s_I;m,n)\≤h(s_P;m,n)\\ s_1(m,n)=P(m,n),s_2(m,n)=I(m,n)\mathrm{ifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$ ⑤

Therefore, fuzzy mearue

g(A ₁)＝g({s ₁(m，n)，s ₂(m，n)})＝1 ⑥

g(A ₂)＝g({s ₂(m，n)}) ⑦

As long as determine g ({ s ₂(m, n) }) just can obtain the pixel value after the fusion, and g ({ s ₂(m, n) }) is exactly image I or image P in that (m n) locates the fog-density of pixel value.In order to express easily, remember g ({ (s herein ₂(m, n) }) be g (m, n).

The result of note fuzzy integral is that (m, n), the pixel value after the fusion is P ' (2 to FI ^jM, n)

$P^{\′}(m,n)=\left\{\begin{array}{c}\mathrm{FI}(m,n)\×\mathrm{Mifh}(s_I;m,n)\≤h(s_P;m,n)\\ \mathrm{FI}(m,n)\×\mathrm{Mifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$

$=\left\{\begin{array}{c}I(m,n)+(P(m,n)-I(m,n))\×g(m,n)\mathrm{ifh}(s_I;m,n)\≤h(s_P;m,n)\\ P(m,n)+(I(m,n)-P(m,n))\×g(m,n)\mathrm{ifh}(s_I;m,n)>h(s_P;m,n)\end{array}\right.$ ⑧

G (m wherein, n) size can be decided according to the attribute specification to fused images, if g (m about image P, n) more and more littler, then to the g of I image (m, n) increasing, then the spectral information of fused images is become better and better, this shows that (m n) can be so that the spectrum of fused images and spatial detail information be best in the intermediate value scope for g;

2. according to automatic image registration and rapid super-resolution fusion method under the claim 1, it is characterized in that the formula that adopts the IHS inverse transformation to realize final super-resolution coloured image is based on edge feature:

$\left[\begin{array}{c}F\left(R\right)\\ F\left(G\right)\\ F\left(B\right)\end{array}\right]=\left[\begin{array}{ccc}1& -1/\sqrt{2}& 1/\sqrt{2}\\ 1& -1/\sqrt{2}& -1/\sqrt{2}\\ 1& \sqrt{2}& 0\end{array}\right]\·\left[\begin{array}{c}{P}_1\\ H\\ S\end{array}\right]$ ⑨

Images