CN105488778A - Multi-viewpoint image fusion method based on block SPCA - Google Patents

Abstract

The invention discloses a multi-viewpoint image fusion method based on block SPCA (Steerable Principal Component Analysis). The method comprises the following steps of (1) dividing a color image into monochrome channel images, segmenting each monochrome channel image into an independent small-block image, and respectively performing SPCA by aiming at each small-block image to obtain a principal component analysis image; (2) calculating the transformation relationship between the two multi-viewpoint adjacent color images on pixel points according to the principal component analysis images, and performing registration on the two multi-viewpoint adjacent color images according to the transformation relationship; and (3) performing wavelet-based multi-viewpoint image fusion on the two registered multi-viewpoint adjacent color images, and obtaining a fused image. The multi-viewpoint image fusion method has the advantages that an SPCA method is used; the image processing process is refined; the necessary registration is added in the image preprocessing link; and the advantages are shown in the aspects of image fusion performance and computation complexity.

Description

Based on the multi-view image fusion method of piecemeal SPCA

Technical field

The present invention relates to the collection field of multi-view point video, particularly relate to a kind of based on piecemeal SPCA multi-view image fusion method.

Background technology

Traditional image co-registration is divided into that Pixel-level merges, feature-based fusion and decision level fusion, and they too focus on the processed level of image, but have ignored the feature that method itself has.In practical application, in order to describe and study conveniently, people be more prone to the means of image co-registration process to name various method.

Therefore, researchers propose respectively based on wavelet transformation, pyramid decomposition, HIS space, morphology, statistics, neural network, the image co-registration of PCA etc.In addition, also someone proposes some and is employed relatively less, but the method that effectiveness comparison is desirable, as Digital Holography method, the gradient method and fuzzy technology method etc. of Decomposition of Mixed Pixels method, multi-wavelength.In addition, multi-view image, with the advantage of its uniqueness, plays irreplaceable advantage in actual applications.

The storage of multi-view image, transport is a puzzlement people's difficult problem.Therefore, realize, to the fusion compression of multi-view image, there is very important practical significance.Due to high resolving power, high information quantity feature that multi-view image has, in order to the convenience of subsequent calculations process, in advance the emphasis that dimensionality reduction is also research is carried out to image.Meanwhile, when fusion compression, owing to there is the conversion of displacement and angle between different images, so process as the case may be again after needing to carry out registration to image.

In actual life, for a certain scene, one group of observed image that can obtain about it from different moment or angle, the entirety of these images is just called multi-view image.People realize the tracking to target by multi-view image technology, the identification of face, the estimation of gesture, and the interchange of multi-user.The scene of reality can be presented in face of observer by multi-view point video that the basis of multi-view image grows up better, has more the sense of reality.But along with the increase of number of cameras, the data volume of multi-view point video also increases exponentially, this brings great difficulty to the storage of video data and transmission.Therefore, realize becoming to the Efficient Compression of multi-view point video the difficult problem that people must solve.

At present, to the compression of multi-view point video mainly centered by de-redundancy, but from the view mode of human eye, to be fused to the compression at center closer to human vision property.Due to the derived relation between multi-view point video and multi-view image, to the fusion of fusion after all still to multi-view image of multi-view point video.Carry out multi-view image fusion, need the information comprised in original each visual point image to be intactly incorporated in the new image of a width to go as far as possible.Like this, people could by newly-generated image, to former scene have vivider, understand intuitively.From the angle of set, if be rc size, image I regards a finite data collection as, so each pixel (i, j), i ∈ { 1,2, ..., r}, j ∈ { 1,2 ..., the pixel value I (i of c}, j) be all the element belonging to this set, i.e. I={I (o, j) | 1≤i≤r, 1≤j≤c}.

When image is very large, the data point that set I comprises is also just many.When regarding each element of I as belong to it an one dimension statistical nature, the dimension of I will inevitably be very high.Practice shows, the image set I very large to intrinsic dimensionality directly merges by traditional method, although may obtain reasonable effect, computation complexity is general all higher, and this is worthless for some pursue high efficiency application program.So, before carrying out fusion treatment, preferably under the main information prerequisite of not losing ensureing original large scale, high-definition picture, dimensionality reduction is carried out to image.Processed image, because data point minimizing can bring great convenience to follow-up calculation process.

Multi-view image, with the advantage of its uniqueness, plays irreplaceable advantage in actual applications.The storage of multi-view image, transport is a puzzlement people's difficult problem.Therefore, realize, to the fusion of multi-view image, there is very strong practical value.

Due to high resolving power, high information quantity feature that multi-view image has, in order to the convenience of follow-up fusion calculation process, treat fused images in advance and carry out dimensionality reduction.Meanwhile, when fusion, owing to there is translation, rotation, the conversion of amplifying, reducing between different images, so need to carry out efficient registration to remove the impact of these conversion to image.

Summary of the invention

The present invention is with SPCA (principal component analysis (PCA) [C ' edricVonesch that direction is adjustable, Fred ' ericStauber, andMichaelUnser.SteerablePCAforRotation-InvariantImageRe cognition.SIAMJ.ImagingSciences.Vol.8, No.3, pp.1857 – 1873]) based on, the conversion registration technology of combining image, fusion treatment is carried out to multi-view image, multi-view image fusion method based on piecemeal SPCA is proposed, to meet the demand that multi-view image merges.

Based on a multi-view image fusion method of piecemeal SPCA, it is characterized in that, comprising:

(1) coloured image is divided into each individual color channel image, each individual color channel Iamge Segmentation is become independently subimage block, carries out SPCA respectively for after each subimage block, obtain principal component analysis (PCA) image;

(2) on pixel, there is transformation relation according between the adjacent two width coloured images of principal component analysis (PCA) image calculating multiple views, according to described transformation relation, registration is carried out to the adjacent two width coloured images of multiple views;

Described transformation relation comprises translation, convergent-divergent and angular transformation.

(3) carry out merging based on the multi-view image of small echo to the adjacent two width coloured images of the multiple views after registration, obtain the image after merging.

In step (1), because general picture size to be fused is all larger, directly carry out mixing operation (matrix operation), calculated amount is very large.Become by large-sized Iamge Segmentation to be fused and independently use SPCA dimensionality reduction to operate again after subimage block, although on the surface because the piecemeal of image causes the number of times calculated to add, the total computation complexity of image co-registration declines.Like this, both remain the degree of accuracy based on entire image method, greatly reduce again the complexity of calculating.

Many videos visual point image is due to the change of the illumination, position etc. during shooting, pixel between adjacent two width images is caused to vary widely on color, position, wherein change in location comprises translation, convergent-divergent, rotation etc., these variation relations directly affects the registration of image, therefore in step (2), obtain the matrix representing these variation relations, then ensure that adjacent image carries out registration in the situation of invariant position, the precision of registration can be higher.

Although the scene of some region performance in two width adjacent images is identical, owing to taking the difference of viewpoint, the light levels that they show and colouring information still have a great difference.So, the pixel value of two sub-picture corresponding parts can not be carried out simple weighted sum-average arithmetic.

As preferably, in order to make the pixel of fused images steady in boundary transition, step (3) being weighted interpolation with the operator S of 3 × 3 before fusion from the intersection of two width coloured images adjacent to the multiple views after registration, and wherein S is:

$\left[\begin{array}{ccc}1/8& 1/8& 1/8\\ 1/8& 0& 1/8\\ 1/8g& 1/8& 1/8\end{array}\right].$

The present invention adopts the multi-view image based on small echo to merge, and step (3) current color image when merging carries out N layer wavelet decomposition, obtains (3N+1) individual different frequency bands, comprises 3N high frequency subimage and 1 low frequency subgraph picture.

Step (3) for HFS, directly gets the coefficient of dissociation of value as fused images of corresponding coefficient of wavelet decomposition maximum absolute value person in two width coloured images when merging.

Step (3) for low frequency part, set a matching degree threshold value T when merging, and calculates two bags of fusion coloured images in the Local Deviation matching degree of each point, according to the relation of Local Deviation matching degree and matching degree threshold value T, adopts corresponding convergence strategy.

After having processed, just can obtain the desirable fused images based on wavelet transformation.

The present invention is based on piecemeal SPCA multi-view image fusion method, use the method for SPCA by the process refinement of image procossing, consider the spatial correlation of 2-D data, have good effect for during gray level image dimensionality reduction.SPCA due to need to consider conventional multi-view image different points of view between there is the fact of displacement difference, the pre-service link of image adds necessary registration and projective transformation operation (direction is adjustable).Therefore, the inventive method all shows advantage in the performance and computation complexity of image co-registration.

Embodiment

Below in conjunction with specific embodiment, the present invention is described in detail.

The present embodiment based on piecemeal SPCA (principal component analysis (PCA) that direction is adjustable) multi-view image fusion method, comprise the steps:

(1) the SPCA piecemeal of single channel image: because general picture size to be fused is all larger, directly carry out mixing operation (matrix operation), calculated amount is very large.Become by large-sized Iamge Segmentation to be fused and independently use SPCA dimensionality reduction to operate again after small images, although on the surface because the piecemeal of image causes the number of times calculated to add, the total computation complexity of image co-registration declines.Like this, both remain the degree of accuracy based on entire image method, greatly reduce again the complexity of calculating.

PCA is a kind of traditional manifold learning, and it by extracting principal character to reach the object reducing calculated amount from sample data.Suppose the sample set S=I that there is K width image ₁, I ₂..., I _k{ }, every width image I _k(K=1,2 ..., k) size is all r × c.First, I _kall pixels be rearranged into a row vector s by row _k, s _ksize be 1 × N, wherein N=r × c; Then, all s _kform a new matrix A=[s ₁s ₂... s _k] ^t; Again svd is carried out to A, ask for the proper vector α corresponding to the maximum d of an A eigenwert; Finally, with the projective transformation of vector by each s _kproject, just obtain the d dimension statement of original image.The PCA method of Here it is gray level image.

The present invention adopt SPCA be PCA improve, compared to PCA with a dimensional vector for starting point, SPCA to the process of data based on two-dimensional matrix.Due to this difference, SPCA can retain the spatial correlation between data to a great extent, insensitive in the change such as translation, rotation to image.In theory, PCA can be replaced to process all single channel band fused images with SPCA.But because general picture size to be fused is all larger, directly carry out the matrix operation of merging, calculated amount is quite large.Become by large-sized Iamge Segmentation to be fused and independently use SPCA again after small images, although calculative number of times adds on the surface, total computation complexity reduces.Like this, both remain the degree of accuracy based on view picture method to be fused, greatly reduce the complexity of calculating again.During concrete operations, coloured image is divided into each individual color channel image, each individual color channel image A is divided into the identical subimage block of n size, and uses A _i, i ∈ (1,2 ..., n) represent i-th subimage block, then the step based on piecemeal SPCA method is as follows:

1) computed image sample set A ₁, A ₂..., A _naverage

2) for all i ∈ 1,2 ..., n}, calculates

3) L is made _o← (E _d, O) ^r, wherein E _dbe the unit matrix of d × d, d is the dimension that image is expected to drop to, and O is null matrix;

4) k=O is made, the initialisation image root-mean-square error REMS (k) when SPCA inverse transformation is gone back ← ∞;

5) by compute matrix M _rthe proper vector corresponding to d eigenvalue of maximum l ₀the L as k=0 _k;

6) make k=k+1, carry out assignment simultaneously

7) by compute matrix M _lthe proper vector corresponding to d eigenvalue of maximum

8) assignment operation is carried out

9) calculate $REMS\left(k\right)=\sqrt{\left(\frac{1}{n}{\mathrm{\Σ}}_{i=1}^n\left|\right|{\tilde{A}}_i-L_k{L}_k^T{\tilde{A}}_i{R}_k{R}_k^T|{|}_F^2\right)},$ Wherein || || _fthe F norm of representing matrix;

10) if REMS (k-1)-REMS (k)≤η, so 11 are turned), otherwise, jump to 5) continue to perform, wherein η is the threshold value preset;

11) to transformation matrix L _{r × d}and R _{c × d}assignment, i.e. L ← L respectively _k, R ← R _k;

12) for each i ∈ 1,2 ..., n}, by formula calculate original-gray image through conversion after Gray Projection (i.e. principal component analysis (PCA) image) and return.

(2) registration of image: many videos visual point image is due to the change of the illumination, position etc. during shooting, pixel between adjacent two width images is caused to vary widely on color, position, wherein change in location comprises translation, convergent-divergent, rotation etc., these variation relations directly affects the registration of image, first the matrix representing these variation relations will be obtained, then ensure that adjacent image carries out registration in the situation of invariant position, the precision of registration can be higher.

Suppose to have visual point image I1 and I2 to be fused that two are adjacent, so in I1, in any point (x1, y1) to I2, the transformation relation of corresponding point (x2, y2) can describe [7] with transformation for mula below:

$\mathrm{\λ}\left[\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right]=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& 1\end{array}\right]\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]---\left(1\right)$

In formula: λ is the yardstick of conversion, and a ~ h is the coefficient of correlating transforms.Homography matrix H is defined as

$H=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& l\end{array}\right]---\left(2\right)$

From formula (2):

$\{\begin{array}{c}{x}_2=\frac{{\mathrm{ax}}_1+{\mathrm{by}}_1+c}{\mathrm{\λ}\left({\mathrm{gx}}_1+{\mathrm{hy}}_1+l\right)}\\ y_2=\frac{{\mathrm{dx}}_1+{\mathrm{ey}}_1+f}{\mathrm{\λ}\left({\mathrm{gx}}_1+{\mathrm{hy}}_1+l\right)}\end{array}---\left(3\right)$

For processing conveniently, when change of scale is constant, only consider the horizontal and vertical displacement between pixel.Then can obtain a=e=1, λ=1, b=d=g=h=O, have simultaneously

$\left\{\begin{array}{c}{x}_2=x_1+c\\ y_2=y_1+f\end{array}\right.---\left(4\right)$

In formula, c and f represents the horizontal and vertical displacement of pixel respectively.

Then registration can be carried out according to formula (4).

In image registration, rough registration only considered the translation problem between adjacent image pixels point, when being used for representing Image1, when representing Image1, their overlapping regions roughly can be calculated for (1:(m-f), 1:(n-c)) and ((1+f): m, (1+c): n).

(3) image co-registration: although the scene of some region performance in two width adjacent images is identical, owing to taking the difference of viewpoint, the light levels that they show and colouring information still have a great difference.So the pixel value of two sub-picture corresponding parts can not be carried out simple weighted sum-average arithmetic.

In order to make the pixel of fused images steady in boundary transition, from their intersection, be weighted interpolation with the operator s of one 3 × 3, wherein S is:

$\left[\begin{array}{ccc}1/8& 1/8& 1/8\\ 1/8& 0& 1/8\\ 1/8g& 1/8& 1/8\end{array}\right]---\left(5\right)$

The present invention adopts the thought of wavelet transformation to merge overlay region, first will carry out N layer wavelet decomposition to image, obtain (3N+1) individual different frequency bands, and these frequency bands comprise 3N high frequency subimage and 1 low frequency subgraph picture.For high frequency imaging part, directly get the coefficient of dissociation of value as fused images of corresponding coefficient of wavelet decomposition maximum absolute value person in two width source images, for low frequency part, the regular relative complex of process some, concrete steps are as follows:

1) suppose:

C (I) represents the matrix of coefficients of the wavelet low frequency composition of image I;

P=(m, n) represents the locus of wavelet coefficient;

So C (I, p) is designated as the value of the element of (m, n) under representing wavelet low frequency composition matrix of coefficients;

2) centered by P, selected a zonule Q, u (I, P) represent that C (I) is centered by P, the average in Q, and G (I, P) is the Local Deviation conspicuousness of C (I) in Q, meet

$G(I,P)={\mathrm{\Σ}}_{q\⋐Q}w\left(q\right)|C(I,q)-u(I,p){|}^2---\left(6\right)$

In formula: W (q) is weights, from p more away from, be worth less;

3) according to formula (6) computed image I respectively ₁, image I ₂local Deviation conspicuousness G (I ₁, P), G (I ₂, P), then calculate their Local Deviation matching degrees at P point:

$M_2\left(p\right)=\frac{2{\mathrm{\Σ}}_{q\⋐Q}w\left(q\right)|C(I_1,q)-u(I_1,p)||C(I_2,q)-u(I_2,p)|}{G(I_1,p)+G(I_2+p)}---\left(7\right)$

4) set a matching degree threshold value T, work as M ₂(P), during <T, convergence strategy is

$C\left(I_f,p\right)=\left\{\begin{array}{c}C\left(I_1,p\right),C\left(I_1,p\right)\&GreaterEqual;C\left(I_2,p\right)\\ C\left(I_2,p\right),C\left(I_1,p\right)<C\left(I_2,p\right)\end{array}\right.---\left(8\right)$

Work as M ₂p, during ()>=T, convergence strategy is Average Strategy

$C\left(I_f,p\right)=\left\{\begin{array}{c}{W}_{\max }C\left(I_1,p\right)+W_{\min }C\left(I_2,p\right),G\left(I_1,p\right)\&GreaterEqual;G\left(I_2,p\right)\\ W_{\max }C\left(I_2,p\right)+W_{\min }C\left(I_1,p\right),G\left(I_1,p\right)<G\left(I_2,p\right)\end{array}\right.---\left(9\right)$

Wherein,

$W_{\min }=0.5-0.5\left(\frac{1-M_2\left(p\right)}{1-T}\right),W_{\max }=1-W_{\min }---\left(10\right)$

After above-mentioned process completes, just can obtain the desirable fused images based on wavelet transformation.

Claims (6)

1., based on a multi-view image fusion method of piecemeal SPCA, it is characterized in that, comprising:

(1) coloured image is divided into each individual color channel image, each individual color channel Iamge Segmentation is become independently subimage block, carries out SPCA respectively for after each subimage block, obtain principal component analysis (PCA) image;

(2) on pixel, there is transformation relation according between the adjacent two width coloured images of principal component analysis (PCA) image calculating multiple views, according to described transformation relation, registration is carried out to the adjacent two width coloured images of multiple views;

(3) carry out merging based on the multi-view image of small echo to the adjacent two width coloured images of the multiple views after registration, obtain the image after merging.

2., as claimed in claim 1 based on the multi-view image fusion method of piecemeal SPCA, it is characterized in that, described transformation relation comprises translation, convergent-divergent and angular transformation.

3. as claimed in claim 2 based on the multi-view image fusion method of piecemeal SPCA, it is characterized in that, step (3) being weighted interpolation with the operator S of 3 × 3 before fusion from the intersection of two width coloured images adjacent to the multiple views after registration, and wherein S is:

$\left[\begin{array}{ccc}1/8& 1/8& 1/8\\ 1/8& 0& 1/8\\ 1/8g& 1/8& 1/8\end{array}\right].$

4. as claimed in claim 3 based on the multi-view image fusion method of piecemeal SPCA, it is characterized in that, step (3) current color image when merging carries out N layer wavelet decomposition, obtain (3N+1) individual different frequency bands, comprise 3N high frequency subimage and 1 low frequency subgraph picture.

5. as claimed in claim 4 based on the multi-view image fusion method of piecemeal SPCA, it is characterized in that, step (3) for HFS, directly gets the coefficient of dissociation of value as fused images of corresponding coefficient of wavelet decomposition maximum absolute value person in two width coloured images when merging.

6. as claimed in claim 5 based on the multi-view image fusion method of piecemeal SPCA, it is characterized in that, step (3) merge time for low frequency part, set a matching degree threshold value T, calculate two bags and merge the Local Deviation matching degree of coloured image at each point, according to the relation of Local Deviation matching degree and matching degree threshold value T, adopt corresponding convergence strategy.