CN105678719A - Panoramic stitching seam smoothing method and panoramic stitching seam smoothing device - Google Patents

Abstract

The invention relates to the technical field of image processing, and especially relates to a panoramic stitching seam smoothing method and a panoramic stitching seam smoothing device. An image is divided into a fusion area and a non-fusion area by selecting the fusion area from an overlapping area (with the rest being the non-fusion area), and the fusion area and the non-fusion area are processed and synthesized into a panoramic image, thus smoothing the seam of panoramic stitching. Through panoramic image generation, the requirement of large-field video monitoring is met. As the monitoring scope expands continuously, there is a need to monitor a whole bridge or airport, a long road or a high-rise building. However, the field of view of most cameras cannot achieve such a large monitoring scope, and therefore, a high-end demand is formed gradually. Panoramic stitching seam smoothing can make a panoramic image smoother.

Description

A kind of panoramic mosaic seam crossing smoothing method and device

Technical field

The present invention relates to technical field of image processing, especially one panoramic mosaic seam crossing smoothing method and device.

Background technology

In today that monitoring system is fast-developing, system scale is increasing, control point gets more and more, staff has no time to attend in the face of increasing monitoring image, continuous expansion along with monitoring range, it is necessary to bed rearrangement bridge, airport, very Chang Yiduan highway or skyscraper are monitored, and the visual field of most of video cameras does not reach so big monitoring range, therefore the requirement of big visual field video monitoring is increased gradually, gradually formed a kind of high demand. Video panorama splices as a solution, have also been obtained increasing concern. It is very smooth that panoramic mosaic contact position often cannot be spliced, accordingly, it would be desirable to a kind of panoramic mosaic seam crossing smoothing method and device.

Summary of the invention

The technical problem to be solved is: provide a kind of panoramic mosaic seam crossing smoothing method and device, it is achieved splicing seam crossing smooths.

In order to solve above-mentioned technical problem, the technical solution used in the present invention is:

A kind of panoramic mosaic seam crossing smoothing method, including:

S101, gather the panorama block plan that more than two width are to be spliced;

S102, S101 gained panorama block plan is carried out image characteristics extraction;

S103, S102 gained panorama block plan is carried out Image Feature Matching;

S104, by the projection of S103 gained panorama block plan under the same coordinate system, it is thus achieved that overlapping region;

S105, choosing a region as integration region in S104 gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

S106, each pixel value obtained according to mapping relations on panoramic picture, carry out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

S107, output panoramic picture.

Another technical scheme that the present invention adopts is:

A kind of panoramic mosaic seam crossing smoothing apparatus, including the image capture module being sequentially connected with, characteristic extracting module, characteristic matching module, image processing module, FPGA processing module and image output module;

Described image capture module, for gathering the panorama block plan that more than two width are to be spliced;

Described characteristic extracting module, for carrying out image characteristics extraction by more than two width collected panorama block plan to be spliced;

Described characteristic matching module, for carrying out Image Feature Matching by the panorama block plan after described feature extraction;

Described image processing module, for projecting the panorama block plan after described characteristic matching to the same coordinate system, it is thus achieved that overlapping region;

Described FPGA processing module includes choosing unit and acquiring unit;

Described choosing unit, for choosing a region as integration region in image processing module gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

Described acquiring unit, for obtaining each pixel value on panoramic picture according to mapping relations, carries out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

Described image output module, is used for exporting panoramic picture.

The beneficial effects of the present invention is: by choosing integration region in overlapping region, image is divided into integration region and non-fused region, respectively integration region and non-fused region are processed, synthesis panorama sketch, realize the smooth of panoramic mosaic seam crossing, synthesis panorama sketch, it is achieved smoothing of panoramic mosaic seam crossing. The generation of panorama sketch meets the requirement of big visual field video monitoring, continuous expansion along with monitoring range, need bed rearrangement bridge, airport, very Chang Yiduan highway or skyscraper are monitored, the visual field of most of video cameras does not reach so big monitoring range, has gradually formed a kind of high demand. The smooth of panoramic mosaic seam crossing can make panorama sketch seem more smooth.

Accompanying drawing explanation

Fig. 1 is the block diagram of specific embodiment of the invention panoramic mosaic seam crossing smoothing method;

Fig. 2 is the metric space extremum extracting schematic diagram in the specific embodiment of the invention;

Fig. 3 extracts sift characteristic point to generate local feature descriptor schematic diagram in the specific embodiment of the invention;

Fig. 4 is the gray area integration schematic diagram in the specific embodiment of the invention;

Fig. 5 extracts surf characteristic point to generate local feature descriptor schematic diagram in the specific embodiment of the invention;

Fig. 6 is the harris angle point schematic diagram extracted in the specific embodiment of the invention;

Fig. 7 is specific embodiment of the invention midplane projection model schematic diagram;

Fig. 8 is specific embodiment of the invention central column face projection model schematic diagram;

Fig. 9 is spherical projection model schematic in the specific embodiment of the invention;

Figure 10 is average addition method schematic diagram in the specific embodiment of the invention;

Figure 11 is the schematic diagram choosing integration region in the specific embodiment of the invention;

Figure 12 is the structural representation of panoramic mosaic seam crossing smoothing apparatus in the specific embodiment of the invention;

Label declaration:

10, image capture module; 20, characteristic extracting module; 30, characteristic matching module; 40, image processing module; 50, FPGA processing module; 60, image output module.

Detailed description of the invention

By describing the technology contents of the present invention in detail, being realized purpose and effect, below in conjunction with embodiment and coordinate accompanying drawing to be explained.

The design of most critical of the present invention is in that: by choosing integration region in overlapping region, and image is divided into integration region and non-fused region, respectively integration region and non-fused region is processed, and synthesizes panorama sketch, it is achieved smoothing of panoramic mosaic seam crossing.

Refer to Fig. 1, be the block diagram of specific embodiment of the invention panoramic mosaic seam crossing smoothing method, specific as follows:

A kind of panoramic mosaic seam crossing smoothing method, including:

S101, gather the panorama block plan that more than two width are to be spliced;

S102, S101 gained panorama block plan is carried out image characteristics extraction;

S103, S102 gained panorama block plan is carried out Image Feature Matching;

S104, by the projection of S103 gained panorama block plan under the same coordinate system, it is thus achieved that overlapping region;

S105, choosing a region as integration region in S104 gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

S106, each pixel value obtained according to mapping relations on panoramic picture, carry out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

S107, output panoramic picture.

Known from the above, the beneficial effects of the present invention is: by choosing integration region in overlapping region, image is divided into integration region and non-fused region, respectively integration region and non-fused region are processed, synthesis panorama sketch, it is achieved smoothing of panoramic mosaic seam crossing. The generation of panorama sketch meets the requirement of big visual field video monitoring, continuous expansion along with monitoring range, need bed rearrangement bridge, airport, very Chang Yiduan highway or skyscraper are monitored, the visual field of most of video cameras does not reach so big monitoring range, has gradually formed a kind of high demand. The smooth of panoramic mosaic seam crossing can make panorama sketch seem more smooth.

Further, in described step S102, " feature extraction " adopts and extracts sift, surf or harris characteristic point.

Described extraction sift characteristic point step:

1) metric space extremum extracting, primarily determines that key point position and place yardstick. The metric space of two dimensional image is realized by the convolution of gaussian kernel function Yu image.

$G(x,y,\mathrm{\σ})=\frac{1}{2\mathrm{\π}{\mathrm{\σ}}^2}{e}^{-(x^2+y^2)}/2{\mathrm{\σ}}^2,$ L (x, y, σ)=G (x, y, σ) * I (x, y);

When detecting yardstick spatial extrema, 9 × 2 pixels of surrounding neighbors, 26 pixels altogether of 8 pixels of surrounding neighbors and adjacent yardstick correspondence position that the pixel being labeled as cross in Fig. 2 needs the attendant of a stage actor to draw together same yardstick compare, to guarantee local extremum all to be detected at metric space and two dimensional image space;

2) position of precise positioning feature point, by matching three-dimensional quadratic function accurately to determine position and the yardstick of key point, removes the key point of low contrast and unstable skirt response point simultaneously;

At key point place, Taylor expansion obtains:

$D\left(X\right)=D+\frac{{\∂D}^T}{\∂X}X+\frac{1}{2}{X}^T\frac{{\∂}^{2}D}{\∂X^2}X;$

In formula, X=(x, y, σ)^TFor the side-play amount of key point, D is the value at D (x, y, σ) key point place;

3) principal direction of characteristic point is determined;

$m(x,y)=\sqrt{{(L(x+1,y)-L(x-1,y))}^2\text{+}{(L(x,y+1)-L(x,y-1))}^2}$

θ (x, y)=atan2 ((L (x, y+1)-L (x, y-1))/(L (x+1, y)-L (x-1, y)));

4) local feature descriptor is generated;

First coordinate axes is rotated to be the direction of key point, centered by key point, takes the window of 4*4, as shown in Figure 3. In Fig. 3, the stain of left figure is the position of current key point, each little lattice represent a pixel of key point neighborhood place metric space, the direction of arrow represents the gradient direction of this pixel, arrow length represents the size of gradient, and circle represents the scope (the pixel gradient directional information the closer to key point is contributed more big) of Gauss weighting. Next on the fritter of each 4*4, calculate the gradient orientation histogram in 8 directions, draw the accumulated value of each gradient direction, a seed points can be formed, one key point is made up of totally 4*4 16 seed points, each seed points has 8 direction vector information, namely ultimately forms the sift characteristic vector of 128 dimensions.

Described extraction surf characteristic point step:

1) IntegralImages (integrogram);

Integrogram mainly calculate some region in image pixel and, the integrogram definition at x place, position is as follows:

As Fig. 4 grey area integrogram is: A-B-C+D;

2) approximate Hessian matrix;

1 X=in given image I (x, y), its Hessian matrix is:

$L_{\mathrm{xx}}=\frac{{\∂}^{2}g\left(\mathrm{\σ}\right)}{\∂x^2}*I(x,y);$

3) metric space describes;

Surf is the size variation of boxfilter, but not image scaling;

4) positioning feature point;

Scalogram picture is obtained at (x according to Hessian matrix, y) after the extreme value at place, first in the three-dimensional neighborhood of 3 × 3 × the 3 of extreme point, carry out non-maxima suppression, be interpolated in metric space and image space, use quadratic fit function to be interpolated:

Above formula is carried out derivation, and the extreme value obtaining extreme point place is:

$L_{\mathrm{xx}}=\frac{{\∂}^{2}g\left(\mathrm{\σ}\right)}{\∂x^2}*I(x,y);$

When extreme value >=0.03, this point is characteristic point;

5) feature descriptor;

Such as Fig. 5, centered by characteristic point, building a length of side along main formula position is the square of 20, is further divided into the subregion of 4*4, in each zonule, it is divided into again 5*5 sampled point, calculates Haar small echo in the response both horizontally and vertically gone up responded relative to main formula position;

Described extraction harris characteristic point step:

1) each pixel is calculated correlation matrix m;

$m=\left[\begin{array}{cc}{I}_x^2& I_x{I}_y\\ I_x{I}_y& I_y^2\end{array}\right]$

$I_x^2=I_x*I_x$

$I_y^2=I_y*I_y;$

2) four elements of m being carried out Gaussian smoothing filter, obtain new m, Gaussian function is:

$\mathrm{Gauss}=\exp (-\frac{(x^2+y^2)}{2{\mathrm{\σ}}^2});$

3) m is utilized to calculate the angle point amount cim of each pixel;

$\mathrm{cim}=\frac{I_x^2*I_y^2-{\left(I_x{I}_y\right)}^2}{I_x^2+I_y^2};$

4) cim meets more than some threshold value and cim is certain neighborhood local maximum, and what satisfy condition is exactly angle point;

Fig. 6 is the harris angle point extracted in piece image.

Seen from the above description, described " feature extraction " adopts to extract in sift characteristic point and can strengthen coupling stability " by matching three-dimensional quadratic function accurately to determine position and the yardstick of key point, remove the key point of low contrast and unstable skirt response point simultaneously ", improve noise resisting ability; Described " feature extraction " adopts " being interpolated in metric space and image space " in extraction surf characteristic point candidate feature point can be carried out sub-pixel positioning.

Further, in described step S104, " projection " adopts plane projection model or cylindrical surface projecting model or spherical projection model.

Described panorama sketch, adopts plane projection model to carry out synthesis concrete steps:

Such as Fig. 7, plane projection model is first to use image registration techniques to try to achieve the transformation relation between image to be spliced more selected reference plane, and image projects to reference plane one by one, re-uses image fusion technology and generates panoramic picture. Choosing of reference plane can be a width reference picture place plane, it is also possible to is the arbitrary plane in space. Owing to being the plane conversion to plane, with 8 parameter model perspective transformation matrix H.

$H=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& 1\end{array}\right];$

This perspective transformation matrix is the matrix of 3 × 3, and last parameter is fixed as 1, all the other 8 unknown parameters, need to be obtained calculating by characteristic matching, and by least 4 pairs of characteristic matching pair, 8 unknown numbers, 8 equations adopt method of least square to calculate.

Described panorama sketch, adopts cylindrical surface projecting model to carry out synthesis concrete steps:

Such as Fig. 8, all image projection to periphery, set up with the viewpoint o cylinder two-dimensional coordinate system (Xc being initial point, Yc), defining cylindrical section radius be f, plane S is the plane of delineation, puts P, Q is the point on plane S, point M, N are the point on the face of cylinder, and M, N is Q subpoint on the face of cylinder, and plane picture S is with C for zero. Described Xc, Yc formula is as follows:

$X_c=f\×\arctan \frac{x}{f}$

$Y_c=\frac{\mathrm{yf}}{\sqrt{x^2+f^2}};$

Described panorama sketch, adopts spherical projection model to carry out synthesis concrete steps:

Such as Fig. 9, all image projection to spherome surface, set up with the viewpoint O sphere two-dimensional coordinate system (Xs being initial point, Ys), defining cylindrical section radius be f, plane S is the plane of delineation, point P, Q is the point on plane S, and some M, N are the point on sphere, and M, N is P, the Q subpoint on sphere, and plane picture S is with C for zero.

Definition sphere origin is (0,0), and the latitude coordinates of some P corresponding point M under spheric coordinate system is:

$X_s=f\×\arctan \frac{x}{f}$

$Y_s=f\×\arctan \frac{y}{\sqrt{x^2+f^2}};$

Further, in described step S106, " image co-registration " adopts the average addition method or Weighted Fusion.

" image co-registration " in described step S110 adopt the average addition method particularly as follows:

Such as Figure 10, in image co-registration region, the pixel value Pixel of pixel is obtained by the average superposition of pixel value Pixel_L and Pixel_R of corresponding point in two width images, and Pixel here can be the forms such as RGB or YUV, determines according to input picture:

Pixel=0.5* (Pixel_L+Pixel_R);

" image co-registration " in described step S110 adopt Weighted Fusion particularly as follows:

In image co-registration region, the pixel value Pixel of pixel is obtained by the pixel value Pixel_L of corresponding point in two width images and Pixel_R weighted average, it may be assumed that Pixel=k × Pixel_L+ (1-k) × Pixel_R, and wherein k is adjustable factors;

0≤k≤1 under normal circumstances, namely in integration region, the direction along image 1 to image 2, k is faded to 0 by 1, thus realizing the smooth registration of integration region. For making point in image co-registration region and two width images set up bigger dependency, make k=d1/ (d1+d2), such as Figure 11, wherein: d1, d2 represent respectively in integration region o'clock to the distance of the left margin in two width image co-registration regions and right margin. Overlapping region is also not equal to integration region, once 2 original images are determined, then overlay region field width is fixing certainly, but integration region is manual control, it is possible to as long as it is just passable to select arbitrarily to meet integration region W1≤W2 within the scope of overlapping region.

Namely using formula Pixel=d1/ (d1+d2) × Pixel_L+d2/ (d1+d2) × Pixel_R to carry out stitching thread process, non-fused region directly maps from artwork and obtains.

Refer to Figure 12, be embodiment of the present invention panoramic mosaic seam crossing smoothing apparatus structural representation, specific as follows:

A kind of panoramic mosaic seam crossing smoothing apparatus, including the image capture module 10 being sequentially connected with, characteristic extracting module 20, characteristic matching module 30, image processing module 40, FPGA processing module 50 and image output module 60;

Described image capture module 10, for gathering the panorama block plan that more than two width are to be spliced;

Described characteristic extracting module 20, for carrying out image characteristics extraction by more than two width collected panorama block plan to be spliced;

Described characteristic matching module 30, for carrying out Image Feature Matching by the panorama block plan after described feature extraction;

Described image processing module 40, for projecting the panorama block plan after described characteristic matching to the same coordinate system, it is thus achieved that overlapping region;

Described FPGA processing module 50 includes choosing unit and acquiring unit;

Described choosing unit, for choosing a region as integration region in image processing module gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

Described acquiring unit, for obtaining each pixel value on panoramic picture according to mapping relations, carries out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

Described image output module 60, is used for exporting panoramic picture.

Known from the above, the beneficial effects of the present invention is: by choosing integration region in overlapping region, image is divided into integration region and non-fused region, respectively integration region and non-fused region are processed, synthesis panorama sketch, it is achieved smoothing of panoramic mosaic seam crossing. The generation of panorama sketch meets the requirement of big visual field video monitoring, continuous expansion along with monitoring range, need bed rearrangement bridge, airport, very Chang Yiduan highway or skyscraper are monitored, the visual field of most of video cameras does not reach so big monitoring range, has gradually formed a kind of high demand. The smooth of panoramic mosaic seam crossing can make panorama sketch seem more smooth.

Further, the acquiring unit of described FPGA processing module also includes arithmetical unit;

Include the first arithmetic device and the second arithmetic device that are sequentially connected with described arithmetical unit; Described first arithmetic device, for being overlapped the pixel value of the panorama each corresponding point of block plan integration region more than two width; Described second arithmetic device, for averaging to the pixel value after each corresponding point superposition of first arithmetic device gained integration region.

Further, the acquiring unit of described FPGA processing module also includes arranging unit;

Described unit is set, is used for presetting an adjustable parameter, and the pixel value according to each corresponding point of panorama block plan more than two width regulates parameter.

Seen from the above description, adjustable parameter be according to two width more than the pixel value of each corresponding point of panorama block plan carry out parameter adjustment, image syncretizing effect can be made better, more smooth.

In sum, a kind of panoramic mosaic seam crossing smoothing method provided by the invention and device, by calculating panorama block plan overlapping region, and in overlapping region, choose integration region and non-fused region carries out the fusion of panorama block plan, fusion process is according to the form (RGB or YUV etc.) of original image or video, synthesis panorama sketch, it is achieved smoothing of panoramic mosaic seam crossing. The generation of panorama sketch meets the requirement of big visual field video monitoring, continuous expansion along with monitoring range, need bed rearrangement bridge, airport, very Chang Yiduan highway or skyscraper are monitored, the visual field of most of video cameras does not reach so big monitoring range, has gradually formed a kind of high demand. The smooth of panoramic mosaic seam crossing can make panorama sketch seem more smooth, it is simple to the application in life. Described " feature extraction " adopts to extract in sift characteristic point and can strengthen coupling stability " by matching three-dimensional quadratic function accurately to determine position and the yardstick of key point, remove the key point of low contrast and unstable skirt response point simultaneously ", improve noise resisting ability; Described " feature extraction " adopts " being interpolated in metric space and image space " in extraction surf characteristic point candidate feature point can be carried out sub-pixel positioning; Adjustable parameter be according to two width more than the pixel value of each corresponding point of panorama block plan carry out parameter adjustment, image syncretizing effect can be made better, more smooth.

The foregoing is only embodiments of the invention; not thereby the scope of the claims of the present invention is limited; every equivalents utilizing description of the present invention and accompanying drawing content to make, or directly or indirectly it is used in relevant technical field, all in like manner include in the scope of patent protection of the present invention.

Claims (10)

1. a panoramic mosaic seam crossing smoothing method, it is characterised in that including:

S101, gather the panorama block plan that more than two width are to be spliced;

S102, S101 gained panorama block plan is carried out image characteristics extraction;

S103, S102 gained panorama block plan is carried out Image Feature Matching;

S104, by the projection of S103 gained panorama block plan under the same coordinate system, it is thus achieved that overlapping region;

S105, choosing a region as integration region in S104 gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

S106, each pixel value obtained according to mapping relations on panoramic picture, carry out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

S107, output panoramic picture.

2. panoramic mosaic seam crossing smoothing method according to claim 1, it is characterised in that in described step S102, sift, surf or harris characteristic point is extracted in " feature extraction " employing.

3. panoramic mosaic seam crossing smoothing method according to claim 1, it is characterised in that in described step S104 " projection " adopt plane projection model or cylindrical surface projecting model or spherical projection model.

4. panoramic mosaic seam crossing smoothing method according to claim 1, it is characterised in that in described step S106, " image co-registration " adopts the average addition method or Weighted Fusion.

5. panoramic mosaic seam crossing smoothing method according to claim 4, it is characterised in that described image co-registration adopts the average addition method particularly as follows: the pixel value elder generation superposition of each corresponding point of panorama block plan more than two width averaged again.

6. panoramic mosaic seam crossing smoothing method according to claim 4, it is characterised in that described image co-registration adopts weighting particularly as follows: preset an adjustable parameter, and the pixel value according to each corresponding point of panorama block plan more than two width regulates parameter.

7. panoramic mosaic seam crossing smoothing method according to claim 6, it is characterised in that the codomain of described adjustable parameter is 0～1.

8. a panoramic mosaic seam crossing smoothing apparatus, it is characterised in that include the image capture module, characteristic extracting module, characteristic matching module, image processing module, FPGA processing module and the image output module that are sequentially connected with;

Described image capture module, for gathering the panorama block plan that more than two width are to be spliced;

Described characteristic extracting module, for carrying out image characteristics extraction by more than two width collected panorama block plan to be spliced;

Described characteristic matching module, for carrying out Image Feature Matching by the panorama block plan after described feature extraction;

Described image processing module, for projecting the panorama block plan after described characteristic matching to the same coordinate system, it is thus achieved that overlapping region;

Described FPGA processing module includes choosing unit and acquiring unit;

Described choosing unit, for choosing a region as integration region in image processing module gained overlapping region, it is determined that the integration region of matched image, remaining area is non-fused region;

Described acquiring unit, for obtaining each pixel value on panoramic picture according to mapping relations, carries out image co-registration according to acquired pixel value to integration region, and the pixel in non-fused region maps from original image and obtains, it is thus achieved that panoramic picture;

Described image output module, is used for exporting panoramic picture.

9. panoramic mosaic seam crossing smoothing apparatus according to claim 8, it is characterised in that the acquiring unit of described FPGA processing module also includes arithmetical unit;

Include the first arithmetic device and the second arithmetic device that are sequentially connected with described arithmetical unit; Described first arithmetic device, for being overlapped the pixel value of the panorama each corresponding point of block plan integration region more than two width; Described second arithmetic device, for averaging to the pixel value after each corresponding point superposition of first arithmetic device gained integration region.

10. panoramic mosaic seam crossing smoothing apparatus according to claim 8, it is characterised in that the acquiring unit of described FPGA processing module also includes arranging unit;

Described unit is set, is used for presetting an adjustable parameter, and the pixel value according to each corresponding point of panorama block plan more than two width regulates parameter.