CN105046649A - Panorama stitching method for removing moving object in moving video - Google Patents

Abstract

The invention relates to a panorama stitching method for removing a moving object in a moving video, comprising the following steps: (1) acquiring an appropriate number of video screenshots according to the length of a video, wherein the screenshot selection interval is shorter is the video is shorter, and the screenshot selection interval is longer if the video is longer; (2) projecting an original image to a sphere in order to prevent a final panorama from distortion and deformation; (3) carrying out SURF feature point detection on each frame of image to obtain a feature point description sub vector as the basis of matching; (4) carrying out feature point matching; (5) calculating a perspective (homographic) transformation matrix of every two frames of images based on matching feature points, and calculating forward and reverse homographic matrix transformation of two adjacent frames; (6) obtaining the size of the panorama based on the homographic transformation matrix of adjacent frames; and (7) inversely mapping the panorama to a sequence frame, building a candidate point list for each pixel point of the panorama, and removing a moving object, thus, completing generation of the panorama. The resolution of panorama generation is high, background overlay is accurate, and the 'ghost' phenomenon of the moving object is avoided.

Description

A kind of Panoramagram montage method removing moving object in sport video

Technical field

The invention belongs to field of video image processing, particularly relate to a kind of Panoramagram montage method removing moving object in sport video.

Background technology

No matter at sphere of learning or engineering field, Panoramic Image Mosaic Algorithm is all one of hot issue of image procossing.It has a large amount of application in all kinds of software, as camera panorama sketch synthesis etc.Panorama sketch is that observer revolves from a fixed view scene turning around and can see as surrounding, and it needs to express complete true environment information.The synthesis of panorama sketch easily produces " ghost ", and namely moving object has multiple image at panorama sketch and causes panorama sketch to there is the ghost of moving object at different time, have impact on the synthetic effect of panorama sketch.

Current all kinds of algorithms still existing defects, as panorama mosaic speed is slow, within 5 seconds, video needs the generated time of tens of seconds; Synthesis poor visual effect, can not mate completely, namely background cannot superpose unanimously, generation image blurring; Video is only limited to and moves horizontally, can not vertically movement; Moving object is eliminated not exclusively, has afterimage etc.

Summary of the invention

The object of the invention is to: realize using the video of certain hour length as input, within the several seconds, synthesize the panorama sketch that is removed moving object in video.This panorama picture formation sharpness is higher, and background superposition is accurate, without " ghost " phenomenon of moving object.

Technical scheme of the present invention is as follows:

Remove a Panoramagram montage method for moving object in sport video, it is characterized in that described method comprises the steps:

(1) according to the video interception of the length acquisition suitable quantity of video;

(2) image of sectional drawing gained is hinted obliquely on sphere, to prevent the final panorama sketch torsional deformation formed;

(3) SURF feature point detection is carried out to every two field picture, obtain the descriptor vector of unique point, as the foundation of coupling;

(4) Feature Points Matching;

(5) calculated the perspective homograph matrix of every two two field pictures by matching characteristic point, calculate the both forward and reverse directions homography matrix conversion of adjacent two frames;

(6) by the homograph matrix of consecutive frame, panorama sketch size is obtained;

(7) be incident upon sequence frame by panorama sketch reflection, be that each pixel of panorama sketch builds candidate point chained list, remove moving object, panorama map generalization is complete.

Further, in described step (1), the concrete mode that video interception is selected is: if video length is within 2 seconds, then get a two field picture at interval of 5 frames; If video length is between 2 seconds to 5 seconds, then get a two field picture at interval of 10 frames; If video length was more than 5 seconds, then get a two field picture at interval of 15 frames.

Further, in described step (2), image projection is as follows to the method for sphere:

Hypothetical world coordinate is XYZ, and camera coordinate is xyz, and camera coordinates system is obtained by the X-axis rotation alpha angle of world coordinate system in world coordinate system;

If the image coordinate of any one pixel p is (x, y) on real scene image f, its image coordinate corresponding on spherical projection image is wherein θ is feathering angle, be the angle of pitch, the height of real scene image is H, and width is W, and sets ball centre as coordinate origin, and camera focus is λ, and radius of a ball r is camera focus; The coordinate of this time point p (x, y) under camera coordinates system xyz is (x – W/2, y – H/2 ,-λ), then the coordinate (u, v, w) it be converted under world coordinate system XYZ is:

$\left[\begin{array}{c}u\\ v\\ w\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \mathrm{\α}& -\sin \mathrm{\α}\\ 0& \sin \mathrm{\α}& \cos \mathrm{\α}\end{array}\right]\left[\begin{array}{c}x-W/2\\ y-H/2\\ -\mathrm{\λ}\end{array}\right]$

And have

Can be obtained by above two formulas

Therefore spherical co-ordinate formula is obtained:

Wherein (x ', y ') is corresponding to the development of a sphere planimetric coordinates value of photographic plane coordinate (x, y); Formula in utilization, then can be projected to plane original image, generates development of a sphere planimetric coordinates image.

Further, in described step (4), the method for Feature Points Matching is as follows:

Euclidean distance is adopted to be used as the canonical function of the evaluation of two proper vector distances:

$D(P_1,P_2)=\sqrt{\underset{i-1}{\overset{n}{\Σ}}{(k_{1i}-k_{2i})}^2}$

Wherein P1, P2 represent the unique point in two two field pictures, descriptor vector i-th component of what k1i and k2i represented is P1, P2; For reducing erroneous matching rate further, adopt arest neighbors than the method for secondary neighbour, minimum distance and time is closely represented respectively with ND (NearestDistance) and NND (NextNearestDistance), then nearest and secondary ratio Rod is closely ND/NND, we can set a threshold value threshold and (be greater than 0, be less than the positive number of 1), then:

$\left\{\begin{array}{ccc}if& Rod\≤threshold& ,success\\ else& & ,failure\end{array}\right.$

Rule of thumb, threshold value is between 0.5 to 0.7.

Further, in described step (5), computation process adopts based on RANSAC matching process, because homography matrix needs 4 pairs of match points to calculate, therefore the subset of the multiple 4 pairs of match points of random selecting, obtain optimal subset by least error method, and estimate the homograph matrix between adjacent two two field pictures thus.

Further, in described step (6), concrete grammar is as follows:

Centre one frame choosing all image sequence frames, as panorama sketch base coordinate frame, because perspective matrix transformation model has transferability between frames, therefore can calculate forward between any two frames or transformation by reciprocal direction by the transformation matrix of consecutive frame; By four of the image of other frames summits, be mapped in base coordinate frame, be chosen at wherein minimum x coordinate and the minimum y coordinate top left corner apex as panorama sketch, with maximum x coordinate and maximum y coordinate as the summit, the lower right corner of panorama sketch, the size of panorama sketch can be obtained thus.

Further, in described step (7), concrete grammar is as follows:

From the upper left angle point of panorama sketch, travel through the pixel of panoramic picture by row; Reverse transform matrix is utilized obtain the position in single-frame images coordinate system for pixel P (x, y); Because panorama sketch pixel coordinate not all exists back projection's point at every two field picture, therefore only select the point alternatively point that there is back projection's point;

For all candidate point Pi (xi, yi) sort according to gray-scale value, pixel is wherein likely the point of foreground moving object, also be likely background dot, in order to background dot can be utilized to carry out assignment to panoramic picture, our hypothesis is in the middle of whole frame sequence, and the time that some some passive movement object blocks is less than the time that it comes out, so we can utilize the intermediate value of choosing candidate point as the value of panoramic picture vegetarian refreshments (x, y); By the method that this median pixel is chosen, we can filtering moving object, and only retain background; Thus, panorama map generalization is complete.

Beneficial effect of the present invention is:

(1) panorama sketch synthesis result is clear, does not have fuzzy ghost image.

First the extraction of SURF unique point has Scale invariant shape, and namely not by image translation, the impact of rotation, it can extract the unique point in image in good condition, mates point set accurately for coupling provides.Moreover use arest neighbors to be used as the difference metric method of unique point than the method for secondary neighbour, reduce erroneous matching rate.And algorithm adopts the unique point of RANSAC least error method to two two field pictures to mate, and it can find four pairs of optimum unique point subsets.

(2) panorama sketch aggregate velocity is very fast, can complete panorama sketch synthesis within the several seconds.

SURF feature point detection has efficient real-time, by setting suitable minHessian threshold parameter, can reduce the feature point extraction time.RANSAC least error method also has higher aging performance, makes overall image mosaic have computing velocity faster.

(3) synthesis supports that video level moves and vertical movement, can generate the panorama sketch of horizontal direction and vertical direction.

Owing to hinting obliquely to spherical co-ordinate by original sectional drawing image, define development of a sphere planimetric coordinates image, therefore transverse movement and lengthwise movement can be processed when splicing simultaneously, generate the panorama sketch of horizontal direction and vertical direction.

(4) vestige of moving object can be eliminated in good condition, there is no ghost.

Algorithm in panorama sketch for each pixel maintains the pixel value chained list of an order, and adopt the mode choosing intermediate value to determine the pixel value of this pixel, due in the most of the time, the pixel value at this place is background and non-moving objects, therefore can choose intermediate value using background pixel point as the pixel value of this pixel.

(5) can moving object be extracted, support further to operate, such as, change background, insert moving object etc.

Owing to obtaining the panorama sketch not having moving object, only need to use frame difference method just can extract moving object, then carry out follow-up operation.

(6) image vision is comparatively level and smooth, avoids because single image is too bright or too dim and cause the regional luminance of panorama sketch inconsistent.

Be get median operation after the candidate point of pixel each in panorama sketch is sorted, therefore can avoid choosing bright or excessively dark pixel, achieve vision smooth effect.

(7) algorithm can do self-adaptative adjustment to video length, if video length is longer, then choose frame period large, if video is shorter, then choose frame period little, such selection can take into account algorithm execution efficiency and panorama sketch synthetic effect.

Accompanying drawing explanation

Fig. 1 is the schematic diagram of world coordinate system XYZ of the present invention and camera coordinate system xyz.

Fig. 2 is the process flow diagram of the inventive method.

Embodiment

Below in conjunction with accompanying drawing, technical scheme of the present invention is further elaborated.

The process flow diagram of the inventive method as shown in Figure 2.Remove a Panoramagram montage method for moving object in sport video, it is characterized in that described method comprises the steps:

(1) according to the video interception of the length acquisition suitable quantity of video;

(2) because people are under normal circumstances that health does not move and rotating camera carries out the acquisition of panoramic video in shooting custom, therefore need the image of sectional drawing gained to hint obliquely on sphere, to prevent the final panorama sketch torsional deformation formed;

(3) SURF feature point detection is carried out to every two field picture, obtain the descriptor vector of unique point, as the foundation of coupling;

(4) Feature Points Matching;

(5) calculated the perspective homograph matrix of every two two field pictures by matching characteristic point, calculate the both forward and reverse directions homography matrix conversion of adjacent two frames;

(6) by the homograph matrix of consecutive frame, panorama sketch size is obtained;

(7) be incident upon sequence frame by panorama sketch reflection, be that each pixel of panorama sketch builds candidate point chained list, remove moving object, panorama map generalization is complete.

Wherein, in described step (1), the concrete mode that video interception is selected is: if video length is within 2 seconds, then get a two field picture at interval of 5 frames; If video length is between 2 seconds to 5 seconds, then get a two field picture at interval of 10 frames; If video length was more than 5 seconds, then get a two field picture at interval of 15 frames.

Wherein, in described step (2), image projection is as follows to the method for sphere:

As shown in Figure 1, hypothetical world coordinate is XYZ, and camera coordinate is xyz, and camera coordinates system is obtained by the X-axis rotation alpha angle of world coordinate system in world coordinate system;

If the image coordinate of any one pixel p is (x, y) on real scene image f, its image coordinate corresponding on spherical projection image is wherein θ is feathering angle, be the angle of pitch, the height of real scene image is H, and width is W, and sets ball centre as coordinate origin, and camera focus is λ, and radius of a ball r is camera focus; The coordinate of this time point p (x, y) under camera coordinates system xyz is (x – W/2, y – H/2 ,-λ), then the coordinate (u, v, w) it be converted under world coordinate system XYZ is:

$\left[\begin{array}{c}u\\ v\\ w\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \mathrm{\α}& -\sin \mathrm{\α}\\ 0& \sin \mathrm{\α}& \cos \mathrm{\α}\end{array}\right]\left[\begin{array}{c}x-W/2\\ y-H/2\\ -\mathrm{\λ}\end{array}\right]$

And have

Can be obtained by above two formulas

Therefore spherical co-ordinate formula is obtained:

Wherein (x ', y ') is corresponding to the development of a sphere planimetric coordinates value of photographic plane coordinate (x, y); Formula in utilization, then can be projected to plane original image, generates development of a sphere planimetric coordinates image.

Wherein, in described step (4), the method for Feature Points Matching is as follows:

Euclidean distance is adopted to be used as the canonical function of the evaluation of two proper vector distances:

$D(P_1,P_2)=\sqrt{\underset{i-1}{\overset{n}{\Σ}}{(k_{1i}-k_{2i})}^2}$

Wherein P1, P2 represent the unique point in two two field pictures, descriptor vector i-th component of what k1i and k2i represented is P1, P2; For reducing erroneous matching rate further, adopt arest neighbors than the method for secondary neighbour, minimum distance and time is closely represented respectively with ND (NearestDistance) and NND (NextNearestDistance), then nearest and secondary ratio Rod is closely ND/NND, we can set a threshold value threshold and (be greater than 0, be less than the positive number of 1), then:

$\left\{\begin{array}{ccc}if& Rod\≤threshold& ,success\\ else& & ,fai1ure\end{array}\right.$

Rule of thumb, threshold value is between 0.5 to 0.7.

Wherein, in described step (5), computation process adopts based on RANSAC matching process, because homography matrix needs 4 pairs of match points to calculate, therefore the subset of the multiple 4 pairs of match points of random selecting, obtain optimal subset by least error method, and estimate the homograph matrix between adjacent two two field pictures thus.

Wherein, in described step (6), concrete grammar is as follows:

Centre one frame choosing all image sequence frames, as panorama sketch base coordinate frame, because perspective matrix transformation model has transferability between frames, therefore can calculate forward between any two frames or transformation by reciprocal direction by the transformation matrix of consecutive frame; By four of the image of other frames summits, be mapped in base coordinate frame, be chosen at wherein minimum x coordinate and the minimum y coordinate top left corner apex as panorama sketch, with maximum x coordinate and maximum y coordinate as the summit, the lower right corner of panorama sketch, the size of panorama sketch can be obtained thus.

Wherein, in described step (7), concrete grammar is as follows:

From the upper left angle point of panorama sketch, travel through the pixel of panoramic picture by row; Reverse transform matrix is utilized obtain the position in single-frame images coordinate system for pixel P (x, y); Because panorama sketch pixel coordinate not all exists back projection's point at every two field picture, therefore only select the point alternatively point that there is back projection's point;

For all candidate point Pi (xi, yi) sort according to gray-scale value, pixel is wherein likely the point of foreground moving object, also be likely background dot, in order to background dot can be utilized to carry out assignment to panoramic picture, our hypothesis is in the middle of whole frame sequence, and the time that some some passive movement object blocks is less than the time that it comes out, so we can utilize the intermediate value of choosing candidate point as the value of panoramic picture vegetarian refreshments (x, y); By the method that this median pixel is chosen, we can filtering moving object, and only retain background; Thus, panorama map generalization is complete.

Claims (7)

1. remove a Panoramagram montage method for moving object in sport video, it is characterized in that described method comprises the steps:

(1) according to the video interception of the length acquisition suitable quantity of video;

(2) image of sectional drawing gained is hinted obliquely on sphere, to prevent the final panorama sketch torsional deformation formed;

(3) SURF feature point detection is carried out to every two field picture, obtain the descriptor vector of unique point, as the foundation of coupling;

(4) Feature Points Matching;

(5) calculated the perspective homograph matrix of every two two field pictures by matching characteristic point, calculate the both forward and reverse directions homography matrix conversion of adjacent two frames;

(6) by the homograph matrix of consecutive frame, panorama sketch size is obtained;

(7) be incident upon sequence frame by panorama sketch reflection, be that each pixel of panorama sketch builds candidate point chained list, remove moving object, panorama map generalization is complete.

2. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, it is characterized in that, in described step (1), the concrete mode that video interception is selected is: if video length is within 2 seconds, then get a two field picture at interval of 5 frames; If video length is between 2 seconds to 5 seconds, then get a two field picture at interval of 10 frames; If video length was more than 5 seconds, then get a two field picture at interval of 15 frames.

3. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, is characterized in that, in described step (2), image projection is as follows to the method for sphere:

Hypothetical world coordinate is XYZ, and camera coordinate is xyz, and camera coordinates system is obtained by the X-axis rotation alpha angle of world coordinate system in world coordinate system;

If the image coordinate of any one pixel p is (x, y) on real scene image f, its image coordinate corresponding on spherical projection image is wherein θ is feathering angle, be the angle of pitch, the height of real scene image is H, and width is W, and sets ball centre as coordinate origin, and camera focus is λ, and radius of a ball r is camera focus; The coordinate of this time point p (x, y) under camera coordinates system xyz is (x – W/2, y – H/2 ,-λ), then the coordinate (u, v, w) it be converted under world coordinate system XYZ is:

$\left[\begin{array}{c}u\\ \mathrm{\ν}\\ w\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos a& -\sin a\\ 0& \sin a& cosa\end{array}\right]\left[\begin{array}{c}x-W/2\\ y-H/2\\ -\mathrm{\λ}\end{array}\right]$

And have

Can be obtained by above two formulas

Therefore spherical co-ordinate formula is obtained:

Wherein (x ', y ') is corresponding to the development of a sphere planimetric coordinates value of photographic plane coordinate (x, y); Formula in utilization, then can be projected to plane original image, generates development of a sphere planimetric coordinates image.

4. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, is characterized in that, in described step (4), the method for Feature Points Matching is as follows:

Euclidean distance is adopted to be used as the canonical function of the evaluation of two proper vector distances:

$D(P_1,P_2)=\sqrt{\underset{i=1}{\overset{n}{\Σ}}{(k_{1i}-k_{2i})}^2}$

Wherein P1, P2 represent the unique point in two two field pictures, descriptor vector i-th component of what k1i and k2i represented is P1, P2; For reducing erroneous matching rate further, adopt arest neighbors than the method for secondary neighbour, minimum distance and time is closely represented respectively with ND (NearestDistance) and NND (NextNearestDistance), then nearest and secondary ratio Rod is closely ND/NND, we can set a threshold value threshold and (be greater than 0, be less than the positive number of 1), then:

$\left\{\begin{array}{ccc}if& Rod\≤threshold& ,success\\ else& & ,failure\end{array}\right.$

Rule of thumb, threshold value is between 0.5 to 0.7.

5. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, it is characterized in that, in described step (5), computation process adopts based on RANSAC matching process, because homography matrix needs 4 pairs of match points to calculate, therefore the subset of the multiple 4 pairs of match points of random selecting, obtain optimal subset by least error method, and estimate the homograph matrix between adjacent two two field pictures thus.

6. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, is characterized in that, in described step (6), concrete grammar is as follows:

Centre one frame choosing all image sequence frames, as panorama sketch base coordinate frame, because perspective matrix transformation model has transferability between frames, therefore can calculate forward between any two frames or transformation by reciprocal direction by the transformation matrix of consecutive frame; By four of the image of other frames summits, be mapped in base coordinate frame, be chosen at wherein minimum x coordinate and the minimum y coordinate top left corner apex as panorama sketch, with maximum x coordinate and maximum y coordinate as the summit, the lower right corner of panorama sketch, the size of panorama sketch can be obtained thus.

7. a kind of Panoramagram montage method removing moving object in sport video according to claim 1, is characterized in that, in described step (7), concrete grammar is as follows:

From the upper left angle point of panorama sketch, travel through the pixel of panoramic picture by row; Reverse transform matrix is utilized obtain the position in single-frame images coordinate system for pixel P (x, y); Because panorama sketch pixel coordinate not all exists back projection's point at every two field picture, therefore only select the point alternatively point that there is back projection's point;

For all candidate point Pi (xi, yi) sort according to gray-scale value, pixel is wherein likely the point of foreground moving object, also be likely background dot, in order to background dot can be utilized to carry out assignment to panoramic picture, our hypothesis is in the middle of whole frame sequence, and the time that some some passive movement object blocks is less than the time that it comes out, so we can utilize the intermediate value of choosing candidate point as the value of panoramic picture vegetarian refreshments (x, y); By the method that this median pixel is chosen, we can filtering moving object, and only retain background; Thus, panorama map generalization is complete.