CN101894380A - Method for tracing target object in panoramic video automatically - Google Patents

Abstract

The invention discloses a method for tracing a target object in a panoramic video automatically, which comprises the following steps: (1) acquiring the position of a target object in the panoramic video; (2) detecting if plane crossing appears in the motion of the target object in the cubic panorama, and performing a step (3) if plane crossing appears in the motion of the target object in the cubic panorama or performing a step (4) if plane crossing does not appear in the motion of the target object in the cubic panorama; (3) expanding an image on the original plane of the target object; (4) searching and matching along an epipolar line to obtain the motion vector of the target object; and (5) making a judgment according to the result of the step (4), if a matched target is found, ending the tracing or performing the step (2). The method can be used for tracing the target object in the panoramic video automatically and can narrow search range in a tracing process and ensure the quickness and stability of the tracing of the target object.

Description

The automatic tracking method of target object in a kind of panoramic video

Technical field

The present invention relates to panoramic video technology and target tracking domain, more specifically, relate to a kind of method target following in the panoramic video.

Background technology

Panoramic picture is to be spliced by the real scene image at certain several different directions of a bit taking.Its concrete generative process is: to the simple geometric surface, as sphere, cube face or the face of cylinder, then projected image is carried out seamless splicedly overlapped image mapped, just can obtain panoramic picture.According to the solid that is shone upon, panoramic picture can be divided into cylinder panoramic image, spherical panorama image and cube panoramic picture.Panoramic video is formed by having some free related panoramic image sequence, can obtain real panorama scene from video, and can be by time positioning video frame.

The global motion of the cube panoramic video of shooting static scene has covered 360 degree scene domains usually based on tangential movement, and for the panoramic picture of continuous multiple frames, object cognition moves to adjacent surface from a face of cube panorama.Because it has dynamic background, is not suitable for using the method for background subtraction to carry out the detection and tracking of dynamic object, therefore need the method for tracking target in a kind of cube panoramic video rapidly and efficiently of design.The patent No. is U.S.'s patent of invention of US 6-924-832B1 " METHOD; APPARATUS AND COMPUTER PROGRAM PRODUCT FOR TRACKING OBJECT IN A WARPED VIDEO IMAGE " a kind of method of following the tracks of object in panoramic video proposed, it is mapped to User according to the angle that the user watches by full-view video image, and with its input picture as target following.Set up the model of target object in the tracing process, and set up the probability that tracking target appears at zones of different, with the position of big probability reposition as target according to the front and back frame.The prerequisite of this invention is that the user omission can not occur, and its direction of paying close attention to necessarily comprises the object that needs are followed the tracks of, and does not utilize the movement tendency and the corresponding relation of target object in the panoramic video, thereby not high for the tracking efficient of panoramic video.Application number is 200410016455.9, the Chinese invention patent of publication number CN1658670A " intelligent-tracking supervisory system " with multiple-camera, a panoramic camera and one or more tracking camera have been adopted, when panoramic camera is found moving target, follow the tracks of a plurality of moving targets respectively by tracking camera, obtain HD image.Camera position in this method is static, the video that obtains has static background, so use the method for background subtraction to be easy to realize the tracking of foreground object, mainly be applicable to multiobject video monitoring, for the panoramic video with dynamic background, it is inappropriate utilizing background subtraction detection and tracking moving target.

For the cube panoramic video of taking static scene, the global motion of panoramic camera is usually based on translation, the rotation amount of camera is very little between the two field picture of front and back, for the panoramic picture of continuous multiple frames, the static scene that photographs can move along specific direction, and can move on the adjacent surface from a cubical face.

Summary of the invention

The technical problem to be solved in the present invention is: overcome the deficiencies in the prior art, the automatic tracking method of target object in a kind of panoramic video is provided, this method can effectively reduce the hunting zone in the tracing process, and it is quick and stable to guarantee that target object is followed the tracks of.

In order to obtain above-mentioned technique effect, the technical solution adopted in the present invention is: imported as data along the captured full-view video image of road by the panoramic camera that is installed on the vehicle, after obtaining the position of target object it is carried out from motion tracking, the step that specifically comprises is as follows:

(1) obtains the position of target object in full-view video image.

(2) check whether the motion of this target in the cube panorama phenomenon of the face that passes through occurs,, otherwise enter step (4) if then enter step (3);

(3) contiguous image of utilizing the target object motion vector direction is carried out a kind of expansion to the image of former place face;

(4) along the polar curve direction is carried out search matched, upgrade the motion vector of target object thing;

(5) result according to step (4) judges, if there is not object matching, then finishes to follow the tracks of; Otherwise change step (2) over to.

Principle of the present invention is: for the cube panoramic video of taking static scene, the global motion of video camera is usually based on tangential movement, for the panoramic picture of continuous multiple frames, the static scene that photographs can move along specific direction, and can move on the adjacent surface from a cubical face, so can be according to target object movement tendency and the corresponding relation in the panoramic picture between consecutive frame, the structure scounting line is followed the tracks of target, obtains corresponding motion vector.

The present invention's advantage compared with prior art is: for the cube panoramic video of taking static scene, the present invention in conjunction with the panoramic camera global motion based on tangential movement, rotation amount characteristics seldom, the movement tendency and the corresponding relation of adjacent video interframe target object have been made full use of, target object is carried out from motion tracking, effectively reduce the hunting zone in the tracing process, reduced search time, and it is quick and stable to guarantee that target object is followed the tracks of.

Description of drawings

Fig. 1 is the corresponding relation of pixel between consecutive frame in the cube panoramic video, and wherein a is cubical side, and b is cubical front, and c is the cubical back side;

Fig. 2 is the automatic tracking method flow process of the present invention's target object in panoramic video;

Fig. 3 is for expanding synoptic diagram to associated picture among the present invention; Wherein a is cube panorama top and bottom tessellation synoptic diagram, and b expands synoptic diagram above the cube panorama, and c is cube left side expansion synoptic diagram, and d is cube right side expansion synoptic diagram;

Fig. 4 is a pixel motion trend synoptic diagram in the cube panoramic video;

Fig. 5 is for using the image after proposition method of the present invention is expanded, and the image that a generates for back above the expansion cube, b are the image after the expansion of the cube panorama left side, and c is the image that expansion back, the cube panorama right side generates.

Fig. 6 is the tracking effect to a vehicle target, and a～f is respectively this video the 15th, 30,35,40,40 and 60 frame tracking results.

Specific implementation method

For the panoramic picture of continuous multiple frames, the static scene that photographs can move along specific direction, and can move on the adjacent surface from a cubical face, so when carrying out the target object tracking, should consider the motion of target on the cube adjacent surface, and utilize the movement tendency of pixel to follow the tracks of, as shown in Figure 1.The present frame of Fig. 1 (a) expression cuboid sides and the pixel corresponding relation of former frame, the S1 partial pixel of present frame and the S2 partial pixel of former frame are identical, this two parts pixel only is that relatively moving on the horizontal direction arranged on the position.Fig. 1 (b) and Fig. 1 (c) have represented the present frame at the cube front and the back side and the pixel corresponding relation of former frame respectively, and the T1 partial pixel of present frame and the T2 partial pixel of former frame are similar.In the panoramic video that the present invention proposes the automatic tracking method flow process of target object as shown in Figure 2, its step is as follows:

(1) obtains the position of target object in full-view video image.In panoramic video, obtain the position of target object, can demarcate by the image-region of manual select target; Also can detect automatically by the panoramic video that utilizes input and select; Can also set a region-of-interest, after moving object enters region-of-interest, trigger an alarm, and start tracing process.The target area utilizes bounding box to represent, the target location is expressed as the central point of this bounding box.If detecting the target bounding box is box=([x _Min, x _Max], [y _Min, y _Max]), x wherein _MinAnd x _MaxBe respectively minimum and the maximum coordinates value of bounding box on the image X-axis; Equally, y _MinAnd y _MaxBe minimum and the maximum coordinates value of this target on the image Y-axis.

(2) check whether this target the phenomenon of the face that passes through occurs,, otherwise enter step (4) if then enter step (3).Because panoramic video covers 360 degree scenes, so the situation that a face from the cube panorama moves to other face, the phenomenon of promptly passing through face may take place between the consecutive frame of video target.When object of which movement during to the intersection of two faces of cube, target will be made of the related pixel on two faces; When moving to cubical drift angle place, target then can comprise the image pixel of three faces.

(3) utilize the contiguous image of target object motion vector direction that the image of former place face is expanded.Carry out the expansion of full-view video image according to the current place of target object face, as shown in Figure 3.In Fig. 3 (a), be four triangles with the top of cube panorama along diagonal division, comprise by ABQ form 301, ADQ form 302, BCQ 303 and the CDQ that form form 304.Wherein, straight line AM and straight line AD overlap, and in like manner straight line BC and straight line BN overlap, and straight line DC overlaps with the border of back in the cube panorama.The geometric relationship of tessellation below the cube and correspondence is similar.

When target object move to the cube panorama above the time, in Fig. 3 (b), utilize formula (1) that former place face is expanded, with the pixel-expansion of triangle 301 to rectangle ABFE.(x y) is the coordinate of expansion back image to I wherein, and the distance of establishing this point and straight line AB is d, and d equals the length of line segment IP, calculates the P point coordinate then and is (x, 1/2), asks for straight line PQ and goes up apart from straight line AB apart from the coordinate (x ', y ') that is the some I of d.At last the pixel of (x ', y ') on the triangle 301 is mapped to expansion back image (x y) gets final product.

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}\\ y^{\&prime;}=y\end{array}\right.\frac{-\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2}<x<\frac{\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2},\frac{\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2}<y<l,l\&NotEqual;0---\left(1\right)$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

When target object move to the cube panorama above the time, in Fig. 3 (c), utilize formula (2) that former place face is expanded, process is similar to the extended method above the cube panorama, with the pixel-expansion of triangle 302 to rectangle AEM ' M.

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}-2y+l\\ y^{\&prime;}=y\end{array}\right.\frac{-3\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2}<x<-\frac{\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2},\frac{\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2}<y<l,l\&NotEqual;0---\left(2\right)$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

When target object move to the cube panorama above the time, in Fig. 3 (d), utilize formula (3) that former place face is expanded, process is similar to the extended method above the cube panorama, with the pixel-expansion of triangle 303 to rectangle BNN ' F.

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}+2y-l\\ y^{\&prime;}=y\end{array}\right.\frac{l}{2}<x<\frac{3\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2},\frac{\mathrm{<figure-callout\; id="2"\; label="l"\; filenames="HSA00000197983600021.png,HSA00000197983600022.png"\; state="\{\{state\}\}">l</figure-callout>}}{2}<y<l,l\&NotEqual;0---\left(3\right)$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

Similar for the tessellation below the cube panorama and extended method and said method.

(4) along the polar curve direction is carried out search matched, upgrade the motion vector of target object thing.For the cube panoramic video of taking static scene, the global motion of video camera is usually based on tangential movement.According to the Camera extrinsic number, Florian Kangni has proposed the computing method of cube panorama to polar curve, as shown in Figure 4.Polar curve 403 is sent from the limit 401 that is positioned at cube panorama front, to around spread cube four faces be tangential movement basically in upper and lower, left and right, be pooled to the limit 402 that is positioned at the cube back side at last.Pixel corresponding relation between can obtaining in the panoramic video between consecutive frame to the direction of motion of polar curve 403, it has represented the movement tendency of static scene in the panoramic video.According to the movement tendency of target object, be that initial point is set up plane right-angle coordinate with the picture centre of cube panorama front, the four direction that is adjacent is respectively top, following, the left side and the right side image of cube panorama:

When-l/2≤y _mDuring≤l/2, the search straight-line equation is x=ky+p, and wherein parameter k represents the slope of initial ranging straight line, and p represents to search for the intercept of straight line, and the two calculates by formula (4):

p＝x _m-ky _m

Wherein, (x _m, y _m) expression target to be searched position coordinates, (x _c, y _c) coordinate at presentation video center, l is for being the length of side of a face of cube panorama.

Work as y _m＜-l/2 or y _mDuring＞l/2, the search straight-line equation is x=x _m, same, (x _m, y _m) expression target to be searched position coordinates.

According to this movement tendency and corresponding relation, can adopt the Mean-Shift tracking that target object is followed the tracks of, at initial tracking frame, by the target location that gets access to, determine that search window W selects moving target, calculate the histogram distribution of the search window W under the kernel function weighting, use the same method and calculate the histogram distribution of the corresponding window W of present image, similarity with two distributions is principle to the maximum, makes search window W increase maximum direction along density, moves to the actual position of target.Next frame is according to the actual position of target and the polar curve direction is determined the reposition of search window.Step is as follows:

(4.1) selected target obtains initial frame Mean-Shift target nuclear histogram.

(4.2) gather the next frame image.

(4.3) use the Mean-Shift method to follow the tracks of, obtain new target location.

(4.4) according to the reposition to polar position renewal search window at new target location and place thereof, forward (4.2) to.

(5) result according to step (4) judges, if there is not object matching, then finishes to follow the tracks of; Otherwise change step (2) over to.

The present invention tests the automatic tracking method of target object in the panoramic video.The cube panoramic video of the data that experiment is adopted for taking along BJ University of Aeronautics ﹠ Astronautics's campus road, collecting device is Ladybug3, and wherein the frame number of video is 800 frames, and the resolution of each face is respectively 512 * 512 pixels.According to the image of the described method of step (3) after as shown in Figure 5 with image spreading, wherein Fig. 5 (a) moves to the expansion that carry out former place face the back above the cube for target object, Fig. 5 (b) is for moving to effect after the left side is expanded it when target object, Fig. 5 (c) enters the result who image is expanded behind the right side for target.From the image comparison of expansion front and back, target object can effectively reduce deformation at the adjacent surface intersection in the image after the expansion, and the search matched during for tracking provides good basis.

Fig. 6 has shown final tracking effect, input video is for adopting the video data (only having expanded the top of cube panorama) after said method is expanded, and Fig. 6 (a)-(f) is respectively the tracking results to same vehicle target of the 15th, 30,35,40,45 and 60 frames in the video.The gray scale quantification manner that experiment is adopted is 32 grades of quantifications, has 65 two field pictures can the real-time follow-up target in 70 two field pictures, and following the tracks of efficient is 92.8%, and average calculating operation time be 28.73ms, satisfy the requirement of real-time, and the tracking rate is higher.

Claims (6)

1. the automatic tracking method of target object in the panoramic video, it is characterized in that: the panoramic video of continuous acquisition is carried out the detection of target object, and in subsequent frame testing result is carried out from motion tracking, it is as follows specifically to comprise step:

(1) obtains the position of target object in full-view video image;

(2) check whether the motion of this target in the cube panorama phenomenon of the face that passes through occurs,, otherwise enter step (4) if then enter step (3);

(3) utilize the contiguous image of target object motion vector direction that the image of former place face is expanded;

(4) along the polar curve direction is carried out search matched, upgrade the motion vector of target object thing;

(5) result according to step (4) judges, if there is not object matching, then finishes to follow the tracks of; Otherwise change step (2) over to.

2. the automatic tracking method of target object in the panoramic video according to claim 1, it is characterized in that: described step further comprises in (1): the position that obtains target object in panoramic video, choose the panoramic video front as positive dirction, image-region by manual select target is demarcated, perhaps by utilizing the panoramic video of importing to detect automatically and selecting, perhaps set a region-of-interest, after moving object enters region-of-interest, trigger a signal, and start tracing process; The target area utilizes bounding box to represent, the target location is expressed as the central point of this bounding box; If detecting the bounding box of target is box=([x _Min, x _Max], [y _Min, y _Max]), x wherein _MinAnd x _MaxBe respectively minimum and the maximum coordinates value of bounding box on the image X-axis; y _MinAnd y _MaxBe minimum and the maximum coordinates value of this target on the image Y-axis.

3. the automatic tracking method of target object in the panoramic video according to claim 1, it is characterized in that: described step further comprises in (3): when target object move to the cube panorama above the time, front center with the cube panorama is an initial point, set up plane right-angle coordinate, and utilize following formula that former place face is expanded:

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}\\ y^{\&prime;}=y\end{array}\right.\frac{-l}{2}<x<\frac{l}{2},\frac{l}{2}<y<l,l\&NotEqual;0$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

4. the automatic tracking method of target object in the panoramic video according to claim 1, it is characterized in that: described step further comprises in (3): when target object moves to the left side of cube panorama, front center with the cube panorama is an initial point, set up plane right-angle coordinate, and utilize following formula that former place face is expanded:

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}-2y+l\\ y^{\&prime;}=y\end{array}\right.\frac{-3l}{2}<x<-\frac{l}{2},\frac{l}{2}<y<l,l\&NotEqual;0$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

5. the automatic tracking method of target object in the panoramic video according to claim 1, it is characterized in that: described step further comprises in (3): when target object moves to the right side of cube panorama, front center with the cube panorama is an initial point, set up plane right-angle coordinate, use following formula that former place face is expanded:

$\left\{\begin{array}{c}{x}^{\&prime;}=\frac{2(l-y)x}{l}+2y-l\\ y^{\&prime;}=y\end{array}\right.\frac{l}{2}<x<\frac{3l}{2},\frac{l}{2}<y<l,l\&NotEqual;0$

Wherein, (x y) is the coordinate of expansion back image, (x ', y ') be the coordinate of target object when front institute abut upper, l is the length of side of a face of cube panorama.

6. the automatic tracking method of target object in the panoramic video according to claim 1, it is characterized in that: described step further comprises in (4), searches for the equation of straight line according to the movement tendency decision of detected target object in each face image of cube panorama; Picture centre with cube panorama front is that initial point is set up plane right-angle coordinate, and the four direction that is adjacent is respectively top, following, the left side and the right side image of cube panorama:

When-l/2≤y _mDuring≤l/2, the search straight-line equation is x=ky+p, and wherein parameter k represents the slope of initial ranging straight line, and p represents to search for the intercept of straight line, and the two calculates by following formula:

p＝x _m-ky _m

Wherein, (x _m, y _m) expression target to be searched position coordinates, (x _c, y _c) coordinate at presentation video center, l is for being the length of side of a face of cube panorama;

Work as y _m＜-l/2 or y _mDuring＞l/2, the search straight-line equation is x=x _m, same, (x _m, y _m) expression target to be searched position coordinates.

Images