CN103136526B - Based on the online method for tracking target of multi-source image feature fusion - Google Patents

Abstract

The invention discloses a kind of online method for tracking target based on multi-source image feature fusion, for solving the technical matters based on the optimal characteristics method for tracking target poor robustness chosen online.Technical scheme first utilizes the mode of linear combination visible images and infrared image to be merged, and makes target reach maximum with the contrast of background on present image, the outstanding characteristic information of target on image; Secondly, by obtaining clarification of objective information to the mode of Objective extraction angle point, and the tracking of optical flow algorithm realize target is utilized.In order to improve the robustness of tracking further, add the information of detection sorting algorithm to object and background sample to classify, again on this basis, utilize on-line study, associated treatment optical flow tracking result and detection classification results, obtain optimum target following result, tracking results rate of accuracy reached is to more than 85%.

Description

Based on the online method for tracking target of multi-source image feature fusion

Technical field

The present invention relates to a kind of online method for tracking target, particularly relate to a kind of online method for tracking target based on multi-source image feature fusion.

Background technology

The image utilizing visible light sensor and infrared sensor to obtain presents different physical characteristics respectively, automatically, effectively visible images and infrared image carries out merging and carries out the online target following of robust, having very important significance.Existing online method for tracking target mainly contains: the stencil matching tracking based on on-line study and the optimal characteristics tracking based on on-line study.

Document " Onlineselectionofdiscriminativetrackingfeatures.PAMI, 27 (10): 1631-1643, Oct.2005. " discloses a kind of optimal characteristics method for tracking target based on choosing online.The method adopts the mode of statistics with histogram to obtain optimum target signature linear combination image, adopts the method for mean-shift to follow the tracks of target afterwards.Choosing the optimum linearity combination image stage, utilize no optimum configurations to the R in visible images, G, B tri-channel image carry out linear fusion, then on newly-generated a large amount of linear combination images, statistical study is carried out to the histogram of the object and background chosen, obtain the linear fusion image of maximum-contrast result, utilize the linear fusion parameter of this linear fusion image to carry out the same manner fusion treatment to next frame image.But the method is mainly for the linear fusion of visible images three passages, after increase infrared image, the optimum configurations of this linear fusion can not directly be suitable for.At mean-shift tracking phase, owing to lacking necessary template renewal, when attitudes vibration occurs target itself, follow the tracks of unsuccessfully; Because window width size remains unchanged in tracing process, when target scale changes to some extent, follow the tracks of unsuccessfully; When target velocity is very fast, tracking effect is bad.In sum, the robustness of mean-shift tracking is not fine.

Summary of the invention

In order to overcome the deficiency of the existing optimal characteristics method for tracking target poor robustness based on choosing online, the invention provides a kind of online method for tracking target based on multi-source image feature fusion.The method utilizes the mode of linear combination visible images and infrared image to be merged, and makes target reach maximum with the contrast of background on present image, the outstanding characteristic information of target on image; Secondly, by obtaining clarification of objective information to the mode of Objective extraction angle point, and the tracking of optical flow algorithm realize target is utilized.In order to improve the robustness of tracking further, adding the information of detection sorting algorithm to object and background sample and classifying, then on this basis, utilizing on-line study, associated treatment optical flow tracking result and detection classification results, obtain optimum target following result.

The technical solution adopted for the present invention to solve the technical problems is: a kind of online method for tracking target based on multi-source image feature fusion, is characterized in comprising the following steps:

Step one, to the R in visible images, the information of G, B tri-passages carries out linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces fused images.The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2，-1，0，1，2]}(1)

In formula, the image information of three passages in the corresponding visible images of R, G, B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w _*for corresponding linear combination parameter, span is-2 to 2.Reject equivalent combinations mode (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄').

On step 2, image after each linear combination, calculate the statistics with histogram information of target area and background area respectively.The pixel characteristic histogram of target is made to be H _obji (), the pixel characteristic histogram of background sample is H _bgi (), the probability density being normalized calculating target and background respectively obtains

p(i)=H _obj(i)/n _obj(2)

q(i)=H _bg(i)/n _bg(3)

In formula, n _obj, n _bgrepresent the quantity of target sample and background sample respectively, p (i), q (i) represent the discrete probability density of target sample and background sample respectively.Utilize p (i), q (i) obtains likelihood function

$L\left(i\right)=\log \frac{\max \{p\left(i\right),\mathrm{\δ}\}}{\max \{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

In formula, δ=0.001, places the situation that log appears as 0.Judged the diversity factor of target sample characteristic sum background sample characteristics by the variance calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²obtain

$\mathrm{var}(L;p,q)=\underset{i}{\mathrm{\Σ}}a\left(i\right)L^2\left(i\right)-{\left[\underset{i}{\mathrm{\Σ}}a\left(i\right)L\left(i\right)\right]}^2---\left(5\right)$

In formula, a (i) is probability density function.Thus obtain the variance ratio formula of likelihood function

$\mathrm{VR}(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

Obtain the data of all samples, the data according to sample sort, and obtain maximum target signature and the contrast of background characteristics, obtain optimum Fusion Features linear combination mode.

Step 3, the first method passed through based on Corner Detection and non-maxima suppression carry out feature extraction to the object in the fused images obtained; Then RANSAC method is utilized to remove exterior point; Finally calculate the characteristic light stream of the invariant feature point remained by optical flow method, the position that estimating target occurs at next frame.The motion at the observed object adopting light stream to describe to cause relative to the motion of observer, surface or edge.Instantaneous velocity or the discrete picture displacement of motion is detected by image.All there is the vector set (x, y, t) of a two dimension or multidimensional in each moment, and (x, y, t) represents the instantaneous velocity of specified coordinate at t point.If the intensity that I (x, y, t) puts for t (x, y), in very short time Δ t, x, y increase Δ x, Δ y respectively:

$I(x+\mathrm{\Δx},y+\mathrm{\Δy},t+\mathrm{\Δt})=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}---\left(7\right)$

Meanwhile, consider that the displacement of two frame adjacent images is enough short, therefore

I(x，y，t)=I(x+Δx，y+Δy，t+Δt)(8)

Thus obtain

$\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δx}}{\mathrm{\Δt}}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δy}}{\mathrm{\Δt}}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δt}}{\mathrm{\Δt}}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

In formula, V _xand V _ybe the speed of x and y, be called the light stream of I (x, y, t), the partial derivative of image (x, y, t) at t specific direction.I _x, I _y, I _trelation as follows:

I _xV _x+I _yV _y=-I _t(12)

In formula, I _x, I _ybe respectively the x direction of Feature point correspondence or the Grad in y direction, I _tthe gray scale difference value of front and back two two field picture in corresponding pixel points position.Draw the light stream of unique point selected in target, the basis of former frame target location estimates the position that next frame target occurs.

Step 4, first, produce all positions that target may occur, be input to ground floor probability classification and vote, result passes to NearestNeighbor sorter; Secondly, NearestNeighbor sorter carries out decision-making to it, is exported in position with a high credibility and its intrinsic similarity; Again, the input using the output of NearestNeighbor sorter as distance restraint sorter, filters out the possible position apart from confirmed tracking results.The net result of the i.e. detecting portion finally exported.If the net result number of detecting portion is too much or very few, feed back to NearestNeighbor sorter and the distance restraint sorter stage adjustment threshold value carry out decision-making again.

Step 5, according to the tracking results in each two field picture and testing result, first judge whether tracking results and testing result all exist, then adopt and follow the tracks of and detect synergistic mechanism net result is adjusted; Finally by feedback mechanism, net result is used for adjusting pursuit path, and upgrades each sorter in testing process.

The invention has the beneficial effects as follows: the method utilizes the mode of linear combination visible images and infrared image to be merged, make target reach maximum with the contrast of background on present image, the outstanding characteristic information of target on image; Secondly, by obtaining clarification of objective information to the mode of Objective extraction angle point, and the tracking of optical flow algorithm realize target is utilized.In order to improve the robustness of tracking further, add the information of detection sorting algorithm to object and background sample to classify, again on this basis, utilize on-line study, associated treatment optical flow tracking result and detection classification results, obtain optimum target following result, tracking results rate of accuracy reached is to more than 85%.

Below in conjunction with embodiment, the present invention is elaborated.

Embodiment

The online method for tracking target concrete steps that the present invention is based on multi-source image feature fusion are as follows:

1, multi-source image feature fusion.

A (), to the R in visible images, the information of G, B tri-passages carries out multiple linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces a large amount of fused images.The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2，-1，0，1，2]}(1)

Wherein, the image information of three passages in the corresponding visible images of R, G, B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w _*for corresponding linear combination parameter, span is-2 to 2.After calculating, the image altogether after generation 625 linear combinations, but in these images, be similar to (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄') array mode is equivalent, therefore after the array mode weeding out equivalence, remaining computable linear combination image one has 215.

On (b) image after each linear combination, calculate the statistics with histogram information of target area and background area respectively.The pixel characteristic histogram of target is made to be H _obji (), the pixel characteristic histogram of background sample is H _bgi (), the probability density being normalized calculating target and background respectively obtains

p(i)=H _obj(i)/n _obj(2)

q(i)=H _bg(i)/n _bg(3)

Wherein, n _obj, n _bgrepresent the quantity of target sample and background sample respectively.Utilize p (i), q (i) obtains likelihood function

$L\left(i\right)=\log \frac{\max \{p\left(i\right),\mathrm{\δ}\}}{\max \{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

Wherein, δ=0.001, places the situation that log appears as 0.Judged the diversity factor of target sample characteristic sum background sample characteristics by the variance calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²obtain

$\mathrm{var}(L;p,q)=\underset{i}{\mathrm{\Σ}}a\left(i\right)L^2\left(i\right)-{\left[\underset{i}{\mathrm{\Σ}}a\left(i\right)L\left(i\right)\right]}^2---\left(5\right)$

Wherein, a (i) is probability density function.Thus obtain the variance ratio formula of likelihood function

$\mathrm{VR}(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

After calculating, obtain the data of all samples, the data according to sample sort, and obtain maximum target signature and the contrast of background characteristics, thus obtain optimum Fusion Features linear combination mode.

2, on-line study target following.

A () first carries out feature extraction by the method based on Corner Detection and non-maxima suppression to the object in the fused images obtained; Then RANSAC method is utilized to remove exterior point; Finally calculate the characteristic light stream of the invariant feature point remained by optical flow method, the position occurred at next frame in order to estimating target.Adopt light stream (opticalflow) be used for describing cause relative to the motion of observer observed object, surface or edge motion.A series of image detects instantaneous velocity or the discrete picture displacement of motion.All there is the vector set of a two dimension or multidimensional in each moment, as (x, y, t), represents the instantaneous velocity of specified coordinate at t point.If the intensity that I (x, y, t) puts for t (x, y), in very short time Δ t, x, y increase Δ x, Δ y respectively:

$I(x+\mathrm{\Δx},y+\mathrm{\Δy},t+\mathrm{\Δt})=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}---\left(7\right)$

Meanwhile, consider that the displacement of two frame adjacent images is enough short, therefore:

I(x，y，t)=I(x+Δx，y+Δy，t+Δt)(8)

Therefore

$\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δx}}{\mathrm{\Δt}}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δy}}{\mathrm{\Δt}}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δt}}{\mathrm{\Δt}}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

V _xand V _ybe the speed of x and y, or be called the light stream of I (x, y, t), the partial derivative of image (x, y, t) at t specific direction.I _x, I _y, I _trelation represent as follows:

I _xV _x+I _yV _y=-I _t(12)

Here I is understood _x, I _ybe respectively the x direction of Feature point correspondence or Grad, the I in y direction _tthe gray scale difference value of front and back two two field picture in corresponding pixel points position.Draw the light stream of unique point selected in target thus, and then on the basis of former frame target location, estimate the position of next frame target appearance.

B () utilizes light stream to carry out in tracing process; frequent meeting is because be subject to illumination variation; occlusion issue and occur lost efficacy; because we are while utilizing optical flow tracking; also detection sorting algorithm is added; by a learning process to optical flow tracking result and classification results, choose optimum target following result.Multi classifier combination testing mechanism is made up of three parts, and every part all produces positive sample and the negative sample of judgement, and wherein, negative sample adds for later stage on-line study in the negative sample set of system, and positive sample enters lower one deck sorter and judges further.First, produce all positions that target may occur, be input to ground floor probability classification and vote, result passes to NearestNeighbor sorter; Secondly, NearestNeighbor sorter carries out decision-making to it, is exported in position with a high credibility and its intrinsic similarity; Again, the input using the output of NearestNeighbor sorter as distance restraint sorter, filters out the possible position apart from confirmed tracking results.The net result of the i.e. detecting portion finally exported.If the net result number of detecting portion is too much or very few, feed back to NearestNeighbor sorter and the distance restraint sorter stage adjustment threshold value carry out decision-making again.

C (), according to the tracking results in each two field picture and testing result, first judges whether tracking results and testing result all exist, then adopt to follow the tracks of to adjust net result with detection synergistic mechanism; Finally by feedback mechanism, net result is used for adjusting pursuit path, and upgrades each sorter in testing process.

Claims (1)

1., based on an online method for tracking target for multi-source image feature fusion, it is characterized in that comprising the following steps:

Step one, to the R in visible images, the information of G, B tri-passages carries out linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces fused images; The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2,-1,0,1,2]}(1)

In formula, the image information of three passages in the corresponding visible images of R, G, B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w _*for corresponding linear combination parameter, span is-2 to 2; Reject equivalent combinations mode (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄');

On step 2, image after each linear combination, calculate the statistics with histogram information of target area and background area respectively; The pixel characteristic histogram of target is made to be H _obji (), the pixel characteristic histogram of background sample is H _bgi (), the probability density being normalized calculating target and background respectively obtains

p(i)＝H _obj(i)/n _obj(2)

q(i)＝H _bg(i)/n _bg(3)

In formula, n _obj, n _bgrepresent the quantity of target sample and background sample respectively, p (i), q (i) represent the discrete probability density of target sample and background sample respectively; Utilize p (i), q (i) obtains likelihood function

$L\left(i\right)=log\frac{max\{p\left(i\right),\mathrm{\δ}\}}{max\{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

In formula, δ=0.001, prevents log from appearing as the situation of 0; Judged the diversity factor of target sample characteristic sum background sample characteristics by the variance calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²obtain

$\mathrm{var}(L;p,q)=\underset{i}{\Σ}a\left(i\right)L^2\left(i\right)-\[{\underset{i}{\Σ}a\left(i\right)L\left(i\right)\]}^2---\left(5\right)$

In formula, a (i) is probability density function; Thus obtain the variance ratio formula of likelihood function

$VR(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

Obtain the data of all samples, the data according to sample sort, and obtain maximum target signature and the contrast of background characteristics, obtain optimum Fusion Features linear combination mode;

Step 3, the first method passed through based on Corner Detection and non-maxima suppression carry out feature extraction to the object in the fused images obtained; Then RANSAC method is utilized to remove exterior point; Finally calculate the characteristic light stream of the invariant feature point remained by optical flow method, the position that estimating target occurs at next frame; The motion at the observed object adopting light stream to describe to cause relative to the motion of observer, surface or edge; Instantaneous velocity or the discrete picture displacement of motion is detected by image; All there is the vector set (x, y, t) of a two dimension or multidimensional in each moment, and (x, y, t) represents the instantaneous velocity of specified coordinate at t point; If the intensity that I (x, y, t) puts for t (x, y), in very short time △ t, x, y increase △ x, △ y respectively:

$I(x+\mathrm{\Δ}x,y+\mathrm{\Δ}y,t+\mathrm{\Δ}t)=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δ}x+\frac{\∂I}{\∂y}\mathrm{\Δ}y+\frac{\∂I}{\∂t}\mathrm{\Δ}t---\left(7\right)$

Meanwhile, consider that the displacement of two frame adjacent images is enough short, therefore

I(x,y,t)＝I(x+△x,y+△y,t+△t)(8)

Thus obtain

$\frac{\∂I}{\∂x}\mathrm{\Δ}x+\frac{\∂I}{\∂y}\mathrm{\Δ}y+\frac{\∂I}{\∂t}\mathrm{\Δ}t=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δ}x}{\mathrm{\Δ}t}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δ}y}{\mathrm{\Δ}t}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δ}t}{\mathrm{\Δ}t}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

In formula, V _xand V _ybe the speed of x and y, be called the light stream of I (x, y, t), the partial derivative of image (x, y, t) at t specific direction; I _x, I _y, I _trelation as follows:

I _xV _x+I _yV _y＝-I _t(12)

In formula, I _x, I _ybe respectively the x direction of Feature point correspondence or the Grad in y direction, I _tthe gray scale difference value of front and back two two field picture in corresponding pixel points position; Draw the light stream of unique point selected in target, the basis of former frame target location estimates the position that next frame target occurs;

Step 4, first, produce all positions that target may occur, be input to ground floor probability classification and vote, result passes to NearestNeighbor sorter; Secondly, NearestNeighbor sorter carries out decision-making to it, is exported in position with a high credibility and its intrinsic similarity; Again, the input using the output of NearestNeighbor sorter as distance restraint sorter, filters out the possible position apart from confirmed tracking results; The net result of the i.e. detecting portion finally exported; If the net result number of detecting portion is too much or very few, feed back to NearestNeighbor sorter and the distance restraint sorter stage adjustment threshold value carry out decision-making again;

Step 5, according to the tracking results in each two field picture and testing result, first judge whether tracking results and testing result all exist, then adopt and follow the tracks of and detect synergistic mechanism net result is adjusted; Finally by feedback mechanism, net result is used for adjusting pursuit path, and upgrades each sorter in testing process.