CN103136526A - Online target tracking method based on multi-source image feature fusion - Google Patents

Abstract

The invention discloses an online target tracking method based on multi-source image feature fusion. The method is used for solving the technical problem that an optimal feature target tracking method based on on-line choosing is poor in robustness. The technical scheme of the method includes firstly utilizing a linear combination manner to fuse a visible image and an infrared image, enabling contrast of a target on a current image and the background to achieve maximum, and protruding feature information of the target on the image; secondly, obtaining character information of the target by extracting of angular points of the target, and utilizing an optical flow algorithm to achieve tracking of the target. In order to further improve robustness of tracking, a detection classification algorithm is added to conduct classification on information of the target and a background sample, based on the algorithm, on-line learning is utilized, an optical flow tracking result and a detection classification result are processed in a synergy mode to obtain the optimal target tracking result, and accuracy rate of the tracking result is up to more than 85%.

Description

Online method for tracking target based on the multi-source image Fusion Features

Technical field

The present invention relates to a kind of online method for tracking target, particularly relate to a kind of online method for tracking target based on the multi-source image Fusion Features.

Background technology

Utilize the image that visible light sensor and infrared sensor obtain to present respectively different physical characteristics, automatically, effectively visible images with infrared image merges and carry out the online target following of robust, have very important significance.Existing online method for tracking target mainly contains: based on the stencil matching tracking of on-line study with based on the optimal characteristics tracking of on-line study.

Document " Online selection of discriminative tracking features.PAMI, 27 (10): 1631-1643, Oct.2005. " discloses a kind of optimal characteristics method for tracking target based on choosing online.The method adopts the mode of statistics with histogram to obtain optimum target signature linear combination image, adopts afterwards the method for mean-shift that target is followed the tracks of.Choosing the optimum linearity combination image stage, utilize no parameter to arrange the R in visible images, G, three channel image of B are carried out linearity and are merged, then on newly-generated a large amount of linear combination images, the target chosen and the histogram of background are carried out statistical study, obtain the linear fused images of maximum-contrast result, utilize the linear fusion parameters of this linearity fused images to carry out the same manner fusion treatment to the next frame image.But the method is mainly the linearity fusion for three passages of visible images, and after increasing infrared image, the parameter setting that this linearity merges can not directly be suitable for.At the mean-shift tracking phase, owing to lacking necessary template renewal, when the attitude variation occurs target itself, follow the tracks of unsuccessfully; Because the window width size remains unchanged, when target scale changes to some extent, follow the tracks of unsuccessfully in tracing process; When target velocity was very fast, tracking effect was bad.In sum, the robustness of mean-shift tracking is not fine.

Summary of the invention

In order to overcome existing deficiency based on the optimal characteristics method for tracking target poor robustness of choosing online, the invention provides a kind of online method for tracking target based on the multi-source image Fusion Features.The mode that the method is utilized linear combination merges visible images and infrared image, makes target reach maximum, the outstanding characteristic information of target on image with the contrast of background on present image; Secondly, obtain clarification of objective information by the mode of target being extracted angle point, and utilize the tracking of optical flow algorithm realize target.For the further robustness of following the tracks of that improves, add the detection sorting algorithm that the information of target and background sample is classified, then on this basis, utilized on-line study, associated treatment optical flow tracking result and detection classification results obtain optimum target following result.

The technical solution adopted for the present invention to solve the technical problems is: a kind of online method for tracking target based on the multi-source image Fusion Features is characterized in comprising the following steps:

Step 1, to the R in visible images, G, the information of three passages of B is carried out linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces fused images.The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2，-1，0，1，2]} (1)

In formula, R, G, the image information of three passages in the corresponding visible images of B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w _*Be corresponding linear combination parameter, span is-2 to 2.Reject equivalent combinations mode (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄').

Step 2, on the image after each linear combination, calculate respectively the statistics with histogram information of target area and background area.The pixel characteristic histogram that makes target is H _obj(i), the pixel characteristic histogram of background sample is H _bg(i), calculate respectively the probability density of target and background and carry out normalization and obtain

p(i)=H _obj(i)/n _obj (2)

q(i)=H _bg(i)/n _bg (3)

In formula, n _obj, n _bgRepresent respectively the quantity of target sample and background sample, p (i), q (i) represent respectively the discrete probability density of target sample and background sample.Utilize p (i), q (i) to obtain likelihood function

$L\left(i\right)=\log \frac{\max \{p\left(i\right),\mathrm{\δ}\}}{\max \{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

In formula, δ=0.001 is placed log and is appeared as 0 situation.Judge the diversity factor of target sample feature and background sample characteristics by the variance of calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²Obtain

$\mathrm{var}(L;p,q)=\underset{i}{\mathrm{\Σ}}a\left(i\right)L^2\left(i\right)-{\left[\underset{i}{\mathrm{\Σ}}a\left(i\right)L\left(i\right)\right]}^2---\left(5\right)$

In formula, a (i) is probability density function.Thereby obtain the variance ratio formula of likelihood function

$\mathrm{VR}(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

Obtain the data of all samples, sort according to the data of sample, obtain maximum target signature and the contrast of background characteristics, obtain the Fusion Features linear combination mode of optimum.

Step 3, at first by the method that suppresses based on Corner Detection and non-maximum value, the object on the fused images of having obtained is carried out feature extraction; Then utilize the RANSAC method to remove exterior point; Calculate at last the characteristic light stream of the invariant feature point that remains with optical flow method, estimating target is in the position of next frame appearance.Adopt light stream to describe the motion at observed object, surface or edge that the motion with respect to the observer causes.Go out instantaneous velocity or the discrete picture displacement of motion by image detection.Each all has the vector set (x, y, t) of a two dimension or multidimensional constantly, the instantaneous velocity that (x, y, t) expression specified coordinate is ordered at t.If I (x, y, t) is the t intensity of (x, y) point constantly, in very short time Δ t, x, y increase respectively Δ x, Δ y:

$I(x+\mathrm{\Δx},y+\mathrm{\Δy},t+\mathrm{\Δt})=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}---\left(7\right)$

Simultaneously, consider that the displacement of two frame adjacent images is enough short, therefore

I(x，y，t)=I(x+Δx，y+Δy，t+Δt) (8)

Thereby obtain

$\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δx}}{\mathrm{\Δt}}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δy}}{\mathrm{\Δt}}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δt}}{\mathrm{\Δt}}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

In formula, V _xAnd V _yBe the speed of x and y, be called the light stream of I (x, y, t),

That image (x, y, t) is at the partial derivative of t moment specific direction.I _x, I _y, I _tRelation as follows:

I _xV _x+I _yV _y=-I _t (12)

In formula, I _x, I _yBe respectively the Grad of x direction corresponding to unique point or y direction, I _tBefore and after being, two two field pictures are at the gray scale difference value of corresponding pixel points position.Draw the light stream of unique point selected on target, estimate the position that the next frame target occurs on the basis of former frame target location.

Step 4, at first produces all positions that target may occur, and is input to the ground floor probability classification and votes, and result passes to Nearest Neighbor sorter; Secondly, Nearest Neighbor sorter carries out decision-making to it, with position with a high credibility and its intrinsic similarity output; Again, with the output of the Nearest Neighbor sorter input as the distance restraint sorter, filter out the possible position of the confirmed tracking results of distance.Output is the net result of test section at last.If the net result number of test section is too much or very few, feeds back to Nearest Neighbor sorter and distance restraint sorter stage adjustment threshold value and carry out decision-making again.

Step 5, according to the tracking results in each two field picture and testing result, judge at first whether tracking results and testing result all exist, then adopt to follow the tracks of and detect synergistic mechanism net result is adjusted; By feedback mechanism, net result is used for adjusting pursuit path at last, and upgrades each sorter in testing process.

The invention has the beneficial effects as follows: the mode that the method is utilized linear combination merges visible images and infrared image, makes target reach maximum, the outstanding characteristic information of target on image with the contrast of background on present image; Secondly, obtain clarification of objective information by the mode of target being extracted angle point, and utilize the tracking of optical flow algorithm realize target.In order further to improve the robustness of following the tracks of, added the detection sorting algorithm that the information of target and background sample is classified, again on this basis, utilize on-line study, associated treatment optical flow tracking result and detection classification results, obtain optimum target following result, more than tracking results rate of accuracy reached to 85%.

Below in conjunction with embodiment, the present invention is elaborated.

Embodiment

The online method for tracking target concrete steps that the present invention is based on the multi-source image Fusion Features are as follows:

1, multi-source image Fusion Features.

(a) to the R in visible images, G, the information of three passages of B is carried out multiple linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces a large amount of fused images.The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2，-1，0，1，2]} (1)

Wherein, R, G, the image information of three passages in the corresponding visible images of B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w _*Be corresponding linear combination parameter, span is-2 to 2.After calculating, produce altogether 625 images after linear combination, but in these images, be similar to (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄') array mode is equivalent, so after the array mode that weeds out equivalence, remaining computable linear combination image one has 215.

(b) on the image after each linear combination, calculate respectively the statistics with histogram information of target area and background area.The pixel characteristic histogram that makes target is H _obj(i), the pixel characteristic histogram of background sample is H _bg(i), calculate respectively the probability density of target and background and carry out normalization and obtain

p(i)=H _obj(i)/n _obj (2)

q(i)=H _bg(i)/n _bg (3)

Wherein, n _obj, n _bgRepresent respectively the quantity of target sample and background sample.Utilize p (i), q (i) to obtain likelihood function

$L\left(i\right)=\log \frac{\max \{p\left(i\right),\mathrm{\δ}\}}{\max \{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

Wherein, δ=0.001 is placed log and is appeared as 0 situation.Judge the diversity factor of target sample feature and background sample characteristics by the variance of calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²Obtain

$\mathrm{var}(L;p,q)=\underset{i}{\mathrm{\Σ}}a\left(i\right)L^2\left(i\right)-{\left[\underset{i}{\mathrm{\Σ}}a\left(i\right)L\left(i\right)\right]}^2---\left(5\right)$

Wherein, a (i) is probability density function.Thereby obtain the variance ratio formula of likelihood function

$\mathrm{VR}(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

By after calculating, obtain the data of all samples, sort according to the data of sample, obtain maximum target signature and the contrast of background characteristics, thereby obtain the Fusion Features linear combination mode of optimum.

2, on-line study target following.

(a) at first by based on the method for Corner Detection and the inhibition of non-maximum value, the object on the fused images of having obtained being carried out feature extraction; Then utilize the RANSAC method to remove exterior point; Calculate at last the characteristic light stream of the invariant feature point that remains with optical flow method, in order to the position of estimating target in the next frame appearance.Adopt light stream (optical flow) to be used for describing the motion at observed object, surface or edge that the motion with respect to the observer causes.A series of image detection goes out instantaneous velocity or the discrete picture displacement of motion.Each all has the vector set of a two dimension or multidimensional constantly, as (x, y, t), and the instantaneous velocity that the expression specified coordinate is ordered at t.If I (x, y, t) is the t intensity of (x, y) point constantly, in very short time Δ t, x, y increase respectively Δ x, Δ y:

$I(x+\mathrm{\Δx},y+\mathrm{\Δy},t+\mathrm{\Δt})=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}---\left(7\right)$

Simultaneously, consider that the displacement of two frame adjacent images is enough short, therefore:

I(x，y，t)=I(x+Δx，y+Δy，t+Δt) (8)

Therefore

$\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δx}}{\mathrm{\Δt}}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δy}}{\mathrm{\Δt}}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δt}}{\mathrm{\Δt}}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

V _xAnd V _yBe the speed of x and y, or be called the light stream of I (x, y, t),

That image (x, y, t) is at the partial derivative of t moment specific direction.I _x, I _y, I _tThe following expression of relation:

I _xV _x+I _yV _y=-I _t (12)

Here understand I _x, I _yBe respectively Grad, the I of x direction corresponding to unique point or y direction _tBefore and after being, two two field pictures are at the gray scale difference value of corresponding pixel points position.Draw thus the light stream of unique point selected on target, and then estimate the position that the next frame target occurs on the basis of former frame target location.

(b) utilize light stream to carry out in tracing process; often can be because be subject to illumination variation; occlusion issue and occur losing efficacy; because we are when utilizing optical flow tracking; also added the detection sorting algorithm; by a learning process to optical flow tracking result and classification results, choose optimum target following result.The multi classifier combination testing mechanism is comprised of three parts, and every part all produces positive sample and the negative sample of judgement, and wherein, negative sample adds and is used for the later stage on-line study in the negative sample set of system, and positive sample enters lower one deck sorter and further judges.At first, produce all positions that target may occur, be input to the ground floor probability classification and vote, result passes to Nearest Neighbor sorter; Secondly, Nearest Neighbor sorter carries out decision-making to it, with position with a high credibility and its intrinsic similarity output; Again, with the output of the Nearest Neighbor sorter input as the distance restraint sorter, filter out the possible position of the confirmed tracking results of distance.Output is the net result of test section at last.If the net result number of test section is too much or very few, feeds back to Nearest Neighbor sorter and distance restraint sorter stage adjustment threshold value and carry out decision-making again.

(c) according to the tracking results in each two field picture and testing result, judge at first whether tracking results and testing result all exist, then adopt to follow the tracks of with the detection synergistic mechanism net result is adjusted; By feedback mechanism, net result is used for adjusting pursuit path at last, and upgrades each sorter in testing process.

Claims (1)

1. online method for tracking target based on the multi-source image Fusion Features is characterized in that comprising the following steps:

Step 1, to the R in visible images, G, the information of three passages of B is carried out linear combination in conjunction with the thermal infrared half-tone information in infrared image again, produces fused images; The expression formula of linear combination is as follows,

F ₁≡{w ₁R+w ₂G+w ₃B+w ₄I|w _*∈[-2，-1，0，1，2]} (1)

In formula, R, G, the image information of three passages in the corresponding visible images of B difference, the thermal infrared half-tone information of the corresponding infrared image of I, w* is corresponding linear combination parameter, span is-2 to 2; Reject equivalent combinations mode (w ₁, w ₂, w ₃, w ₄)=k (w ₁', w ₂', w ₃', w ₄');

Step 2, on the image after each linear combination, calculate respectively the statistics with histogram information of target area and background area; The pixel characteristic histogram that makes target is H _obj(i), the pixel characteristic histogram of background sample is H _bg(i), calculate respectively the probability density of target and background and carry out normalization and obtain

p(i)=H _obj(i)/n _obj (2)

q(i)=H _bg(i)/n _bg (3)

In formula, n _obj, n _bgRepresent respectively the quantity of target sample and background sample, p (i), q (i) represent respectively the discrete probability density of target sample and background sample; Utilize p (i), q (i) to obtain likelihood function

$L\left(i\right)=\log \frac{\max \{p\left(i\right),\mathrm{\δ}\}}{\max \{q\left(i\right),\mathrm{\δ}\}}---\left(4\right)$

In formula, δ=0.001 is placed log and is appeared as 0 situation; Judge the diversity factor of target sample feature and background sample characteristics by the variance of calculating L (i), utilize variance computing formula var (x)=Ex ²-(Ex) ²Obtain

$\mathrm{var}(L;p,q)=\underset{i}{\mathrm{\Σ}}a\left(i\right)L^2\left(i\right)-{\left[\underset{i}{\mathrm{\Σ}}a\left(i\right)L\left(i\right)\right]}^2---\left(5\right)$

In formula, a (i) is probability density function; Thereby obtain the variance ratio formula of likelihood function

$\mathrm{VR}(L;p,q)\≡\frac{\mathrm{var}(L;(p+q)/2)}{\mathrm{var}(L;p)+\mathrm{var}(L;q)}---\left(6\right)$

Obtain the data of all samples, sort according to the data of sample, obtain maximum target signature and the contrast of background characteristics, obtain the Fusion Features linear combination mode of optimum;

Step 3, at first by the method that suppresses based on Corner Detection and non-maximum value, the object on the fused images of having obtained is carried out feature extraction; Then utilize the RANSAC method to remove exterior point; Calculate at last the characteristic light stream of the invariant feature point that remains with optical flow method, estimating target is in the position of next frame appearance; Adopt light stream to describe the motion at observed object, surface or edge that the motion with respect to the observer causes; Go out instantaneous velocity or the discrete picture displacement of motion by image detection; Each all has the vector set (x, y, t) of a two dimension or multidimensional constantly, the instantaneous velocity that (x, y, t) expression specified coordinate is ordered at t; If I (x, y, t) is the t intensity of (x, y) point constantly, in very short time Δ t, x, y increase respectively Δ x, Δ y:

$I(x+\mathrm{\Δx},y+\mathrm{\Δy},t+\mathrm{\Δt})=I(x,y,t)+\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}---\left(7\right)$

Simultaneously, consider that the displacement of two frame adjacent images is enough short, therefore

I(x，y，t)=I(x+Δx，y+Δy，t+Δt) (8)

Thereby obtain

$\frac{\∂I}{\∂x}\mathrm{\Δx}+\frac{\∂I}{\∂y}\mathrm{\Δy}+\frac{\∂I}{\∂t}\mathrm{\Δt}=0---\left(9\right)$

$\frac{\∂I}{\∂x}\frac{\mathrm{\Δx}}{\mathrm{\Δt}}+\frac{\∂I}{\∂y}\frac{\mathrm{\Δy}}{\mathrm{\Δt}}+\frac{\∂I}{\∂t}\frac{\mathrm{\Δt}}{\mathrm{\Δt}}=0---\left(10\right)$

Finally reach a conclusion:

$\frac{\∂I}{\∂x}{V}_x+\frac{\∂I}{\∂y}{V}_y+\frac{\∂I}{\∂t}=0---\left(11\right)$

In formula, V _xAnd V _yBe the speed of x and y, be called the light stream of I (x, y, t),

That image (x, y, t) is at the partial derivative of t moment specific direction; I _x, I _y, I _tRelation as follows:

I _xV _x+I _yV _y=-I _t (12)

In formula, I _x, I _yBe respectively the Grad of x direction corresponding to unique point or y direction, I _tBefore and after being, two two field pictures are at the gray scale difference value of corresponding pixel points position; Draw the light stream of unique point selected on target, estimate the position that the next frame target occurs on the basis of former frame target location;

Step 4, at first produces all positions that target may occur, and is input to the ground floor probability classification and votes, and result passes to Nearest Neighbor sorter; Secondly, Nearest Neighbor sorter carries out decision-making to it, with position with a high credibility and its intrinsic similarity output; Again, with the output of the Nearest Neighbor sorter input as the distance restraint sorter, filter out the possible position of the confirmed tracking results of distance; Output is the net result of test section at last; If the net result number of test section is too much or very few, feeds back to Nearest Neighbor sorter and distance restraint sorter stage adjustment threshold value and carry out decision-making again;

Step 5, according to the tracking results in each two field picture and testing result, judge at first whether tracking results and testing result all exist, then adopt to follow the tracks of and detect synergistic mechanism net result is adjusted; By feedback mechanism, net result is used for adjusting pursuit path at last, and upgrades each sorter in testing process.