CN103414853B - Support the sequence of video images real-time stabilization apparatus and method of multifreedom motion - Google Patents

Abstract

The invention provides a kind of sequence of video images real-time stabilization device supporting multifreedom motion, including PLD, PLD connection mode number converter, digital to analog converter, two digital signal processors, image sequence buffer, parameter configuration static memory and image sequence buffer.Present invention also offers a kind of sequence of video images real-time stabilization method supporting multifreedom motion, use image sequence interframe epipole transformation model, predicted characteristics point is in the exact position of each picture frame, by building video image predicted characteristics point set imaginary line bunch, depiction is as sequence interframe multifreedom motion, use dimensionality reduction image processing method, obtain predicted characteristics point corresponding its in the exact position of stable image, build constraint matrix, by rebuilding input picture, obtain stable output video.The apparatus and method that the present invention provides are capable of electronic steady image quick, high-precision, and the treatment effeciency of system is greatly improved.

Description

Support the sequence of video images real-time stabilization apparatus and method of multifreedom motion

Technical field

The present invention relates to a kind of sequence of video images real-time stabilization apparatus and method supporting multifreedom motion, belong to several Word technical field of video processing.

Background technology

At present, with the needs of modern military technology and industrial development, scout, monitor information war, anti-terrorism, antitheft in Status more show prominent.Owing to the scouting of fixed pedestal imaging platform and the supervision visual field are by the shadow of the optical system angle of visual field and environment Ring, be constrained to the ability as system acquisition information.For making up the deficiency of fixed pedestal imaging platform, countries in the world are scouting, prison Viewing system is placed in and expands the dynamic of optical system on mobile platform (such as dynamic load bodies such as surface car, naval vessel, aircraft, satellites) Visual field, to increase collecting information content.But, the thing followed is that the change of the vibration of dynamic load body, flow perturbation and movement velocity is right The threat of optical system imaging quality and impact.When being used for scouting and monitoring such as space flight, aviation and vehicular platform, moving base each Plant motion and random vibration will have a strong impact on stablizing of video image.Use the method for machinery or photorefractive crystals, although can reach Certain is steady as effect, but with the raising of stable accuracy, it will bringing system hardware cost to be that geometry magnitude increases, this is Image stabilization system is difficult to bear.Use electronic steady image as the follow-up device of machinery or optical stabilization platform, can further improve Steady as precision.Electronic steady image has advantages such as volume is little, lightweight, low in energy consumption and intelligent, but the complexity of its algorithm and steady As the contradiction of precision remains the bottleneck of its application of restriction.

Owing to motion during continuously acquiring image sequence for the imaging platform has many free random nature, to electronics For image stabilization system, its multivariant compound movement parameter Estimation of image sequence interframe overall situation is the decision of image stabilization system performance Sexual factor.According to steady as model, use globe motion parameter estimation technique can be roughly divided into translation random motion two dimension ginseng at present Number detection technique and complicated multiple degrees of freedom random motion parameter detecting technology.Owing to the complexity of translational motion parameter Estimation is relative Relatively low, the sequence of video images unstability that caused by it is easier to process, and has that algorithm is simple, easily realizes and the spy such as real-time Point, but due to its to sequence of video images between multifreedom motion lack detectability.When sequence of video images interframe comprises When having the motion of on-plane surface geometry and camera in many free vibration of 3d space and cause the factors such as parallax, two dimensional model is difficult to send out Wave effect, it is impossible to be applicable to the steady picture of high-precision electronic.Complicated multiple degrees of freedom random motion parameter detecting technology, uses similar mostly Conversion motion model, only minority algorithm are to use affine Transform Model, or distant view transformation model.Owing to its utilization is more multiple Miscellaneous 3D rendering model processes the above-mentioned difficulty being previously mentioned, and image real time transfer amount is big, and the digital image stabilization method of current 3D model is By tracking one sparse feature point set in dynamic image change procedure, and utilize the corresponding pass of image sequence interframe characteristic point System, recovers the 3D attitude of video camera, builds newly stable by processing smooth camera motion path and re-projection 3D point Picture frame.But, due to the introducing of more complicated steady picture model, bring various defect can to the hardware unit of the realization of algorithm Property.As used SFM (cross matrix) method, actually process nonlinear problem, be typically to use light beam to adjust to solve Problem, and based in extensive trace point and three-dimensional information process, the efficiency degradation of algorithm can be caused, affect system Real-time.Meanwhile, characteristic point error hiding and when video image scene change complexity less than model require change when, Owing to being difficult to extract enough characteristic informations, image stabilization system can be made to be difficult to play its effect, even there will be motion vector detection Mistake.In addition, the small movement of camera or almost plane scene geometry (object of distant place) also are difficult to determine the change of three-dimensional position Change.

Content of the invention

The technical problem to be solved in the present invention is to provide a kind of sequence of video images height that can support multifreedom motion Precision, real-time stabilization apparatus and method.

In order to solve above-mentioned first technical problem, the technical scheme is that offer one supports multifreedom motion Sequence of video images real-time stabilization device, it is characterised in that: include PLD FPGA, PLD FPGA connect modulus converter A/D, digital to analog converter D A, digital signal processor DSP 1#, digital signal processor DSP 2#, Image sequence buffer DDR3, parameter configuration static memory SRAM and image sequence buffer DDR3；

PLD FPGA is responsible for carrying out subregion to image；To the reconstruction from digital signal processor DSP 2# Picture frame carries out reconfiguring new video, and uses tcp/ip communication agreement, generates standard to newly stablizing sequence of video images Digital video frequency flow exports；

Digital signal processor DSP 1# and digital signal processor DSP2#, uses parallel pipeline working method to video Image sequence is processed；Wherein, image information is analyzed and processes by digital signal processor DSP 1#, including image district The detection of characteristic of field point and selection, calculate image interframe constraint matrix, set up the core transformation model of image interframe and carry out characteristic point Position prediction；Digital signal processor DSP 2# utilizes the predicted characteristics point sequence that digital signal processor DSP 1# provides, and builds Video image characteristic point set imaginary line bunch, meanwhile, uses view data dimension-reduction treatment method to be filtered imaginary line bunch Process, obtain predicted characteristics point corresponding its in the exact position of stable image, build constraint matrix, be originally inputted figure for calculating As characteristic point and the corresponding exact position stablizing characteristics of image point set, and current image frame is rebuild, meanwhile, by rebuild Image sends PLD FPGA to；

Image sequence buffer DDR3, for during Stabilization of video sequences, to present frame, associated frame and mediant According to interim storage；

Parameter configuration static memory SRAM, before processing for Video Stabilization, stores to the systematic parameter setting.

In order to solve above-mentioned second technical problem, the technical scheme is that offer one supports multifreedom motion Sequence of video images real-time stabilization method, it is characterised in that: use image sequence interframe epipole transformation model, it was predicted that characteristic point In the exact position of each picture frame, and by building video image predicted characteristics point set imaginary line bunch, depiction is as sequence frame Between multifreedom motion, meanwhile, use dimensionality reduction image processing method, obtain predicted characteristics point corresponding its at stable image Exact position, thus build constraint matrix, be used for calculating original input picture feature point set its stable image characteristic point corresponding The exact position of collection, rebuilds finally by input picture, obtains stable output video.

Preferably, the conversion of multiframe epipole and average decision making algorithm is utilized to realize the height to current image frame feature point set position Accuracy prediction, and by the variable condition of frame institute predicted characteristics point position each to image sequence, obtain video camera at shooting process In multifreedom motion information.

Preferably, the method by setting up image sequence predicted characteristics point set virtual track set of curves, represents that 3d space is many The motion state of the image sequence interframe acquired in free degree motion cameras；Use dimension-reduction treatment method, by predicted characteristics point Collection virtual track set of curves projects respectively at X and Y-direction, by carrying out 2D smoothing processing respectively to drop shadow curve, obtains Predicted characteristics point set is in the smooth change of sequence of video images each frame position coordinate, so that it is determined that original input picture feature point pairs Should its core constraint matrix in stable picture position.

Preferably for the static nature point in scene, by original input picture Feature point correspondence its at stable image The core constraint matrix of position, uses epipole conversion and average decision making algorithm, accurately estimates that original input picture is stably schemed with corresponding As characteristic point position change.

Preferably for behavioral characteristics point process, based on moving object in scene at image sequence interframe movement track Stationarity, determines the feature of minimum external force constraint suffered by object, sets up behavioral characteristics point dynamics mathematical model, solve in scene Motion feature point is at each image frame position of image sequence.

Preferably, in stable image process, choosing and following the tracks of of key feature points, it is distributed at image-region based on it Uniformity and conspicuousness rule；The replacement policy of characteristic point is dynamically pre-with characteristic point position based on the continuity of virtual track curve Survey.

Preferably, the stablizing of sequence of video images, uses below step to realize:

Step 1: set and need sequence of video images to be processed as f_s, f_tFor present image.To f_sIn each two field picture, first Pass to digital signal processor DSP #1 after being divided into 16 image regions by PLD FPGA to carry out Reason, in each region, uses KLT point track algorithm to select ten characteristic points, and image behind, by feature point tracking skill The all characteristic points chosen are tracked by art, constitute f_sA feature point set sequenceWhereinRepresent video sequence The ith feature point that s two field picture is chosen；

Step 2: to video sequence f_s, with present image f_tCentered on, continuous 17 two field pictures constitute a process list being associated Unit, it was predicted that present frame characteristic point accurate coordinate position in the picture.Process picture frame f to current_t, track algorithm is obtained this frame It with the feature point set sequence of its forward and backward adjacent 8 frame continuous processing images, is designated as: ${\mathrm{\Ψ}}_s^i=\{{\mathrm{\Ψ}}_{t-8}^i,...,{\mathrm{\Ψ}}_{t-1}^i,{\mathrm{\Ψ}}_t^i,...,{\mathrm{\Ψ}}_{t+1}^i,...,{\mathrm{\Ψ}}_{t+8}^i\},$ Wherein: t-8≤s≤t+8；

Step 3: based on area distribution and conspicuousness feature, at the feature point set of processing unitIn sequence, to each frame The feature point set of image, selects eight key feature points therein, calculates currently processed picture frame f_tIt is adjacent forward and backward eight frames The core constraint transformation matrix F of image^{S, t}；

Step 4: according to image Nuclear goemetry constraint principles, uses epipole conversion method, byAnd F^{S, t}Calculate each frame characteristic pointAt image f_tIn projection lineAnd byIntersection point or the mean value of each position of intersecting point, as characteristic point i at image f_t The prediction of middle position, thus constitutive characteristic point setPredict point set at present image

Step 5: often gather a new images and enter video sequence, then can make: t=t+1, repeat step 2～step 4, Obtain predicted characteristics sequence of point sets:And as 1 ＜ s ＜ 8, orderSo can obtain To a continuously predicted characteristics sequence of point sets ${\mathrm{\Ω}}_s^i=\{{\mathrm{\Ω}}_1^i,{\mathrm{\Ω}}_2^i,...,{\mathrm{\Ω}}_t^i,...{\mathrm{\Ω}}_n^i,...\};$

Step 6: digital signal processor DSP #1 will predict sequence of point sets continuouslyPass to digital signal processor DSP # 2, digital signal processor DSP #2 using frame number s as time Z axis,Middle coordinate at two dimensional image respectively constitutes X-axis and Y Axle, forms three-dimensional system of coordinate, according toThe frame number of point sequence and at the coordinate (x of each frame_i, y_i) value, determine each predicted characteristics Put the position in newly-built coordinate system, and the corresponding points by each picture frame be linked to be curve, constitute the imaginary line bunch along time shaft, And constantly gathering new video image in time, its imaginary line constantly extends；The corresponding predicted characteristics point institute structure of each frame The smoothness of every the curve becoming, reflects degree of stability in sequence of video images for this feature point, whole imaginary line Bunch then reflect degree of stability in shooting process for the video camera；

Step 7: digital signal processor DSP #2 is along Z-direction, by each curve of imaginary line bunch respectively at X Direction and Y-direction project, and with present image as processing center, forward and backward adjacent totally 11 two field pictures are a processing unit, Use convolution algorithm method successively, it is achieved smoothing filtering operation is carried out to curve, available predicted characteristics point setCorresponding Critical sequences:

Step 8: according to primitive image features sequence of point setsWith in critical sequencesCorresponding relation, according to its region Distribution and conspicuousness feature, exist againWithIn, select eight key feature points therein, calculate current original image frame f_tCharacteristic pointIt is adjacent forward and backward five two field pictures correspondingCore constraint transformation matrixT-5 ＜ s ＜ t+5；

Step 9: according to image Nuclear goemetry restriction relation, uses epipole conversion method and average decision making algorithm, byEstimate Go out present imageIt is at the location point of stabilizer frame

Step 10: to the current image frame processing, if characteristic point i is corresponding to the moving object in non-static image, then divide Analyse its motion feature, use dynamics Mathematical Modeling Methods, resolve its smooth motion status flag and put, rightMiddle corresponding points Position be adjusted correspondingly；

Step 11: to frame under process f_t, by starting the feature set selectedObtained right with after above-mentioned process Answer feature setPosition relationship, to present image f_tRebuild, and the image that will rebuildPass to PLD FPGA；

Step 12: PLD FPGA is to the image rebuildIt is reassembled into new video, and use TCP/ New stable sequence of video images is generated the output of standard digital video stream by IP communication protocol.

A kind of sequence of video images real-time stabilization apparatus and method supporting multifreedom motion that the present invention provides are with now There is technology to compare, there is following beneficial effect:

1st, propose to use the constraint of video image interframe Nuclear goemetry and epipole transformation model, jointly joined by adjacent multiple image With, it is achieved the prediction to present frame characteristic point position, the matching precision of characteristic point can be significantly increased, overcome picture quality and with The error that machine factor and the parameter of camera own are brought, improves the robustness of algorithm.

2nd, dimension-reduction treatment method is used, to image sequence predicted characteristics point set virtual track family of curves in X and Y-direction respectively Project, and by carrying out 2D smoothing processing respectively to drop shadow curve, to obtain feature point set at each frame of sequence of video images The smooth change of interior position, can remove because video camera multifreedom motion causes image sequence interframe unstability dexterously, simultaneously big Amplitude ground improves system treatment effeciency.

3rd, the stationarity according to the movement locus in image interframe for the moving object determines the pact of the minimum external force suffered by object The feature of bundle, propose to set up motion feature point position solves Mathematical Modeling, tradition can be avoided steady based on static situation interpolation The steady picture error determined moving object and bring, it is achieved under camera and objects in images moving condition simultaneously, the image to video Sequence enter line stabilization.

4th, steady picture Processing Algorithm complexity and real time problems can effectively be solved, it is achieved electronic steady image quick, high-precision.

The apparatus and method that the present invention provides overcome the deficiencies in the prior art, and the matching precision of characteristic point is high, algorithm Robustness is high, is capable of electronic steady image quick, high-precision, the treatment effeciency of system has been significantly increased.

Brief description

The sequence of video images of the support multifreedom motion that Fig. 1 provides for the present invention is steady in real time as apparatus structure signal Figure；

Fig. 2 is image interframe core constraint principles figure；

Fig. 3 is characterized a position prediction schematic diagram；

Fig. 4 is behavioral characteristics point position smooth change schematic diagram.

Detailed description of the invention

For making the present invention become apparent, hereby with a preferred embodiment, and accompanying drawing is coordinated to be described in detail below.

The sequence of video images real-time stabilization device schematic diagram of the support multifreedom motion that Fig. 1 provides for the present invention, institute The sequence of video images real-time stabilization device of the support multifreedom motion stated includes PLD FPGA, picture signal Modulus converter A/D and digital to analog converter D A, two digital signal processor DSP#1 and DSP#2, image buffer storage dynamic memories The critical pieces such as device DDR3 and parameter configuration static memory SRAM.

Wherein, FPGA PLD is the central control unit of device, and control is whole steady as the workflow of device Journey, is responsible for carrying out subregion to image simultaneously.Finally, carry out reconfiguring new video to from DSP2# reconstruction picture frame, and adopt Use tcp/ip communication agreement, generate the output of standard digital video stream to newly stablizing sequence of video images；Processor DSP1# and by Reason device DSP2# uses parallel pipeline working method to process sequence of video images.Wherein, processor DSP1# is to image Information is analyzed and processes, including feature point detection and selection in region, calculates image interframe constraint matrix, sets up picture frame Between core transformation model and carry out characteristic point position prediction；Processor DSP2# utilizes the predicted characteristics point sequence that DSP1# provides, Build video image characteristic point set imaginary line bunch, meanwhile, use view data dimension-reduction treatment method to carry out imaginary line bunch Filtering process, obtain predicted characteristics point corresponding its in the exact position of stable image, build constraint matrix, be used for calculating original defeated Enter image characteristic point and the corresponding exact position stablizing characteristics of image point set, and realize rebuilding present frame, meanwhile, will weight The image built sends FPGA to；DDR3 builds image sequence buffer, for steady as during to current image frame and related regard Frequently the interim storage of image sequence, data.SRAM is for surely as the systematic parameter of front setting stores.

A kind of sequence of video images real-time stabilization method supporting multifreedom motion, it utilizes the conversion of multiframe epipole and puts down Equal decision making algorithm realizes the high-precision forecast to current image frame feature point set position, and is predicted by frame each to image sequence The changing condition of characteristic point position, obtains multifreedom motion information in shooting process for the video camera.It is different from tradition spy Levying the search strategy of Point matching, the present invention proposes to utilize core restriction relation, epipole conversion and the multi-frame mean of image sequence interframe Decision making algorithm realizes the high accuracy method of estimation of each frame image features point position.As in figure 2 it is shown, set photographed scene three dimensions 1 P, if same camera shoots at two diverse locations respectively, then this projection on its imaging plane I and I ' For p and p ', i.e. p and p ' is corresponding match point, and video camera two viewpoint C and C ' are respectively e with the intersection point of video camera imaging plane It with e ', is referred to as the core of two planes of delineation.Ray l '_pIt is referred to as some core line in plane of delineation I ' for the p, similarly, core line l_pAlso There is similar definition.Nuclear goemetry principle according to image interframe, then put ray l ' on image plane I ' for the p_pMeet one linearly Conversion, it may be assumed that

l′_p=Fp (1)

Wherein, F is 3 × 3 matrixes that order is 2, and the match point p ' of p is at its core line l '_pOn, intrinsic:

p′^TFp=0 (2)

Based on above-mentioned Nuclear goemetry conversion general principle, to sequence image frame, as it is shown on figure 3, as supposed p^tIt is static 3D Scene point in t in the projection of imaging plane, corresponding points p^tAt subpoint on imaging surface for moment s=t-1 and s=t+1 It is respectively p^t-1And p^t+1, its projection core line in t imaging surface, it is respectively as follows:With Wherein F^{S, t}Being exactly the core constraint fundamental matrix of image sequence interframe, this matrix represents present image f_tIt is adjacent image f_sPact Bundle relation.For image stabilization system, due to the correlation of the height of continuous videos interframe, when 3D shared by the core line of adjacent image frame During same in scene object point, they will intersect at a single-point at present imageFor each point of t ≠ s, all can be by l_t×l_sBe given, and this point is exactly projection (picture point) on present image for the scene 3D object point.Therefore, based on Nuclear goemetry relation, Do not need to extract the parameter information of characteristic point 3D position concrete in the scene and camera, utilize current image frame to be adjacent frame Restriction relation, just can calculate to a nicety this coordinate position at present frame.In general, utilize epipole conversion to predict to work as The coordinate of front frame characteristic point, it is only necessary to two consecutive frames just can realize.Because using more image to participate in, its core line is being worked as Front image still can meet at a bit.But owing to image existing the track algorithm of noise and employing, the error etc. of model, can affect The accuracy of required position coordinates.For the position of the characteristic point that more calculates to a nicety, the present invention uses adjacent multiple image, asks The crosspoint of its core line, and the position coordinates in all crosspoints is averaged, calculate the exact position of its characteristic point, improve To characteristic point forecasting reliability and robustness.In characteristic point position prediction implementation method, first to the image sequence processing f_sA selected stack features pointThen KLT point track algorithm is utilized to follow the tracks of this initial characteristics point set along video flowing, and based on Characteristic point distributing homogeneity and conspicuousness, choose 8 key feature points, utilizes the corresponding pass of image sequence interframe key feature points System, estimates video sequence current image frame f_tWith its consecutive frame f in time_sThe basic constraint matrix of (t-8 ＜ s ＜ t+8) F^{S, t}.Each characteristic point uses the method for core line crosspoint mean place asked, it is achieved present frame feature point set position accurately pre- Survey, thus obtain the more accurate predicted characteristics point set in positionIn the present invention, have employed KLT (Kanade-Lucas- Tomasi) putting track algorithm, this algorithm is by carrying out Gray-scale Matching based on the translation model of 2D, it is achieved the tracking of characteristic point, and Carry out selecting based on the characteristic point of track algorithm, improve feature point tracking quality.It passes through characteristic point selection algorithm at initial figure Select characteristic point in Xiang, then utilize translation model to carry out feature point tracking, for the characteristic point tracing on N width image, Enter Line Continuity by affine model to judge, the characteristic point following the tracks of mistake is rejected.At research different images in KLT algorithm Between matching problem when, by calculate two translation windows gray scale residual errors, and find minimum residual error realize coupling.

After being accurately obtained predicted characteristics point set, use and set up image sequence predicted characteristics point set virtual track set of curves Method, represent the motion state of image sequence interframe acquired in 3d space multifreedom motion video camera, by projection Geometric locus bunch carries out projecting, smoothing processing, obtains the settling position of sequence of video images predicted characteristics point.In the present invention, By the conversion constraint matrix setting up original input picture characteristic point with stablize image sequence characteristic pointEstimate original defeated Enter characteristics of image point set in the corresponding position stablized in image, and by this change in location relation of feature point set, it is achieved right Original input picture sequence is rebuild.Conversion constraint matrixIt is by building fantasy sport track bunch and being located by related Reason realizes.Start in system work, be the fortune in 3d space camera multifreedom motion acquired image sequence for the Precise Representation Dynamic status information, can select sufficient amount of characteristic point by the every two field picture from image sequenceAnd at the some frames starting, Coordinate value first with the characteristic point followed the tracks of initializes these future positions, such as: when 0 ＜ s ＜ 8, orderFollow-up Picture frame, then pass through said method, the coordinate position of characteristic point followed the tracks of be predicted, thus build its virtual rail continuous Trace bunch.It should be noted that choosing of characteristic point is by the image of process is divided into some pieces, and from block, choosing Take some characteristic points be tracked and calculate, so that tracing point can relatively evenly be distributed in frame, effectively make the track of composition Line changes the motion change representing that 3D scene is thrown on image in two dimensional surface.Phase for 3d space multifreedom motion Machine,Comprised feature point set sequence is shake, because it is to be obtained the feature point set on sequence image by unstable camera Sequence passes through F^{S, t}It is converted to.As long asThere is (i.e. video is not blocked by object or blocks) in video sequence image Also will be withEqually will continue.In order to obtain stable predicted characteristics sequence of point sets, the present invention (selects suitable by Fuzzy Template When parameter) right in real timeThe track being projected in 2D plane smooths, and smoothing processing separately in level and Carrying out in vertical direction, the virtual track line that all corresponding predicted characteristics points are constituted passes through Fuzzy Template process, equals composition SteadyAs long as they keep related to the image sequence being absorbed, generation and calculating to trajectory continue to, once institute The characteristic point followed the tracks of exits handled picture frame, then the dummy line that these predicted characteristics points are constituted just stops, and also calculates Stop.In this process, error threshold can be generated by setting epipole, build to the selection of characteristic point with exit machine System and strategy.The feature point set selected by original input picture is utilized to put downObtain after smoothCalculate that another is basic MatrixThis fundamental matrixThe raw video image f unstable by input will be realized_sCharacteristic pointWith these point At new stable imageOn be associated and retrain.Similarly, basic matrixCan useWithIn correspondence Point, is calculated by 8 algorithms.Use the method that epipole as previously described is changed again, with obtained matrix WithEstimate the exact position at stabilizer frame corresponding thereto for these characteristic pointsFor improving precision, the present invention still uses core The method of some conversion, and use | t-s |≤5 condition, seek the method that core line intersection point is average, determine present frame characteristic point at stable figure Accurate estimation as frame position coordinateFinally utilizeWithMapping relations, a kind of defeated by source for image stabilization system supply Enter frame to the constraint stablizing output frame, it is achieved line position physicsl correct is entered to the pixel of input picture frame.Owing to the present invention is surely It during Xiang, is to be obtained by calculatingCome and build trajectory bunch, and separately Fuzzy smooth operation in 2D plane, Therefore, its coordinate is not required to corresponding in the projection of attainable camera with scene 3D point.But, the some position after these are smooth Putting not is final point of safes position, in addition it is also necessary to by one basic transition matrix of definitionAccurately estimate input figure As characteristic point stablizes the position in image in correspondence, finally realize rebuilding new image frame.Matrix definition due to each two dimension One effective several picture, matrixActually simulate a physically attainable camera to regard, it is achieved by jitter points CollectionPoint set with smooth motionRelated, the video correction being obtained by it is optimized by camera in 3d space smooth motion.

In the present invention, owing to the behavioral characteristics point in scene is unsatisfactory for the core restriction relation of image sequence interframe, therefore It for the behavioral characteristics point process in scene, is based on steady at image sequence interframe movement track of moving object in scene Property, determine minimum external force constraint feature suffered by object, set up behavioral characteristics point dynamics mathematical model, solve motion spy in scene Levy a little at each image frame position of image sequence, to solve the stable problem at sequence of video images for the non-static scene motion object, Avoid the error that tradition Video stabilization utilizes the object background information of irrelevant motion and interpolation arithmetic to be brought.General tradition is steady As algorithm, when a certain moving object in the corresponding scene of the point followed the tracks of, owing to these points can not reflect the motion of video camera, And usually lost by Video stabilization.And the image-region at moving object place, it is by the picture of the static scene of this areas adjacent Point obtains its steadiness parameter.When moving object place image-region and the spacing of its static scene picture point are relatively big or work as moving object When the depth of field difference of body and its background is very important, using said method, the moving object of picture frame will be difficult to realize stablizing. In the present invention, utilize dynamic (dynamical) relevant nature, the in the scene characteristic point of moving object, if movement locus in image sequence Have stationarity characteristic, then moving object must be retrained by minimum external force, thus for prediction behavioral characteristics point at image sequence Each frame position provides condition.

Consider a bit, a dynamic point p in time-series image frame s^s, it still exists at picture frame t projected core line Time s, it may be assumed that l^s=F^{S, t}p^s.Core restriction relation according to image sequence, it is intended that in picture frame t, with dynamic point p^sCoupling Some p^tThis core line l should be positioned at^sOn.And in fact, due to characteristic point be motion, it is impossible to meet epipole constraints, Fig. 4 Illustrate the sight of three frame consecutive frames.In the present invention, estimate that the method for present image behavioral characteristics point position coordinates is base Dynamic behaviour in behavioral characteristics point.Assume that behavioral characteristics point is easy motion in the motion of image sequence interframe, then it Motion comes from the active force forcing its motion small, according to dynamics relevant nature, within the unit interval, its velocity u^{S, t}=q^{S+1, t}-q^{S, t}Change that must be little as much as possible.Us so can be made to derive q^{S, t}Have enough constraints.Represent with E Following time difference matrix:

$\left(\begin{array}{ccccc}-1& 1& ...& 0& 0\\ 0& -1& ...& 0& 0\\ ...& ...& ...& ...& ...\\ 0& 0& ...& -1& 1\end{array}\right)\·\left(\begin{array}{c}..\\ q^{s-1,t}\\ q^{s,t}\\ q^{s+1,t}\\ ..\end{array}\right)=\left(\begin{array}{c}..\\ u^{s-1}\\ u^s\\ u^{s+1}\\ ..\end{array}\right)---\left(3\right)$

If representing with D and derivation carried out to matrix E, when to velocityWhen operating, at this moment difference equation matrix is just Acceleration will be producedHere,It is the vector at time s.Therefore, minimum force institute is come from order to obtain The movement locus causing, in order to every bit q^{S, t}It is positioned at its corresponding core line l^sOn, it is necessary to solve following constraint:

$\underset{q^{\stackrel{\→}{s},t}}{\min }{\left|\right|\mathrm{\Γ}{q}^{\stackrel{\→}{s},t}\left|\right|}^2,$ Wherein: $\left\{\begin{array}{cc}{\left(l^s\right)}^T{q}^{s,t}=0,& s\≠t\\ q^{s,t}=p^t,& s=t\end{array}\right.---\left(4\right)$

Here, Γ represents matrix of second derivatives DE.Use Lagrange multiplier operator rule can simplify the problems referred to above, from And become and solve following linear system:

$\left\{\begin{array}{c}\left({\mathrm{\Γ}}^T\mathrm{\Γ}\right)\·q^{\stackrel{\→}{s},t}+C^T\·\mathrm{\λ}=0\\ C\·q^{\stackrel{\→}{s},t}=b\end{array}\right.---\left(5\right)$

In above formula equation group, C and b is to represent (4) formula linear restriction matrix and vector.Use above-mentioned equation group, just can be solely On the spot solve each some q^{S, t}Horizontal and vertical coordinate.

Steady as during, the choosing and following the tracks of of characteristic point, based on it in the uniformity of image-region distribution and conspicuousness Rule；In view of the change of scene with the factor such as block, its replacement policy is the continuity and feature based on virtual track curve Point position dynamic prediction.Owing to can identify with surrounding its very little pixel window based on the feature point tracking algorithm of KLT Each point, so these points of coupling interframe, can be realized by the similarity comparing these windows.In order to represent the fortune of frame Dynamic state, each frame in sequence of video images, optional hundreds of such characteristic point is tracked.To these characteristic points Process, can be for recovering this fundamental matrix F^{S, t}WithEnough information is provided.

A kind of sequence of video images real-time stabilization side supporting multifreedom motion proposing based on said method, the present invention Method, uses below step to realize:

(1) need sequence of video images to be processed is set as f_s, f_tFor present image.To f_sIn each two field picture, by FPGA Pass to DSP#1 process after being divided into 16 image regions, in each region, use KLT point track algorithm The all characteristic points chosen are tracked by selected ten characteristic points, and image behind by feature point tracking technology, constitute f_sA feature point set sequenceWhereinRepresent the ith feature point that the s two field picture of video sequence is chosen.

(2) to video sequence f_s, constituted a processing unit being associated with continuous 17 two field pictures, to predict that present frame is special Levy accurate coordinate position a little in the picture.Process picture frame f to current_t, track algorithm is obtained this frame forward and backward adjacent with it The feature point set sequence of 8 frame continuous processing images, is designated as: ${\mathrm{\Ψ}}_s^i=\{{\mathrm{\Ψ}}_{t-8}^i,...,{\mathrm{\Ψ}}_{t-1}^i,{\mathrm{\Ψ}}_t^i,...,{\mathrm{\Ψ}}_{t+1}^i,...,{\mathrm{\Ψ}}_{t+8}^i\},$ Wherein: t-8 ≤s≤t+8。

(3) based on area distribution and conspicuousness feature, at the feature point set of processing unitIn sequence, to each two field picture Feature point set, select eight key feature points therein, calculate currently processed picture frame f_tIt is adjacent forward and backward eight two field pictures Core constraint transformation matrix F^{S, t}。

(4) according to image Nuclear goemetry constraint principles, epipole conversion method is used, byAnd F^{S, t}Calculate each frame characteristic point At image f_tIn projection lineAnd byIntersection point or the mean value of each position of intersecting point, as characteristic point i at image f_tIn The prediction of position, thus constitutive characteristic point setPredict point set at present image

(5) often gather a new images and enter video sequence, then can make: t=t+1, repeat step (2)～(4), i.e. available Predicted characteristics sequence of point sets:As 1 ＜ s ＜ 8, orderSo we can get one Continuous print predicted characteristics sequence of point sets

(6) DSP#1 will predict sequence of point sets continuouslyPass to DSP#2, DSP#2 using frame number s as time Z axis, Middle coordinate at two dimensional image respectively constitutes X-axis and Y-axis, forms three-dimensional system of coordinate, according toThe frame number of point sequence and respectively Coordinate (the x of frame_i, y_i) value, determine each predicted characteristics point position in a coordinate system, and the corresponding points by each picture frame are linked to be song Line, constitutes the imaginary line bunch along time shaft, and constantly gathers new video image in time, and its imaginary line constantly prolongs Stretch.The smoothness of every curve that the corresponding predicted characteristics point of each frame is constituted, reflects this feature point at video image Degree of stability in sequence, whole imaginary line bunch then reflects degree of stability in shooting process for the video camera.

(7) DSP#2 processor is along Z-direction, by each curve of imaginary line bunch respectively in X-direction and Y-direction Project, and with present image as processing center, forward and backward adjacent totally 11 two field pictures are a processing unit, use volume successively Long-pending operation method, it is achieved smoothing filtering operation is carried out to curve, available predicted characteristics point setCorresponding critical sequences:

(8) according to primitive image features sequence of point setsWith in critical sequencesCorresponding relation, according to its area distribution With conspicuousness feature, again existWithIn, select eight key feature points therein, calculate current original image frame f_tSpecial Levy a littleIt is adjacent forward and backward five two field pictures correspondingCore constraint transformation matrixT-5 ＜ s ＜ t+5.

(9) according to image Nuclear goemetry restriction relation, epipole conversion method and average decision making algorithm are used, byEstimate Present imageIt is at the location point of stabilizer frame

(10) to the current image frame processing, if characteristic point i is corresponding to the moving object in non-static image, then it is analyzed Motion feature, uses dynamics Mathematical Modeling Methods, resolves its smooth motion status flag and put, rightThe position of middle corresponding points Put and be adjusted correspondingly.

(11) to frame under process f_t, by starting the feature set selectedThe corresponding spy obtained with after above-mentioned process CollectionPosition relationship, to present image f_tRebuild, and the image that will rebuildPass to FPGA.

(12) FPGA is to the image rebuildIt is reassembled into new video, and uses tcp/ip communication agreement, by new Stablize sequence of video images and generate the output of standard digital video stream.

In implementation process, the following main equipment of employing:

(1) PLD FPGA: use the XC6SLX16-3 CSG324 of Xilinx company.324 are had to draw Pin, 232 usable pins, 2278 slices, each slice comprise 4 CLB, and each CLB comprises the look-up tables of 46 inputs (LUT), totally 36448 6-input LUTs.The built-in hardware resource of this FPGA has 32 DSP48A1, two MCB (storage tubes Reason block) and the built-in RAM of 576K.

(2) digital signal processor DSP: use TMS320DM6446, it is double that this model has DSP (DM64X) and ARM simultaneously SOC (the System on Chip) flush bonding processor of kernel.This kind of processor had both had ARM9 kernel, can run Windows or (SuSE) Linux OS, have again the DSP core of high primary frequency, can quickly run audio/video encoding/decoding, pattern knowledge Etc. not various complicated video processnig algorithms.DaVinci processor also has advantage low in energy consumption simultaneously, can be widely applied to each Kind of battery environment of powering uses.

(3) dedicated video A/D: use the picture signal A/D converter that model is AD9826, have R_G_B3 input Passage, precision is 16bits, and gathering highest frequency is 15MSPS.

(5) SRAM SRAM: memory device uses the SRAM that model is K6R4008v1d, storage size For 512KB, data-line width is 8bit, and storage cycle is 8ns, and maximum throughput is 1Gb/s.

(6) it is MT41J64M16 that dynamic memory DDR3: memory device uses model, and memory capacity is that size is 1Gb, 96-Ball FBGA encapsulates, and memory space is 8Meg*16*8Banks, burst transfer position 512bit of this DDR3, burst A length of 8, maximum operation frequency 533MHz, realize zero access by double data multiplying power.The design clock uses The differential clocks input of 400MHz, data access rate is up to 800Mb/s.

(7) ethernet controller: use 88E1111 network adapter, may operate in 1000M pattern, data transmission rate Up to 1Gb/s.

The sequence of video images of support multifreedom motion that the present invention provides is steady in real time utilizes embedded image as device Processing means, uses the Nuclear goemetry restriction relation of image interframe to realize the accurate prediction of characteristic point position, and uses view data Dimension-reduction treatment new method, obtains original input picture sequence and the restriction relation stablizing image sequence, and input picture carries out weight Build, it is achieved multifreedom motion sequence of video images real-time stabilization.

By Nuclear goemetry restriction relation and the epipole transformation model of image sequence interframe, it is achieved multiframe participates in characteristic point position Prediction, improves the matching precision of characteristic point, meanwhile, uses dimension-reduction treatment technology, and simplification processes the complexity of view data, improves The real-time of system.Algorithmically do not need to recover the shooting attitude of the 3D point position of explicit property or 3D camera, but by building The method of vertical image sequence predicted characteristics point set virtual track family of curves, represents acquired in 3d space multifreedom motion video camera The motion state of image sequence interframe, and by projection and the filtering process of simple 2D virtual track family of curves, obtain source defeated Enter image and the Nuclear goemetry restriction relation stably exporting image, complete to realize the reconstruction to input original sequence, thus complete Become 3D scene point in the physical correction of image planes.Method proposed by the invention, can realize to image sequence multifreedom motion Realize physical correction, be avoided that again some drawbacks of what 3D digital image stabilization method was brought simultaneously, use dual processors parallel pipeline simultaneously Data processing method, provides a kind of method for solving steady picture precision and this contradiction of real-time.

In terms of the demand angle of video stabilization, the main purpose of steady picture is to carry to greatest extent on the premise of ensureing real-time The stability of high video image, meets the vision requirement of the mankind.Traditional electronic image stabilization method, improves surely as precision will carry reflex Miscellaneous computation model and heavy amount of calculation.The present invention utilizes image sequence Nuclear goemetry theoretical, is possible not only to feature is greatly improved The precision of prediction of point coordinates and stability, and by carrying out dimension-reduction treatment to view data computing, reduce amount of calculation, improve system Treatment effeciency.Simultaneously in terms of hardware design, use a plurality of pipeline processing mode, build unified image/video and substantially tie Structure and expression model and corresponding processing method thereof, to realize the multivariant video camera complicated with better simply model treatment Shake, it is to avoid numerous and diverse computing, improves precision and the robustness of image stabilization system.Support multiple degrees of freedom electronics proposed by the invention Digital image stabilization method, is to realize the key that dynamic load body imaging system obtains high-quality video image, contributes to promoting moving base carrier and takes the photograph Camera obtains the ability of information, to China's national defense, anti-terrorism and antitheft etc. have important Research Significance.

Claims (8)

1. the sequence of video images real-time stabilization device supporting multifreedom motion, it is characterised in that: include able to programme patrolling Collecting device FPGA, PLD FPGA connects modulus converter A/D, digital to analog converter D A, digital signal processor DSP1#, digital signal processor DSP 2#, parameter configuration static memory SRAM and image sequence buffer DDR3；

PLD FPGA is responsible for carrying out subregion to image；To the reconstruction image from digital signal processor DSP 2# Frame carries out reconfiguring new video, and uses tcp/ip communication agreement, generates standard digital to newly stablizing sequence of video images Video flowing exports；

Digital signal processor DSP 1# and digital signal processor DSP2#, uses parallel pipeline working method to video image Sequence is processed；Wherein, image information is analyzed and processes by digital signal processor DSP 1#, including image-region is special Levy a detection and select, calculate image interframe constraint matrix, set up the epipole transformation model of image interframe and carry out characteristic point position Put prediction；Digital signal processor DSP 2# utilizes the predicted characteristics point sequence that digital signal processor DSP 1# provides, and structure regards Frequently characteristics of image point set imaginary line bunch, meanwhile, uses view data dimension-reduction treatment method to be filtered place to imaginary line bunch Reason, obtain predicted characteristics point corresponding its in the exact position of stable image, build constraint matrix, be used for calculating original input picture Characteristic point and the corresponding exact position stablizing characteristics of image point set, and present frame is rebuild, the image rebuild is passed meanwhile Give PLD FPGA；

Image sequence buffer DDR3, for during Stabilization of video sequences, to present frame, associated frame and intermediate data Interim storage；

Parameter configuration static memory SRAM, before processing for Video Stabilization, stores to the systematic parameter setting.

2. the sequence of video images real-time stabilization method supporting multifreedom motion, it is characterised in that: use image sequence Interframe epipole transformation model, it was predicted that characteristic point in the exact position of each picture frame, and by build video image predicted characteristics point Collection imaginary line bunch, depiction is as the multifreedom motion of sequence interframe, meanwhile, uses dimensionality reduction image processing method, obtains Take predicted characteristics point corresponding its in the exact position of stable image, thus build core constraint matrix, be originally inputted figure for calculating As the exact position of feature point set its stable characteristics of image point set corresponding, rebuild finally by input picture, obtain steady Fixed output video.

3. a kind of sequence of video images real-time stabilization method supporting multifreedom motion as claimed in claim 2, its feature It is: utilize the conversion of multiframe epipole and average decision making algorithm to realize the high-precision forecast to current image frame feature point set position, And by the variable condition of frame institute predicted characteristics point position each to image sequence, obtain video camera in shooting process how freely Degree movable information.

4. a kind of sequence of video images real-time stabilization method supporting multifreedom motion as claimed in claim 2, its feature It is: by the method setting up image sequence predicted characteristics point set virtual track set of curves, represent 3d space multifreedom motion The motion state of the image sequence interframe acquired in video camera；Use dimension-reduction treatment method, by predicted characteristics point set virtual track Set of curves projects respectively at X and Y-direction, by carrying out 2D smoothing processing respectively to drop shadow curve, obtains predicted characteristics point Collection in the smooth change of sequence of video images each frame position coordinate, so that it is determined that original input picture Feature point correspondence its stable The core constraint matrix of picture position.

5. a kind of sequence of video images real-time stabilization method supporting multifreedom motion as claimed in claim 4, its feature It is: for the static nature point in scene, by its core in stable picture position of original input picture Feature point correspondence about Beam matrix, uses epipole conversion and average decision making algorithm, accurately estimates that original input picture stablizes image characteristic point position with corresponding Put change.

6. a kind of sequence of video images real-time stabilization method supporting multifreedom motion as claimed in claim 4, its feature It is: for behavioral characteristics point process, based on moving object in scene in the stationarity of image sequence interframe movement track, determine Suffered by object, the feature of minimum external force constraint, sets up behavioral characteristics point dynamics mathematical model, solves motion feature point in scene At each image frame position of image sequence.

7. a kind of sequence of video images real-time stabilization method supporting multifreedom motion as claimed in claim 4, its feature It is: that in stable image process choosing and following the tracks of of key feature points and shows in the uniformity of image-region distribution based on it Work property rule；And the replacement policy of characteristic point is based on continuity and the characteristic point position dynamic prediction of virtual track curve.

8. the sequence of video images real-time stabilization method supporting multifreedom motion, it is characterised in that: use following step Rapid realization:

Step 1: set and need sequence of video images to be processed as f_s, f_tFor present image；To f_sIn each two field picture, first by can Programmed logic device FPGA passes to digital signal processor DSP #1 process after being divided into 16 image regions, In each region, KLT point track algorithm is used to select ten characteristic points, and image behind, by feature point tracking technology The all characteristic points chosen are tracked, constitute f_sA feature point set sequenceWhereinRepresent the of video sequence The ith feature point that s two field picture is chosen；

Step 2: to video sequence f_s, with present image f_tCentered on, continuous 17 two field pictures constitute a process list being associated Unit, it was predicted that present frame characteristic point accurate coordinate position in the picture；Process picture frame f to current_t, being obtained by track algorithm should Frame and the feature point set sequence of its forward and backward adjacent 8 frame continuous processing images, be designated as: Wherein: t-8≤s≤t+8；

Step 3: based on area distribution and conspicuousness feature, at the feature point set of processing unitIn sequence, to each two field picture Feature point set, select eight key feature points therein, calculate currently processed picture frame f_tIt is adjacent forward and backward eight two field pictures Core constraint transformation matrix F^s _, ^t；

Step 4: according to image Nuclear goemetry constraint principles, uses epipole conversion method, byAnd F^s _, ^tCalculate each frame characteristic point? Image f_tIn projection lineAnd byIntersection point or the mean value of each position of intersecting point, as characteristic point i at image f_tMiddle position The prediction put, thus constitutive characteristic point setPredict point set at present image

Step 5: often gather a new images and enter video sequence, then can make: t=t+1, repeats step 2～step 4, i.e. available Predicted characteristics sequence of point sets:And when 1 < s < when 8, makesSo can get one Continuously predicted characteristics sequence of point sets

Step 6: digital signal processor DSP #1 will predict sequence of point sets continuouslyPass to digital signal processor DSP #2, number Word signal processor DSP#2 using frame number s as time Z axis,Middle coordinate at two dimensional image respectively constitutes X-axis and Y-axis, shape Become three-dimensional system of coordinate, according toThe frame number of point sequence and at the coordinate (x of each frame_i,y_i) value, determine each predicted characteristics point newly Build the position in coordinate system, and the corresponding points by each picture frame are linked to be curve, constitute the imaginary line bunch along time shaft, and at any time Between constantly gather new video image, its imaginary line constantly extends；It is every that the corresponding predicted characteristics point of each frame is constituted The smoothness of bar curve, reflects degree of stability in sequence of video images for this feature point, and whole imaginary line bunch is then anti- Reflect degree of stability in shooting process for the video camera；

Step 7: digital signal processor DSP #2 is along Z-direction, by each curve of imaginary line bunch respectively in X-direction Project with Y-direction, and with present image as processing center, forward and backward adjacent totally 11 two field pictures are a processing unit, successively Use convolution algorithm method, it is achieved smoothing filtering operation is carried out to curve, available predicted characteristics point setCorresponding stablizes Sequence:

Step 8: according to primitive image features sequence of point setsWith in critical sequencesCorresponding relation, according to its area distribution With conspicuousness feature, again existWithIn, select eight key feature points therein, calculate current original image frame f_tSpecial Levy a littleIt is adjacent forward and backward five two field pictures correspondingCore constraint transformation matrixt-5<s<t+5；

Step 9: according to image Nuclear goemetry restriction relation, uses epipole conversion method and average decision making algorithm, byEstimate and work as Front imageIt is at the location point of stabilizer frame

Step 10: to the current image frame processing, if characteristic point i is corresponding to the moving object in non-static image, then analyze it Motion feature, uses dynamics Mathematical Modeling Methods, resolves its smooth motion status flag and put, rightThe position of middle corresponding points Put and be adjusted correspondingly；

Step 11: to frame under process f_t, by starting the feature set selectedThe corresponding spy obtained with after above-mentioned process CollectionPosition relationship, to present image f_tRebuild, and the image that will rebuildPass to PLD FPGA；

Step 12: PLD FPGA is to the image rebuildIt is reassembled into new video, and use tcp/ip communication New stable sequence of video images is generated the output of standard digital video stream by agreement.