CN110390685B - Feature point tracking method based on event camera - Google Patents

Feature point tracking method based on event camera Download PDF

Info

Publication number
CN110390685B
CN110390685B CN201910672162.2A CN201910672162A CN110390685B CN 110390685 B CN110390685 B CN 110390685B CN 201910672162 A CN201910672162 A CN 201910672162A CN 110390685 B CN110390685 B CN 110390685B
Authority
CN
China
Prior art keywords
event
module
time
template
feature points
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910672162.2A
Other languages
Chinese (zh)
Other versions
CN110390685A (en
Inventor
史殿习
李凯月
李睿豪
伽晗
王明坤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN201910672162.2A priority Critical patent/CN110390685B/en
Publication of CN110390685A publication Critical patent/CN110390685A/en
Application granted granted Critical
Publication of CN110390685B publication Critical patent/CN110390685B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Abstract

The invention discloses a characteristic point tracking method based on an event camera, aiming at improving the tracking precision of characteristic points. The technical scheme is that a characteristic point tracking system based on an event camera, which consists of a data acquisition module, an initialization module, an event set selection module, a matching module, a characteristic point updating module and a template edge updating module, is constructed. An initialization module extracts feature points and edge maps from image frames; an event set selection module selects an event set S of the feature points from the event stream around the feature points; the matching module matches the S with the template edges around the feature points to calculate tkOptical flow set G of n feature points at timekThe feature point update module is according to GkCalculating tk+1Position set FD of n feature points at timek+1The template edge updating module uses the IMU data pair PBDkIs updated to obtain tk+1Position set PBD of template edge corresponding to n characteristic points at momentk+1. The method and the device can improve the accuracy of tracking the characteristic points on the event stream and prolong the average tracking time of the characteristic points.

Description

Feature point tracking method based on event camera
Technical Field
The invention relates to the field of computer image processing, in particular to a method for completing tracking of feature points in an image by using an event camera.
Background
SLAM (full name simultaneous localization and mapping) has been widely studied in recent years as an important branch of the field of robotics. SLAM attempts to solve such problems: a robot moves in an unknown environment, how to determine the motion track of the robot through observation of the environment and how to construct a map of the environment. SLAM technology is just a summation of many technologies involved to achieve this goal. A complete SLAM system mainly comprises a front-end visual odometer part and a back-end optimization part. The vision odometer part estimates the state of the robot, and the estimation method mainly comprises two methods: the characteristic point method and the direct method. The feature point method is a mainstream method for estimating the state of the robot at present, namely, feature points are extracted from an image, the feature points of different frames are matched, and then the matched feature point pairs are subjected to related operation to estimate the pose of a camera. Commonly used point features include Harris corner points, SIFT, SURF, ORB, HOG features. Different from a characteristic point method, the direct method can omit the process of extracting characteristic points and directly utilize the gray information in the image to estimate the state of the robot, but the method is not mature and has poor robustness.
However, whether feature point or direct, the use of standard cameras still presents problems of accuracy and robustness in dealing with extreme environments. The extremes include two main categories: under the condition of high-speed motion of the camera, when a standard camera is adopted to obtain an image, if the camera moves too fast, the obtained image has a motion blur phenomenon; under the condition of a high dynamic range scene, the light intensity change in the scene is strong, and the light and shade change of the front frame and the back frame is obvious. In these extreme cases, the use of standard cameras can severely impact the algorithmic effectiveness of the direct and feature point methods. Furthermore, standard cameras cannot provide a more accurate motion profile of feature points from frame to frame. And a standard camera generates redundant information in a static scene, which not only results in a waste of storage resources, but also consumes a lot of extra computing resources in the subsequent image processing process.
The advent of a bio-inspired event camera overcomes the above limitations of standard cameras. An event camera is typically represented by DVS (Dynamic Vision Sensor, known in chinese as Dynamic Vision Sensor, published 2011 by Patrick Lichtsteiner et al in the Journal of Solid-State Circuits, volume 43, phase 2, page 566, page 576, article "a 128 × 128120 dB 15 μ s Latency Asynchronous time Contrast Vision Sensor", i.e. "one 128 × 128 pixel, 120dB Dynamic range, 15 microsecond delay, produced by vivavation AG of switzerland, unlike standard cameras, the event camera outputs only brightness changes at the pixel level, in a pixel array, whenever a change in intensity exceeds a threshold, the corresponding pixel independently generates an output, referred to as" event ", so, unlike standard cameras, the data output by an event camera is a spatiotemporal event stream due to low Latency, large fast moving cameras, additionally have a great advantage in scene motion, the event camera may also avoid recording redundant information in slowly changing scenes. In 2014, a new event camera, DAVIS (Dynamic and Active-pixel Vision Sensor, known in Chinese as Dynamic and Active pixel Sensor), was proposed by an article "A240 × 180130dB 3 μ s Latency Global Shutter spatial and temporal Vision Sensor", published by Christian Brandli et al in the Journal of Solid-State Circuits 49, 10, 2333. sup. 2344, namely "a 240 × 180 pixels, 130dB Dynamic range, 3 μ s delayed Global Shutter spatio-temporal Vision Sensor", and was later produced by the iVation AG of Switzerland. DAVIS combines a standard camera and an event camera DVS together and can output both image frames and an event stream.
For feature point tracking by using an Event camera, Zhu et al propose a method for directly extracting feature points from an Event stream and calculating a feature point optical flow by using the Event stream to realize feature point tracking in an article "Event-based feature tracking with a predictive data association" which is a method for realizing Event-based feature point tracking by using probability data association (published in Conference International Conference on Robotics and Automation (ICRA) in 2017, page 4465 and 4470). Since the method calculates the feature point optical flow using only the event stream, the tracking accuracy is not high. Kueng et al, in the article Low-Latency Visual evaluation Event-based features Tracks, namely, Low-Latency Visual odometer using Event-based Feature point tracking (2016, published in Conference 2016 International Conference on Intelligent Robots and Systems (IROS), and having an Inspec search number of 16504091), proposed a Feature point tracking method that combines image frames and Event streams simultaneously, and uses information in the image frames to perform geometric registration with the Event streams, thereby tracking the Feature points. With the method, the position of the feature point is updated once an event is received, which increases the calculation amount, and the method introduces more errors, which leads to poor tracking accuracy of the feature point and can not realize long-time tracking.
Therefore, the existing method for tracking the feature points by using the event camera still has the defects of low tracking accuracy and short tracking time.
Disclosure of Invention
The invention aims to provide a feature point tracking method based on an event camera, wherein the DAVIS is used as the event camera, so that the accuracy of tracking feature points on an event stream is improved, and the average tracking time of the feature points is prolonged.
In order to solve the problem, the invention provides a feature point tracking method based on an event camera, the used event camera is DAVIS, the method simultaneously uses edges in an image frame to be matched with events in an event stream to obtain the optical flow of feature points, and the calculated optical flow is more accurate due to the simultaneous utilization of information in two forms, so that the feature point tracking precision is improved. In addition, the invention introduces IMU (Inertial Measurement Unit, named as Inertial Measurement Unit in Chinese) to update the position of the edge, so that the position of the edge in the tracking process is more accurate, and the average tracking time of the feature point is prolonged.
The specific technical scheme is as follows:
firstly, a characteristic point tracking system based on an event camera is constructed. The characteristic point tracking system of the event camera consists of a data acquisition module, an initialization module, an event set selection module, a matching module, a characteristic point updating module and a template edge updating module.
The data acquisition module is connected with the initialization module, the event set selection module and the template edge updating module. The data set acquisition module downloads data from an open Event Camera data set 'The Event-Camera Dataset and Simulator' (named as 'Event Camera data set and Simulator' in Chinese, The data set is acquired by DAVIS, and The data set comprises an image frame, an Event stream and IMU data), sends The acquired image frame to The initialization module, sends The Event stream to The Event set selection module, and sends The IMU data to The template edge updating module.
The initialization module is connected with the data acquisition module, the event set selection module and the matching module. The initialization module receives the image frame from the data acquisition module, extracts the feature points and the edge map from the image frame, obtains the positions of the feature points and the positions of the template edges around the feature points (the edges around the feature points in the edge map are called as the template edges corresponding to the feature points), sends the positions of the feature points to the event set selection module, and sends the positions of the template edges around the feature points to the matching module.
The event set selection module is connected with the data acquisition module, the initialization module, the feature point updating module and the matching module, receives an event stream from the data acquisition module, receives the positions of feature points from the initialization module (the first cycle) or the feature point updating module (from the second cycle), receives the optical flow of the feature points from the feature point updating module (from the second cycle), selects an event set of the feature points from the event stream around each feature point, and sends the position of each feature point and the corresponding event set to the matching module.
The matching module is connected with the initialization module, the event set selection module, the feature point updating module and the template edge updating module, receives the positions of the feature points and the event sets corresponding to the feature points from the event set selection module, receives the positions of template edges around the feature points from the initialization module (the first circulation) or the template edge updating module (from the second circulation), matches the event sets of the feature points with the template edges around the feature points, calculates the optical flow of each feature point, and sends the positions of the feature points, the positions of the template edges around the feature points and the optical flow of the feature points to the feature point updating module.
The feature point updating module is connected with the matching module, the template edge updating module and the event set selection module, receives the positions of the feature points, the positions of template edges around the feature points and the optical flows of the feature points from the matching module, calculates the new positions of the feature points by using the optical flows of the feature points, sends the positions of the template edges around the feature points and the new positions of the feature points to the template edge updating module, sends the optical flows of the feature points and the new positions of the feature points to the event set selection module, and outputs the new positions of the feature points (namely the tracking results of the feature points) for a user to check.
The template edge updating module is connected with the data acquisition module, the feature point updating module and the matching module, receives the positions of the template edges around the feature points and the new positions of the feature points from the feature point updating module, receives IMU data from the data acquisition module, updates the positions of the template edges around the feature points through the IMU data, and sends the updated positions of the template edges to the matching module.
In a second step, The data acquisition module acquires image frames, Event streams and IMU data from The Event-Camera Dataset "The Event-Camera Dataset and Simulator".
Thirdly, using a characteristic point tracking system based on an event camera to track the slave t0Time begins to tNAnd tracking the characteristic points in the event stream obtained by the data acquisition module within the time period when the time is over. Time interval t in tracking process0,tN]Divided into a series of sub-time intervals t0,t1],...,[tk,tk+1],...,[tN-1,tN]The k +1 th time subinterval is denoted as [ t ]k,tk+1]N represents the number of sub-time intervals, the size of N is determined by the time length of the event stream, and the value range is N not less than 1. The tracking process is as follows:
3.1, making the time sequence number k equal to 0;
3.2, the initialization module performs initialization operation, and the specific steps are as follows:
3.2.1 initialization Module uses Harris corner detection method (in 1988, proposed by the article "A combined corner and edge detector" published by C.G. Harris et al in Conference alternate Vision Conference, volume 15, phase 50, namely "Combined corner and edge detector") to extract feature points from the image frames obtained by the data acquisition module, and puts the extracted feature points into a set FD, where FD is set to { f }1,...,fi,...,fn}，fiRepresenting the ith detected characteristic point, and n is the number of the characteristic points. Considering the position of the feature points as a function of time, let t0The positions of n characteristic points in the time FD are placed in the characteristic point position set FD0In (FD)0＝{f1(t0),...,fi(t0),...,fn(t0)}，fi(t0) Representative feature point fiAt t0Position of time, FD0And sending the event to an event set selection module.
3.2.2 the initialization module extracts edge maps from the image frames obtained by the data acquisition module using the Canny edge detection method (in 1987, proposed by article "A computational application to edge detection" published by John Canny et al at conference reading in Computer Vision, page 184-203), where each image frame corresponds to an edge map.
3.2.3 the initialization Module selects the template edge (represented by the set of all template points on the template edge) corresponding to n feature points in FD, and corresponds the n feature points in FD to t0Position set PBD of time-carving template edge0And sending the data to a matching module. The method comprises the following steps:
3.2.3.1 making i ═ 1;
3.2.3.2 characterizing Point fiAt t0Position of time fi(t0) At the center, at fi(t0) Selecting a rectangular area around the periphery
Figure BDA0002142074750000041
The size is s × s, i.e.
Figure BDA0002142074750000042
Is s, which ranges from 20 to 30 pixels. The edge graph detected at 3.2.2 is shown in
Figure BDA0002142074750000043
The inner side is taken as fiThe template edge of (1). The pixel point on the template edge is fiThe template points of (1). Definition of
Figure BDA0002142074750000044
Set of template points within
Figure BDA0002142074750000045
Figure BDA0002142074750000051
I.e. fiCorresponding template edge, will
Figure BDA0002142074750000052
Put into template edge set PB:
Figure BDA0002142074750000053
pjis fiThe jth template point of (a). P is to bejAs a function of time, pj(t0) Represents pjAt t0The location of the time of day.
Figure BDA0002142074750000054
Is shown at t0Time pjIn a rectangular area
Figure BDA0002142074750000055
And (4) the following steps. m isiRepresentation collection
Figure BDA0002142074750000056
The number of middle template points.
3.2.3.3 let i ═ i + 1;
3.2.3.4 if i is less than or equal to n, turning to 3.2.3.2; whether or notThen, a template edge set PB of template edges corresponding to the n feature points in FD is obtained,
Figure BDA0002142074750000057
definition of
Figure BDA0002142074750000058
The middle template point is at t0Set of locations of time of day
Figure BDA0002142074750000059
Let the template point in the n template edges in PB be at t0Location collection of time of day
Figure BDA00021420747500000510
PBD (poly-p-phenylene terephthalamide)0And sending the data to a matching module, and turning to the fourth step.
Fourthly, the event set selection module receives the event stream from the data acquisition module, respectively receives different data from the initialization module or the characteristic point updating module according to different k values, selects an event set S of characteristic points from the event stream around the n characteristic points, and enables the FD to be displayed on the display screenkAnd sending the event set S of the feature points to a matching module, wherein the method comprises the following steps:
4.1 if k is equal to 0, the event set selection module receives the event stream from the data acquisition module and receives t from the initialization modulekPosition set FD of n feature points at timek，FDk＝{f1(tk),...,fi(tk),...,fn(tk) At this time, FD0＝{f1(t0),...,fi(t0),...,fn(t0) }), let t1＝t0+1, unit is second, change 4.3;
4.2 if k is more than or equal to 1, the event set selection module receives the event stream from the data acquisition module and receives t from the feature point updating modulekPosition set FD of n feature points at timekAnd tk-1Optical flow set G of n feature points at timek-1，
Figure BDA00021420747500000511
Wherein
Figure BDA00021420747500000512
Represents tk-1Time characteristic point fiOf the estimated sub-time interval [ t ]k,tk+1]Is calculated according to the formula (2)k+1：
Figure BDA00021420747500000513
Wherein
Figure BDA00021420747500000514
Is expressed in a sub-time interval tk-1,tk]Upper characteristic point fiThe optical flow of (2) is obtained from the feature point update module.
Figure BDA00021420747500000515
Is expressed in a sub-time interval tk-1,tk]Average optical flow of all the above feature points. Calculating t by equation (2)k+1The physical significance of (1) is that the time required for the feature point in the last time interval to move by 3 pixels on average is taken as the current interval [ t [ [ t ]k,tk+1]An estimate of the size is predicted. 4.3, rotating;
4.3 at tkPosition set FD of n feature points at timekAnd selecting an event set corresponding to each characteristic point around each position.
4.3.1 making i ═ 1;
4.3.2 selecting the events meeting the requirement of the formula (3) from the event stream, and putting the events to the characteristic point fiAt tkEvent set corresponding to time
Figure BDA0002142074750000061
In and (2) mixing
Figure BDA0002142074750000062
Put into event set S:
Figure BDA0002142074750000063
Figure BDA0002142074750000064
representing a set of events in a three-dimensional space region, the spatial extent being the characteristic point fiAt tkPosition of time fi(tk) Surrounding rectangular region HfiThe time range is the interval [ tk,tk+1]。edRepresents
Figure BDA0002142074750000065
The d-th event in (e)dFrom the event stream and within the three-dimensional temporal region specified in equation (3), d ═ 1,2i}，ziTo represent
Figure BDA0002142074750000066
The number of internal events. Event edIs expressed in the form of
Figure BDA00021420747500000613
xdRepresents an event edThe coordinates in the pixel coordinate system are,
Figure BDA0002142074750000067
represents an event edThe time of occurrence of the event(s),
Figure BDA0002142074750000068
denotes edHas pixel coordinates in
Figure BDA0002142074750000069
And (4) the following steps.
4.3.3 let i ═ i + 1;
4.3.4 if i is less than or equal to n, turning to 4.3.2; otherwise, it indicates that t is obtainedkEvent sets S corresponding to the n feature points in time FD,
Figure BDA00021420747500000610
and FD will bekAnd S is sent to a matching module, and the fifth step is carried out.
Fifthly, the matching module receives FD from the event set selection modulekAnd S, if k is 0, receiving a position set PBD of the template edge corresponding to the n characteristic points from the initialization modulekTurning to the sixth step; if k is more than or equal to 1, receiving a template edge position set PBD corresponding to the n characteristic points from the template edge updating modulekTurning to the sixth step;
sixthly, the matching module matches the S with the template edges around the feature points to calculate tkOptical flow set G of n feature points at timekThe positions of n feature points are collected to form FDkAnd position sets PBD of template edges corresponding to n characteristic pointskAnd optical flow set GkAnd sending the data to a feature point updating module. The specific method comprises the following steps:
6.1 making i ═ 1;
6.2 pairs of feature points fiConstructing a matching error function:
for the
Figure BDA00021420747500000611
Event e indCorrecting the event edAt tkThe position of the moment is given by the formula:
Figure BDA00021420747500000612
x′drepresenting a calculated event edAt tkLocation of time of day, abbreviated event edThe position of the motion correction is made,
Figure BDA0002142074750000071
is expressed in the time interval tk,tk+1]Upper characteristic point fiThe optical flow of (2). The symbol-represents a dot product, and the places where the symbol appears in the following all represent dot products, and in the case of no ambiguity, some of the formulas in the following omit the symbol. For convenience of presentation, symbols are defined
Figure BDA0002142074750000072
The match error function is constructed as follows:
Figure BDA0002142074750000073
ε denotes the error, pj(tk) Represents tkTime template point pjThe position of (a). r isdjDenotes edFrom template point pjProbability of generation, i.e. edAnd pjThe probability of a match. Here will rdjAs weights, rdjAs an event edAnd template point pjProbability of match, i.e. rdjThe larger, the event edAnd template point pjAt t betweenkThe larger the difference in distance at a time is in proportion to the total error. The double vertical lines in the formula represent the modulus operation of the vectors in the double vertical lines, and the same is applied below. r isdjThe calculation formula of (a) is as follows:
Figure BDA0002142074750000074
the physical significance of this formula is that the numerator is event edMotion corrected position and template point pjDistance squared, denominator event edPosition and feature point f of motion correctioniCorresponding miThe sum of the squares of the distances of the template points is divided by the sum of the squares of the distances of the template points as an event edAnd template point pjThe probability of a match.
6.3 the EM-ICP algorithm (matching algorithm, 2002, Multi-scale EM-ICP: A fast and robust approach for surface registration, multiscale EM-ICP: a fast and robust surface registration method) was used, which was published by Sebastient Granger et al at Conference European Conference on Computer Vision, page 418-. Minimizing the matching error function obtained by 6.2-step construction, and solving to obtain the optimal optical flow
Figure BDA0002142074750000075
The solving process is as follows:
6.3.1 initialization
Figure BDA0002142074750000076
6.3.2 calculating r by equation (6)dj；
6.3.3 updating the optical flow:
Figure BDA0002142074750000081
6.3.4 calculating the amount of change in optical flow
Figure BDA0002142074750000082
And order
Figure BDA0002142074750000083
Will be provided with
Figure BDA0002142074750000084
Is put to tkTemporal optical flow set GkIn (1).
6.3.5 if
Figure BDA0002142074750000085
Shows that the final optimization result is obtained
Figure BDA0002142074750000086
6.4, rotating; if it is
Figure BDA0002142074750000087
Turn 6.3.2. σ is the threshold of the variation of the optical flow, and the value of the threshold ranges from 0.01 to 0.1, and the unit is pixel per second (pixel/s).
6.4 let i ═ i + 1;
if i is less than or equal to n, 6.5, turning to 6.2; otherwise, it indicates that t is obtainedkThe optical flows of n feature points at the moment of time, namely t is obtainedkTemporal optical flow set Gk，
Figure BDA0002142074750000088
FD will bek、PBDkAnd GkSending the data to a feature point updating module, and turning to the seventh stepAnd (5) carrying out the steps.
Seventhly, the feature point updating module receives t from the matching modulekPosition set FD of n feature points at timek、tkPosition set PBD of template edge corresponding to n characteristic points at momentkAnd an optical flow set G of n feature pointskCalculating t by optical flowk+1Position set FD of n feature points at timek+1G iskAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkAnd sending the data to a template edge updating module. The method comprises the following steps:
7.1 making i ═ 1;
7.2 calculating tk+1Time fiPosition f ofi(tk+1) A 1 is to fi(tk+1) Put into the set FDk+1In (1),
Figure BDA0002142074750000089
Figure BDA00021420747500000810
by optical flow multiplied by time, from tkTime tk+1Time characteristic point fiAnd (4) moving distance.
7.3 let i ═ i + 1;
7.4 if i is less than or equal to n, rotating to 7.2; otherwise, it indicates that t is obtainedk+1The positions of n feature points at the moment are obtained, and a set FD is obtainedk+1，FDk+1＝{f1(tk+1),...,fi(tk+1),...,fn(tk+1)}. G is to bekAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkSending the data to a template edge updating module; and will tk+1Position set FD of n feature points at timek+1And displaying or storing the data in a result file for a user to check, and turning to the eighth step.
Eighthly, the template edge updating module receives IMU data from the data acquisition module and t from the feature point updating modulek+1Position set FD of n feature points at timek+1And tkPosition set PBD of template edges around time characteristic pointkUsing IMU data pairs PBDkIs updated to obtain tk+1Position set PBD of template edge corresponding to n characteristic points at momentk+1PBD ofk+1And sending the data to a matching module. The specific method comprises the following steps:
8.1 making i ═ 1;
8.2 updating feature points fiCorresponding template edge at tk+1The method for the time position comprises the following steps:
8.2.1 let j equal 1;
8.2.2 defining the symbol F as being ANDiCorresponding point in three-dimensional space, PjIs a point p corresponding to the templatejPoints in the corresponding three-dimensional space, F and PjAre all represented in the form of three-dimensional coordinates (x, y, z). Template point pjExpressed in homogeneous coordinate form, with the format of (x, y,1), the first two dimensions respectively represent pjAbscissa and ordinate in the pixel coordinate system, the following equation then holds:
at tkAt the moment of time, the time of day,
Figure BDA0002142074750000091
at tk+1At the moment of time, the time of day,
Figure BDA0002142074750000092
and K is an internal parameter matrix of the event camera and is a self-contained parameter of the event camera. R is from tkTime tk+1The rotation matrix of the time-of-day event camera, t being from tkTime tk+1And the translation vector of the moment event camera is calculated through the obtained IMU data.
Figure BDA0002142074750000093
Is PjAt tkThe depth of the time camera coordinate system,
Figure BDA0002142074750000094
Is PjAt tk+1The depth in the camera coordinate system at the moment,
Figure BDA0002142074750000095
is F at tkThe depth of the time camera coordinate system,
Figure BDA0002142074750000096
Is F at tk+1The depth in the camera coordinate system at the moment.
8.2.3 subtracting the two equations in equation (10) to get tk+1Time template point pjRelative to fiRelative position of
Figure BDA0002142074750000097
The calculation formula is as follows:
Figure BDA0002142074750000098
due to template point p in the pixel coordinate systemjAt characteristic point fiSo that the corresponding point P in spacejSimilarly to F, in this field, such a case is considered to be PjAnd F is at tk+1Having the same depth in the camera coordinate system at the moment, i.e. at
Figure BDA0002142074750000101
Substituting equation (9) into equation (11), and simplifying equation (11) yields:
Figure BDA0002142074750000102
consider pjAnd fiAll are represented by homogeneous coordinates, and t is obtained by further simplifying the formula (12)k+1The formula of the relative position of the time template point is as follows:
Figure BDA0002142074750000103
the symbol Nor () represents a homogeneous operation, i.e., the coordinates in parentheses are converted into homogeneous coordinates. Updated t is obtained by equation (13)k+1Time template point pjRelative to fiRelative position of
Figure BDA0002142074750000104
Obtaining t according to equation (14)k+1Time template point pjPosition p ofj(tk+1) A 1 is to pj(tk+1) Is put to tk+1Time characteristic point fiSet of positions of surrounding formwork edges
Figure BDA00021420747500001010
Figure BDA0002142074750000105
8.2.4 let j ═ j + 1;
8.2.5 if j is less than or equal to miTurning to 8.2.2; otherwise it indicates that
Figure BDA0002142074750000106
Figure BDA0002142074750000107
Will be provided with
Figure BDA0002142074750000108
Is put to tk+1Position set PBD of template edges around n characteristic points at momentk+1Middle, 8.3;
8.3 let i ═ i + 1;
8.4 if i is less than or equal to n, rotating to 8.2; otherwise, the PBD is obtainedk+1，
Figure BDA0002142074750000109
PBD (poly-p-phenylene terephthalamide)k+1And sending the data to a matching module, and turning to the ninth step.
And a ninth step of making k equal to k + 1.
Step ten, if k is less than N, turning to the step four; otherwise, ending.
The invention can achieve the following technical effects:
1. the invention simultaneously utilizes two data information of image frames and event streams output by DAVIS, firstly uses a Harris corner detection method to extract characteristic points and a Canny edge detection method to extract an edge graph, then selects a space time window from the event streams, uses an EM-ICP algorithm to match events in the window with template points on the edge graph, estimates the optical flow of the characteristic points, and further tracks the characteristic points through the optical flow.
2. When the template is updated, the IMU is used for updating the position of the template point, and the tracking precision of the template edge is improved in the tracking process, so that the accuracy of calculating the optical flow is improved, and the tracking precision of the feature point is further improved.
The invention carries out experimental verification on an Event Camera data set 'The Event-Camera Dataset and Simulator' (Event Camera data set and Simulator) released by The university of zurich, and carries out comparative experiments with The feature point tracking method proposed by Zhu et al and The method proposed by Kueng et al.
Drawings
FIG. 1 is a general flow diagram of the present invention;
FIG. 2 is a logic structure diagram of a feature point tracking system based on an event camera according to a first step of the present invention;
FIG. 3 is a comparison experiment result of average tracking accuracy errors between the present invention and the existing tracking method;
FIG. 4 is a comparison experiment result of the average tracking time of the present invention and the prior tracking method.
Detailed Description
FIG. 1 is a general flow diagram of the present invention; as shown in fig. 1, the present invention comprises the steps of:
firstly, a characteristic point tracking system based on an event camera is constructed. The feature point tracking system of the event camera is shown in fig. 2 and comprises a data acquisition module, an initialization module, an event set selection module, a matching module, a feature point updating module and a template edge updating module.
The data acquisition module is connected with the initialization module, the event set selection module and the template edge updating module. The data set acquisition module downloads data from a public Event Camera data set 'The Event-Camera Dataset and Simulator' of university of zurich, sends The acquired image frames to The initialization module, sends Event streams to The Event set selection module, and sends IMU data to The template edge updating module.
The initialization module is connected with the data acquisition module, the event set selection module and the matching module. The initialization module receives the image frame from the data acquisition module, extracts the feature points and the edge map from the image frame to obtain the positions of the feature points and the positions of the template edges around the feature points, sends the positions of the feature points to the event set selection module, and sends the positions of the template edges around the feature points to the matching module.
The event set selection module is connected with the data acquisition module, the initialization module, the feature point updating module and the matching module, receives an event stream from the data acquisition module, receives the positions of feature points from the initialization module (the first cycle) or the feature point updating module (from the second cycle), receives the optical flow of the feature points from the feature point updating module (from the second cycle), selects an event set of the feature points from the event stream around each feature point, and sends the position of each feature point and the corresponding event set to the matching module.
The matching module is connected with the initialization module, the event set selection module, the feature point updating module and the template edge updating module, receives the positions of the feature points and the event sets corresponding to the feature points from the event set selection module, receives the positions of template edges around the feature points from the initialization module (the first circulation) or the template edge updating module (from the second circulation), matches the event sets of the feature points with the template edges around the feature points, calculates the optical flow of each feature point, and sends the positions of the feature points, the positions of the template edges around the feature points and the optical flow of the feature points to the feature point updating module.
The feature point updating module is connected with the matching module, the template edge updating module and the event set selection module, receives the positions of the feature points, the positions of template edges around the feature points and the optical flows of the feature points from the matching module, calculates the new positions of the feature points by using the optical flows of the feature points, sends the positions of the template edges around the feature points and the new positions of the feature points to the template edge updating module, sends the optical flows of the feature points and the new positions of the feature points to the event set selection module, and outputs the new positions of the feature points (namely the tracking results of the feature points) for a user to check.
The template edge updating module is connected with the data acquisition module, the feature point updating module and the matching module, receives the positions of the template edges around the feature points and the new positions of the feature points from the feature point updating module, receives IMU data from the data acquisition module, updates the positions of the template edges around the feature points through the IMU data, and sends the updated positions of the template edges to the matching module.
In a second step, The data acquisition module acquires image frames, Event streams and IMU data from The Event-Camera Dataset "The Event-Camera Dataset and Simulator".
Thirdly, using a characteristic point tracking system based on an event camera to track the slave t0Time begins to tNAnd tracking the characteristic points in the event stream obtained by the data acquisition module within the time period when the time is over. Time interval t in tracking process0,tN]Divided into a series of sub-time intervals t0,t1],...,[tk,tk+1],...,[tN-1,tN]The k +1 th time subinterval is denoted as [ t ]k,tk+1]N represents the number of sub-time intervals, the size of N is determined by the time length of the event stream, and the value range is N not less than 1. The tracking process is as follows:
3.1, making the time sequence number k equal to 0;
3.2, the initialization module performs initialization operation, and the specific steps are as follows:
3.2.1 the initialization module extracts feature points from the image frames obtained by the data acquisition module by using a Harris corner detection method, and places the extracted feature points in a set FD, so that the FD is set to be { f }1,...,fi,...,fn}，fiRepresenting the ith detected characteristic point, and n is the number of the characteristic points. Considering the position of the feature points as a function of time, let t0The positions of n characteristic points in the time FD are placed in the characteristic point position set FD0In (FD)0＝{f1(t0),...,fi(t0),...,fn(t0)}，fi(t0) Representative feature point fiAt t0Position of time, FD0And sending the event to an event set selection module.
3.2.2 the initialization module extracts edge maps from the image frames obtained by the data acquisition module by using a Canny edge detection method, wherein each image frame corresponds to one edge map.
3.2.3 the initialization Module selects the template edge (represented by the set of all template points on the template edge) corresponding to n feature points in FD, and corresponds the n feature points in FD to t0Position set PBD of time-carving template edge0And sending the data to a matching module. The method comprises the following steps:
3.2.3.1 making i ═ 1;
3.2.3.2 characterizing Point fiAt t0Position of time fi(t0) At the center, at fi(t0) Selecting a rectangular area around the periphery
Figure BDA0002142074750000131
The size is s × s, i.e.
Figure BDA0002142074750000132
Is s, which ranges from 20 to 30 pixels. The edge graph detected at 3.2.2 is shown in
Figure BDA0002142074750000133
The inner side is taken as fiThe template edge of (1). The pixel point on the template edge is fiThe template points of (1). Definition of
Figure BDA0002142074750000134
Set of template points within
Figure BDA0002142074750000135
Figure BDA0002142074750000136
I.e. fiCorresponding template edge, will
Figure BDA0002142074750000137
Put into template edge set PB:
Figure BDA0002142074750000138
pjis fiThe jth template point of (a). P is to bejAs a function of time, pj(t0) Represents pjAt t0The location of the time of day.
Figure BDA0002142074750000139
Is shown at t0Time pjIn a rectangular area
Figure BDA00021420747500001310
And (4) the following steps. m isiRepresentation collection
Figure BDA00021420747500001311
The number of middle template points.
3.2.3.3 let i ═ i + 1;
3.2.3.4 if i is less than or equal to n, turning to 3.2.3.2; otherwise, the template edge set PB formed by the template edges corresponding to the n feature points in the FD is obtained,
Figure BDA00021420747500001312
definition of
Figure BDA00021420747500001313
The middle template point is at t0Set of locations of time of day
Figure BDA00021420747500001314
Let the template point in the n template edges in PB be at t0Location collection of time of day
Figure BDA00021420747500001315
PBD (poly-p-phenylene terephthalamide)0And sending the data to a matching module, and turning to the fourth step.
Fourthly, the event set selection module receives the event stream from the data acquisition module, respectively receives different data from the initialization module or the characteristic point updating module according to different k values, selects an event set S of characteristic points from the event stream around the n characteristic points, and enables the FD to be displayed on the display screenkAnd sending the event set S of the feature points to a matching module, wherein the method comprises the following steps:
4.1 if k is equal to 0, the event set selection module receives the event stream from the data acquisition module and receives t from the initialization modulekPosition set FD of n feature points at time0Let t1＝t0+1, unit is second, change 4.3;
4.2 if k is more than or equal to 1, the event set selection module receives the event stream from the data acquisition module and receives t from the feature point updating modulekPosition set FD of n feature points at timekAnd tk-1Optical flow set G of n feature points at timek-1，
Figure BDA00021420747500001316
Wherein
Figure BDA0002142074750000141
Represents tk-1Time characteristic point fiOf the estimated sub-time interval [ t ]k,tk+1]Is calculated according to the formula (2)k+1：
Figure BDA0002142074750000142
Wherein
Figure BDA0002142074750000143
Is expressed in a sub-time interval tk-1,tk]Upper characteristic point fiThe optical flow of (2) is obtained from the feature point update module.
Figure BDA0002142074750000144
Is expressed in a sub-time interval tk-1,tk]Average optical flow of all the above feature points. Calculating t by equation (2)k+1The physical significance of (1) is that the time required for the feature point in the last time interval to move by 3 pixels on average is taken as the current interval [ t [ [ t ]k,tk+1]An estimate of the size is predicted. 4.3, rotating;
4.3 at tkPosition set FD of n feature points at timekAnd selecting an event set corresponding to each characteristic point around each position.
4.3.1 making i ═ 1;
4.3.2 selecting the events meeting the requirement of the formula (3) from the event stream, and putting the events to the characteristic point fiAt tkEvent set corresponding to time
Figure BDA0002142074750000145
In and (2) mixing
Figure BDA0002142074750000146
Put into event set S:
Figure BDA0002142074750000147
Figure BDA0002142074750000148
representing a set of events in a three-dimensional space region, the spatial extent being the characteristic point fiAt tkPosition of time fi(tk) A surrounding rectangular area
Figure BDA0002142074750000149
The time range is the interval [ tk,tk+1]。edRepresents
Figure BDA00021420747500001410
The d-th event of (1, 2., z)i}，ziTo represent
Figure BDA00021420747500001411
The number of internal events. Event edIs expressed in the form of
Figure BDA00021420747500001412
xdRepresents an event edThe coordinates in the pixel coordinate system are,
Figure BDA00021420747500001413
represents an event edThe time of occurrence of the event(s),
Figure BDA00021420747500001414
denotes edHas pixel coordinates in
Figure BDA00021420747500001415
And (4) the following steps.
4.3.3 let i ═ i + 1;
4.3.4 if i is less than or equal to n, turning to 4.3.2; otherwise, it indicates that t is obtainedkEvent sets S corresponding to the n feature points in time FD,
Figure BDA00021420747500001416
and FD will bekAnd S is sent to a matching module, and the fifth step is carried out.
Fifthly, the matching module receives FD from the event set selection modulekAnd S, if k is 0, receiving a position set PBD of the template edge corresponding to the n characteristic points from the initialization modulekTurning to the sixth step; if k is more than or equal to 1, receiving a template edge position set PBD corresponding to the n characteristic points from the template edge updating modulekTurning to the sixth step;
sixthly, the matching module matches the S with the template edges around the feature points to calculate tkOptical flow set G of n feature points at timekThe positions of n feature points are collected to form FDkAnd position sets PBD of template edges corresponding to n characteristic pointskAnd lightStream set GkAnd sending the data to a feature point updating module. The specific method comprises the following steps:
6.1 making i ═ 1;
6.2 pairs of feature points fiConstructing a matching error function:
for the
Figure BDA0002142074750000151
Event e indCorrecting the event edAt tkThe position of the moment is given by the formula:
Figure BDA0002142074750000152
x′drepresenting a calculated event edAt tkLocation of time of day, abbreviated event edThe position of the motion correction is made,
Figure BDA0002142074750000153
is expressed in the time interval tk,tk+1]Upper characteristic point fiThe optical flow of (2). The symbol-represents a dot product, and the places where the symbol appears in the following all represent dot products, and in the case of no ambiguity, some of the formulas in the following omit the symbol. For convenience of presentation, symbols are defined
Figure BDA0002142074750000154
The match error function is constructed as follows:
Figure BDA0002142074750000155
ε denotes the error, pj(tk) Represents tkTime template point pjThe position of (a). r isdjDenotes edFrom template point pjProbability of generation, i.e. edAnd pjThe probability of a match. Here will rdjAs weights, rdjAs an event edAnd template point pjProbability of match, rdjThe calculation formula of (a) is as follows:
Figure BDA0002142074750000156
6.3 minimizing the matching error function obtained by 6.2-step construction by adopting an EM-ICP algorithm, and solving to obtain the optimal optical flow
Figure BDA0002142074750000157
The solving process is as follows:
6.3.1 initialization
Figure BDA0002142074750000158
6.3.2 calculating r by equation (6)dj；
6.3.3 updating the optical flow:
Figure BDA0002142074750000161
6.3.4 calculating the amount of change in optical flow
Figure BDA0002142074750000162
And order
Figure BDA0002142074750000163
Will be provided with
Figure BDA0002142074750000164
Is put to tkTemporal optical flow set GkIn (1).
6.3.5 if
Figure BDA0002142074750000165
Shows that the final optimization result is obtained
Figure BDA0002142074750000166
6.4, rotating; if it is
Figure BDA0002142074750000167
Turn 6.3.2. σ is the variation threshold of the optical flowThe value, which ranges from 0.01 to 0.1, in pixels per second.
6.4 let i ═ i + 1;
if i is less than or equal to n, 6.5, turning to 6.2; otherwise, it indicates that t is obtainedkThe optical flows of n feature points at the moment of time, namely t is obtainedkTemporal optical flow set Gk，
Figure BDA0002142074750000168
FD will bek、PBDkAnd GkAnd sending the data to a feature point updating module, and turning to the seventh step.
Seventhly, the feature point updating module receives t from the matching modulekPosition set FD of n feature points at timek、tkPosition set PBD of template edge corresponding to n characteristic points at momentkAnd an optical flow set G of n feature pointskCalculating t by optical flowk+1Position set FD of n feature points at timek+1G iskAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkAnd sending the data to a template edge updating module. The method comprises the following steps:
7.1 making i ═ 1;
7.2 calculating tk+1Time fiPosition f ofi(tk+1) A 1 is to fi(tk+1) Put into the set FDk+1In (1),
Figure BDA0002142074750000169
Figure BDA00021420747500001610
by optical flow multiplied by time, from tkTime tk+1Time characteristic point fiAnd (4) moving distance.
7.3 let i ═ i + 1;
7.4 if i is less than or equal to n, rotating to 7.2; otherwise, it indicates that t is obtainedk+1The positions of n feature points at the moment are obtained, and a set FD is obtainedk+1，FDk+1＝{f1(tk+1),...,fi(tk+1),...,fn(tk+1)}. G is to bekAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkSending the data to a template edge updating module; and will tk+1Position set FD of n feature points at timek+1And displaying or storing the data in a result file for a user to check, and turning to the eighth step.
Eighthly, the template edge updating module receives IMU data from the data acquisition module and t from the feature point updating modulek+1Position set FD of n feature points at timek+1And tkPosition set PBD of template edges around time characteristic pointkUsing IMU data pairs PBDkIs updated to obtain tk+1Position set PBD of template edge corresponding to n characteristic points at momentk+1PBD ofk+1And sending the data to a matching module. The specific method comprises the following steps:
8.1 making i ═ 1;
8.2 updating feature points fiCorresponding template edge at tk+1The method for the time position comprises the following steps:
8.2.1 let j equal 1;
8.2.2 defining the symbol F as being ANDiCorresponding point in three-dimensional space, PjIs a point p corresponding to the templatejPoints in the corresponding three-dimensional space, F and PjAre all represented in the form of three-dimensional coordinates (x, y, z). Template point pjExpressed in homogeneous coordinate form, with the format of (x, y,1), the first two dimensions respectively represent pjAbscissa and ordinate in the pixel coordinate system, the following equation then holds:
at tkAt the moment of time, the time of day,
Figure BDA0002142074750000171
at tk+1At the moment of time, the time of day,
Figure BDA0002142074750000172
wherein K isAnd the internal parameter matrix of the event camera is the self-contained parameters of the event camera. R is from tkTime tk+1The rotation matrix of the time-of-day event camera, t being from tkTime tk+1And the translation vector of the moment event camera is calculated through the obtained IMU data.
Figure BDA0002142074750000173
Is PjAt tkThe depth of the time camera coordinate system,
Figure BDA0002142074750000174
Is PjAt tk+1The depth in the camera coordinate system at the moment,
Figure BDA0002142074750000175
is F at tkThe depth of the time camera coordinate system,
Figure BDA0002142074750000176
Is F at tk+1The depth in the camera coordinate system at the moment.
8.2.3 subtracting the two equations in equation (10) to get tk+1Time template point pjRelative to fiRelative position of
Figure BDA0002142074750000177
The calculation formula is as follows:
Figure BDA0002142074750000178
due to template point p in the pixel coordinate systemjAt characteristic point fiSo that the corresponding point P in spacejSimilarly to F, in this field, such a case is considered to be PjAnd F is at tk+1Having the same depth in the camera coordinate system at the moment, i.e. at
Figure BDA0002142074750000181
Substituting equation (9) into equation (11), and comparing equation (11)The method is simplified to obtain:
Figure BDA0002142074750000182
consider pjAnd fiAll are represented by homogeneous coordinates, and t is obtained by further simplifying the formula (12)k+1The formula of the relative position of the time template point is as follows:
Figure BDA0002142074750000183
the symbol Nor () represents a homogeneous operation, i.e., the coordinates in parentheses are converted into homogeneous coordinates. Updated t is obtained by equation (13)k+1Time template point pjRelative to fiRelative position of
Figure BDA0002142074750000184
Obtaining t according to equation (14)k+1Time template point pjPosition p ofj(tk+1) A 1 is to pj(tk+1) Is put to tk+1Time characteristic point fiSet of positions of surrounding formwork edges
Figure BDA0002142074750000185
Figure BDA0002142074750000186
8.2.4 let j ═ j + 1;
8.2.5 if j is less than or equal to miTurning to 8.2.2; otherwise it indicates that
Figure BDA0002142074750000187
Figure BDA0002142074750000188
Will be provided with
Figure BDA0002142074750000189
Is put to tk+1Position set PBD of template edges around n characteristic points at momentk+1Middle, 8.3;
8.3 let i ═ i + 1;
8.4 if i is less than or equal to n, rotating to 8.2; otherwise, the PBD is obtainedk+1，
Figure BDA00021420747500001810
PBD (poly-p-phenylene terephthalamide)k+1And sending the data to a matching module, and turning to the ninth step.
And a ninth step of making k equal to k + 1.
Step ten, if k is less than N, turning to the step four; otherwise, ending.
FIG. 3 is a comparison experiment result of average tracking accuracy errors between the present invention and the existing tracking method; the experimental results are obtained by testing The data set of The Event-Camera Dataset and Simulator according to The invention. The experimental environment is a notebook configured as i72.8GHz CPU and 8G RAM. The evaluation index of the experiment is the average tracking error of the feature points, and the unit is pixel. The left side of the graph is the name of the data sequence in the data set and the upper right side is the average tracking error of the feature point. The three columns of experimental data in the figure are the results of the test of the present invention, the method of Zhu et al, and the method of Kueng et al under the same test data sequence and the same experimental environment, respectively. Experimental results show that the present invention has a lower average tracking error across all test data sequences compared to the other two methods. In the figure, "x" indicates no data.
FIG. 4 is a comparison experiment result of the average tracking time of the present invention and the prior tracking method. The experiment is identical to the test data set and experimental environment of the experiment corresponding to fig. 3. The evaluation index of the experiment is the average tracking time of the feature points in seconds. The left side of the graph is the name of the data sequence in the data set, and the upper right side is the average tracking time of the feature point. The three columns of experimental data in the figure are the results of the test of the system, the system of Zhu et al, and the system of Kueng et al under the same test data series and the same experimental environment. Experimental results show that the present invention can achieve longer tracking on data sequences other than the data sequences "boxes _ translation" and "boxes _6 dof" compared to the other two methods.

Claims (5)

1. A feature point tracking method based on an event camera is characterized by comprising the following steps:
the method comprises the following steps that firstly, a characteristic point tracking system based on an event camera is constructed, wherein the characteristic point tracking system of the event camera consists of a data acquisition module, an initialization module, an event set selection module, a matching module, a characteristic point updating module and a template edge updating module;
the data acquisition module is connected with the initialization module, the event set selection module and the template edge updating module; the data set acquisition module downloads data from an event camera data set, sends the acquired image frames to the initialization module, sends event streams to the event set selection module, and sends IMU data, namely inertial measurement unit data, to the template edge updating module;
the initialization module is connected with the data acquisition module, the event set selection module and the matching module; the initialization module receives the image frame from the data acquisition module, extracts the feature points and the edge map from the image frame to obtain the positions of the feature points and the positions of the template edges around the feature points, sends the positions of the feature points to the event set selection module, and sends the positions of the template edges around the feature points to the matching module;
the event set selection module is connected with the data acquisition module, the initialization module, the feature point updating module and the matching module, receives the event stream from the data acquisition module, receives the positions of the feature points from the initialization module or the feature point updating module, receives the optical streams of the feature points from the feature point updating module, selects the event set of the feature points from the event stream around each feature point, and sends the positions of the feature points and the corresponding event sets to the matching module;
the matching module is connected with the initialization module, the event set selection module, the feature point updating module and the template edge updating module, receives the positions of the feature points and the event sets corresponding to the feature points from the event set selection module, receives the positions of the template edges around the feature points from the initialization module or the template edge updating module, matches the event sets of the feature points with the template edges around the feature points, calculates the optical flow of each feature point, and sends the positions of the feature points, the positions of the template edges around the feature points and the optical flow of the feature points to the feature point updating module;
the characteristic point updating module is connected with the matching module, the template edge updating module and the event set selecting module, receives the positions of the characteristic points, the positions of template edges around the characteristic points and the optical flows of the characteristic points from the matching module, calculates the new positions of the characteristic points by using the optical flows of the characteristic points, sends the positions of the template edges around the characteristic points and the new positions of the characteristic points to the template edge updating module, sends the optical flows of the characteristic points and the new positions of the characteristic points to the event set selecting module, and outputs the new positions of the characteristic points;
the template edge updating module is connected with the data acquisition module, the feature point updating module and the matching module, receives the positions of template edges around the feature points and the new positions of the feature points from the feature point updating module, receives IMU data from the data acquisition module, updates the positions of the template edges around the feature points through the IMU data, and sends the updated positions of the template edges to the matching module;
secondly, the data acquisition module acquires image frames, event streams and IMU data from the event camera data set;
thirdly, using a characteristic point tracking system based on an event camera to track the slave t0Time begins to tNTracking the characteristic points in the event stream obtained by the data acquisition module within the time period of ending the moment, and carrying out time interval [ t0,tN]Divided into a series of sub-time intervals t0,t1],...,[tk,tk+1],...,[tN-1,tN]N represents the number of sub-time intervals, the size of N is determined by the time length of the event stream, and the value range is N is more than or equal to 1; the k +1 th time subinterval is denoted as [ t ]k,tk+1](ii) a The tracking process is as follows:
3.1, making the time sequence number k equal to 0;
3.2, if k is equal to 0, the initialization module performs an initialization operation, and the method includes:
3.2.1 the initialization module extracts feature points from the image frames obtained by the data acquisition module by using a Harris corner detection method, and places the extracted feature points in a set FD, so that the FD is set to be { f }0,...,fi,...,fn}，fiRepresenting the ith detected characteristic point, wherein n is the number of the characteristic points; considering the position of the feature points as a function of time, let t0The positions of n characteristic points in the time FD are placed in the characteristic point position set FD0In (FD)0＝{f0(t0),...,fi(t0),...,fn(t0)}，fi(t0) Representative feature point fiAt t0Position of time, FD0Sending the event to an event set selection module;
3.2.2 the initialization module extracts edge maps from the image frames obtained by the data acquisition module by using a Canny edge detection method, wherein each image frame corresponds to one edge map;
3.2.3 the initialization Module selects the template edge corresponding to n feature points in FD, and corresponds t to n feature points in FD0Position set PBD of time-carving template edge0Sending the data to a matching module, wherein the method comprises the following steps:
3.2.3.1 making i ═ 1;
3.2.3.2 characterizing Point fiAt t0Position of time fi(t0) At the center, at fi(t0) Selecting a rectangular area around the periphery
Figure FDA0002904542740000021
The size is s multiplied by s, namely the length and the width of the rectangle are both s; the edge graph detected at 3.2.2 is shown in
Figure FDA0002904542740000022
The inner side is taken as fiThe pixel point on the template edge is fiThe template points of (1); definition of
Figure FDA0002904542740000023
Set of template points within
Figure FDA0002904542740000024
Figure FDA0002904542740000025
I.e. fiCorresponding template edge, will
Figure FDA0002904542740000026
Is put into the template edge set PB,
Figure FDA0002904542740000027
pjis fiJ-th template point of (1), pjAs a function of time, pj(t0) Represents pjAt t0The position of the moment of time is,
Figure FDA0002904542740000028
is shown at t0Time pjIn a rectangular area
Figure FDA0002904542740000029
Inner, miTo represent
Figure FDA00029045427400000210
The number of the middle template points;
3.2.3.3 let i ═ i + 1;
3.2.3.4 if i is less than or equal to n, turning to 3.2.3.2; otherwise, the template edge set PB formed by the template edges corresponding to the n feature points in the FD is obtained,
Figure FDA0002904542740000031
definition of
Figure FDA0002904542740000032
The middle template point is at t0Set of locations of time of day
Figure FDA0002904542740000033
Let the template point in the n template edges in PB be at t0Location collection of time of day
Figure FDA0002904542740000034
PBD (poly-p-phenylene terephthalamide)0Sending the data to a matching module, and turning to the fourth step;
fourthly, the event set selection module receives the event stream from the data acquisition module, respectively receives different data from the initialization module or the characteristic point updating module according to different k values, selects an event set S of characteristic points from the event stream around the n characteristic points, and enables the FD to be displayed on the display screenkAnd sending the event set S of the feature points to a matching module, wherein the method comprises the following steps:
4.1 if k is equal to 0, the event set selection module receives the event stream from the data acquisition module and receives t from the initialization modulekPosition set FD of n feature points at time0Let t1＝t0+1, unit is second, change 4.3;
4.2 if k is more than or equal to 1, the event set selection module receives the event stream from the data acquisition module and receives t from the feature point updating modulekPosition set FD of n feature points at timekAnd tk-1Optical flow set G of n feature points at timek-1，
Figure FDA0002904542740000035
Estimate sub-time interval [ t ]k,tk+1]Is calculated according to the formula (2)k+1：
Figure FDA0002904542740000036
Wherein
Figure FDA0002904542740000037
Is expressed in a sub-time interval tk-1,tk]Upper characteristic point fiThe optical flow of (a);
Figure FDA0002904542740000038
is expressed in a sub-time interval tk-1,tk]Turning to 4.3 by the average light flow of all the characteristic points;
4.3 at tkPosition set FD of n feature points at timekSelecting an event set corresponding to each feature point around each position, wherein the method comprises the following steps:
4.3.1 making i ═ 1;
4.3.2 selecting the events meeting the requirement of the formula (3) from the event stream, and putting the events to the characteristic point fiAt tkEvent set corresponding to time
Figure FDA0002904542740000039
In and (2) mixing
Figure FDA00029045427400000310
Put into event set S:
Figure FDA00029045427400000311
Figure FDA00029045427400000312
representing a set of events in a three-dimensional space region, the spatial extent being the characteristic point fiAt tkPosition of time fi(tk) A surrounding rectangular area
Figure FDA00029045427400000313
The time range is the interval [ tk,tk+1]；edRepresents
Figure FDA00029045427400000314
The d-th event of (1, 2., z)i}，ziTo represent
Figure FDA0002904542740000041
The number of internal events;
Figure FDA0002904542740000042
xdrepresents an event edThe coordinates in the pixel coordinate system are,
Figure FDA0002904542740000043
represents an event edThe time of occurrence of the event(s),
Figure FDA0002904542740000044
denotes edHas pixel coordinates in
Figure FDA0002904542740000045
Internal;
4.3.3 let i ═ i + 1;
4.3.4 if i is less than or equal to n, turning to 4.3.2; otherwise, it indicates that t is obtainedkEvent sets S corresponding to the n feature points in time FD,
Figure FDA0002904542740000046
FD will bekS is sent to a matching module, and the fifth step is carried out;
fifthly, the matching module receives FD from the event set selection modulekAnd S, if k is 0, receiving a position set PBD of the template edge corresponding to the n characteristic points from the initialization modulekTurning to the sixth step; if k is more than or equal to 1, receiving a template edge position set PBD corresponding to the n characteristic points from the template edge updating modulekTurning to the sixth step;
sixthly, the matching module matches the S with the template edges around the feature points to calculate tkOptical flow set G of n feature points at timekThe positions of n feature points are collected to form FDkAnd position sets PBD of template edges corresponding to n characteristic pointskAnd optical flow set GkAnd sending the data to a feature point updating module, wherein the specific method comprises the following steps:
6.1 making i ═ 1;
6.2 pairs of feature points fiConstructing a matching error function by the following method:
correcting event e according to equation (4)dAt tkTime of dayThe position of (2):
Figure FDA0002904542740000047
x′drepresenting a calculated event edAt tkLocation of time of day, abbreviated event edThe position of the motion correction is made,
Figure FDA0002904542740000048
is expressed in the time interval tk,tk+1]Upper characteristic point fiThe optical flow of (a); symbol-representing dot product, defining symbol
Figure FDA0002904542740000049
The match error function is constructed as follows:
Figure FDA00029045427400000410
e denotes an error, pj(tk) Represents tkTime template point pjThe position of (a); r isdjDenotes edFrom template point pjProbability of generation, i.e. edAnd pjThe probability of a match; the double vertical lines in the formula represent the modulo operation on the vector in the double vertical lines, rdjThe calculation formula of (a) is as follows:
Figure FDA00029045427400000411
6.3 adopting EM-ICP algorithm to minimize the matching error function, solving and obtaining the optimal light stream
Figure FDA0002904542740000051
The method comprises the following steps:
6.3.1 initialization
Figure FDA0002904542740000052
6.3.2 calculating r by equation (6)dj；
6.3.3 updating the optical flow:
Figure FDA0002904542740000053
6.3.4 calculating the amount of change in optical flow
Figure FDA0002904542740000054
And order
Figure FDA0002904542740000055
Will be provided with
Figure FDA0002904542740000056
Is put to tkTemporal optical flow set GkPerforming the following steps;
6.3.5 if
Figure FDA0002904542740000057
Shows that the final optimization result is obtained
Figure FDA0002904542740000058
6.4, rotating; if it is
Figure FDA0002904542740000059
6.3.2 is rotated; σ is a variation threshold of the optical flow;
6.4 let i ═ i + 1;
if i is less than or equal to n, 6.5, turning to 6.2; otherwise, it indicates that t is obtainedkTemporal optical flow set Gk，
Figure FDA00029045427400000510
FD will bek、PBDkAnd GkSending the data to a feature point updating module, and turning to the seventh step;
seventhly, the characteristic point updating module receives the FD from the matching modulek、PBDkAnd GkCalculating t by optical flowk+1Position set FD of n feature points at timek+1G iskAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkSending the data to a template edge updating module, wherein the method comprises the following steps:
7.1 making i ═ 1;
7.2 calculating tk+1Time fiPosition f ofi(tk+1) A 1 is to fi(tk+1) Put into the set FDk+1In (1),
Figure FDA00029045427400000511
Figure FDA00029045427400000512
by optical flow multiplied by time, from tkTime tk+1Time characteristic point fiA movement distance;
7.3 let i ═ i + 1;
7.4 if i is less than or equal to n, rotating to 7.2; otherwise, it indicates that t is obtainedk+1The positions of n feature points at the moment are obtained, and a set FD is obtainedk+1，FDk+1＝{f1(tk+1),...,fi(tk+1),...,fn(tk+1) }; g is to bekAnd FDk+1Sending the data to an event set selection module to select the FDk+1And PBDkSending the data to a template edge updating module; and will tk+1Position set FD of n feature points at timek+1Displaying or storing the result file, and turning to the eighth step;
eighthly, the template edge updating module receives IMU data from the data acquisition module and FD from the feature point updating modulek+1And PBDkUsing IMU data pairs PBDkIs updated to obtain tk+1Position set PBD of template edge corresponding to n characteristic points at momentk+1PBD ofk+1Sending the data to a matching module, wherein the method comprises the following steps:
8.1 making i ═ 1;
8.2 updating feature points fiCorresponding template edge at tk+1The method for the time position comprises the following steps:
8.2.1 let j equal 1;
8.2.2 defining the symbol F as being ANDiCorresponding point in three-dimensional space, PjIs a point p corresponding to the templatejPoints in the corresponding three-dimensional space, F and PjAre all represented in the form of three-dimensional coordinates (x, y, z); template point pjExpressed in homogeneous coordinate form, with the format of (x, y,1), the first two dimensions respectively represent pjAlong the abscissa and ordinate in the pixel coordinate system, the following equation is obtained:
at tkAt the moment of time, the time of day,
Figure FDA0002904542740000061
at tk+1At the moment of time, the time of day,
Figure FDA0002904542740000062
k is an internal parameter matrix of the event camera and is a self-contained parameter of the event camera; r is from tkTime tk+1The rotation matrix of the time-of-day event camera, t being from tkTime tk+1The translation vector of the moment event camera is obtained through calculation of the obtained IMU data;
Figure FDA0002904542740000063
is PjAt tkThe depth of the time camera coordinate system,
Figure FDA0002904542740000064
Is PjAt tk+1The depth in the camera coordinate system at the moment,
Figure FDA0002904542740000065
is F at tkTime of dayDepth under the camera coordinate system,
Figure FDA0002904542740000066
Is F at tk+1Depth under a time camera coordinate system;
8.2.3 subtracting the two equations in equation (10) to get tk+1Time template point pjRelative to fiRelative position of
Figure FDA0002904542740000067
Figure FDA0002904542740000068
Figure FDA0002904542740000071
PjAnd F is at tk+1Having the same depth in the camera coordinate system at the moment, i.e. at
Figure FDA0002904542740000072
Substituting equation (9) into equation (11), and simplifying equation (11) yields:
Figure FDA0002904542740000073
reduction of equation (12) to tk+1The formula of the relative position of the time template point is as follows:
Figure FDA0002904542740000074
the symbol Nor () represents a homogeneous operation, i.e., the coordinates in parentheses are converted into homogeneous coordinates; obtaining t according to equation (14)k+1Time template point pjPosition p ofj(tk+1) A 1 is to pj(tk+1) Is put to tk+1Time characteristic point fiSet of positions of surrounding formwork edges
Figure FDA0002904542740000075
Figure FDA0002904542740000076
8.2.4 let j ═ j + 1;
8.2.5 if j is less than or equal to miTurning to 8.2.2; otherwise it indicates that
Figure FDA0002904542740000077
Figure FDA0002904542740000078
Will be provided with
Figure FDA0002904542740000079
Is put to tk+1Position set PBD of template edges around n characteristic points at momentk+1Middle, 8.3;
8.3 let i ═ i + 1;
8.4 if i is less than or equal to n, rotating to 8.2; otherwise, the PBD is obtainedk+1，
Figure FDA00029045427400000710
PBD (poly-p-phenylene terephthalamide)k+1Sending the data to a matching module, and turning to the ninth step;
the ninth step, let k equal k + 1;
step ten, if k is less than N, turning to the step four; otherwise, ending.
2. The Event Camera-based feature point tracking method according to claim 1, wherein The Event Camera Dataset refers to "Event-Camera Dataset and Simulator" acquired by DAVIS, The Event-Camera Dataset and Simulator "comprising image frames, Event streams and IMU data; DAVIS refers to dynamic and active pixel sensors.
3. The event camera-based feature point tracking method of claim 1, wherein the template edge is represented by a set of all template points on the template edge.
4. A method for tracking feature points based on an event camera as claimed in claim 1 wherein step 3.2.3.2 said s ranges from 20 to 30 pixels.
5. The event camera-based feature point tracking method according to claim 1, wherein the value of σ in step 6.3.5 ranges from 0.01 to 0.1 in units of pixels per second.
CN201910672162.2A 2019-07-24 2019-07-24 Feature point tracking method based on event camera Active CN110390685B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910672162.2A CN110390685B (en) 2019-07-24 2019-07-24 Feature point tracking method based on event camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910672162.2A CN110390685B (en) 2019-07-24 2019-07-24 Feature point tracking method based on event camera

Publications (2)

Publication Number Publication Date
CN110390685A CN110390685A (en) 2019-10-29
CN110390685B true CN110390685B (en) 2021-03-09

Family

ID=68287327

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910672162.2A Active CN110390685B (en) 2019-07-24 2019-07-24 Feature point tracking method based on event camera

Country Status (1)

Country Link
CN (1) CN110390685B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111062337B (en) * 2019-12-19 2023-08-04 北京迈格威科技有限公司 People stream direction detection method and device, storage medium and electronic equipment
CN113160218B (en) * 2021-05-12 2023-06-20 深圳龙岗智能视听研究院 Method for detecting object motion intensity based on event camera
CN115984336A (en) * 2021-10-14 2023-04-18 华为技术有限公司 Optical flow estimation method and device
CN116188533B (en) * 2023-04-23 2023-08-08 深圳时识科技有限公司 Feature point tracking method and device and electronic equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109697726A (en) * 2019-01-09 2019-04-30 厦门大学 A kind of end-to-end target method for estimating based on event camera
WO2019099337A1 (en) * 2017-11-14 2019-05-23 Kaban Technologies Llc Event camera-based deformable object tracking

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109426782B (en) * 2017-08-29 2023-09-19 北京三星通信技术研究有限公司 Object detection method and neural network system for object detection
CN107808407B (en) * 2017-10-16 2020-12-18 亿航智能设备（广州）有限公司 Binocular camera-based unmanned aerial vehicle vision SLAM method, unmanned aerial vehicle and storage medium
CN109934862A (en) * 2019-02-22 2019-06-25 上海大学 A kind of binocular vision SLAM method that dotted line feature combines

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019099337A1 (en) * 2017-11-14 2019-05-23 Kaban Technologies Llc Event camera-based deformable object tracking
CN109697726A (en) * 2019-01-09 2019-04-30 厦门大学 A kind of end-to-end target method for estimating based on event camera

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Asynchronous Corner Detection and Tracking for Event Cameras in Real Time;Ignacio Alzugaray et al.;《IEEE Robotics and Automation Letters 》;20180622;第3卷(第4期);第3177-3184页 *
Event-based feature tracking with probabilistic data association;Alex Zihao Zhu et al.;《2017 IEEE International Conference on Robotics and Automation (ICRA)》;20170603;第4465-4470页 *
Low-latency visual odometry using event-based feature tracks;Beat Kueng et al.;《2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)》;20161014;第16-23页 *
基于无人机视觉的场景感知方法研究;谢榛;《中国博士学位论文全文数据库 工程科技Ⅱ辑》;20190115(第1期);第C031-73页 *

Also Published As

Publication number Publication date
CN110390685A (en) 2019-10-29

Similar Documents

Publication Publication Date Title
CN110390685B (en) Feature point tracking method based on event camera
CN111462200B (en) Cross-video pedestrian positioning and tracking method, system and equipment
Harville Stereo person tracking with adaptive plan-view templates of height and occupancy statistics
Clipp et al. Parallel, real-time visual SLAM
WO2020259481A1 (en) Positioning method and apparatus, electronic device, and readable storage medium
Irani et al. Computing occluding and transparent motions
Saeedi et al. Vision-based 3-D trajectory tracking for unknown environments
CN108492316A (en) A kind of localization method and device of terminal
CN103854283B (en) A kind of mobile augmented reality Tracing Registration method based on on-line study
CN113286194A (en) Video processing method and device, electronic equipment and readable storage medium
JP4668360B2 (en) Moving object detection method and moving object detection apparatus
US11138742B2 (en) Event-based feature tracking
EP1949339A2 (en) Tracking using an elastic cluster of trackers
US10636190B2 (en) Methods and systems for exploiting per-pixel motion conflicts to extract primary and secondary motions in augmented reality systems
CN106530407A (en) Three-dimensional panoramic splicing method, device and system for virtual reality
WO2023134114A1 (en) Moving target detection method and detection device, and storage medium
CN114494150A (en) Design method of monocular vision odometer based on semi-direct method
CN112861808B (en) Dynamic gesture recognition method, device, computer equipment and readable storage medium
Dardelet et al. An event-by-event feature detection and tracking invariant to motion direction and velocity
Xue et al. Event-based non-rigid reconstruction from contours
CN103345762B (en) Bayes's visual tracking method based on manifold learning
CN108534797A (en) A kind of real-time high-precision visual odometry method
CN114612545A (en) Image analysis method and training method, device, equipment and medium of related model
Zhou et al. Efficient tracking and ego-motion recovery using gait analysis
Soni et al. Real-time hand tracking using integrated optical flow and CAMshift algorithm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant