CN115546681A

CN115546681A - Asynchronous feature tracking method and system based on events and frames

Info

Publication number: CN115546681A
Application number: CN202211141601.5A
Authority: CN
Inventors: 邓若愚; 胡尚薇
Original assignee: Tongji Institute Of Artificial Intelligence Suzhou Co ltd
Current assignee: Tongji Institute Of Artificial Intelligence Suzhou Co ltd
Priority date: 2022-09-20
Filing date: 2022-09-20
Publication date: 2022-12-30

Abstract

The application provides an asynchronous feature tracking method and system based on events and frames, wherein the method comprises the following steps: a characteristic block initialization method based on FAST corner detection updates the position of a characteristic block based on an optical flow of event information integration and an affine transformation optimization method, verifies the effectiveness of the optical flow and the affine transformation based on an optimization result evaluation method of a historical loss function value, and determines new characteristic block initialization based on a characteristic block association method of nearest neighbor search. The method evaluates on the public data set, and the evaluation index is the time interval between the initialization of the characteristic and the loss of the characteristic, namely the characteristic time, and reflects the robustness of the tracker. Compared with the original method, the invention can improve the characteristic time by about 10-30% while ensuring the tracking precision, and lays a foundation for computer vision tasks such as vision odometer and the like.

Description

Asynchronous feature tracking method and system based on events and frames

Technical Field

The application relates to the technical field of Feature Tracking, in particular to an Asynchronous Feature Tracking (AFTEF Asynchronous features Tracking using Events and Frames) method and system based on Events and Frames.

Background

Dynamic vision sensors, an emerging biologically inspired event camera, have attracted the interest of robotics and computer vision researchers. Unlike conventional cameras, where the output information is absolute luminance frames, the output of the event camera is an asynchronous event stream that reacts to the luminance changes of local pixels. The event information includes a timestamp, a polarity, and pixel coordinates. Advantages of the event camera include low power consumption, high dynamic range, and high temporal resolution. Furthermore, the event camera is sensitive to motion of the scene, reflecting the brightness change for each pixel with low latency (1 μ s). Another bio-inspired sensor, the dynamic and active pixel vision sensor, can provide asynchronous event streams and luminance frames.

Feature detection and tracking is a key component of the field of feature-based visual odometry. Since the asynchronous event stream of the event camera is different from the luminance frame, the frame-based feature detection and tracking algorithm cannot be directly applied to the asynchronous event stream. Therefore, new algorithms need to be explored to handle such asynchronous event streams and free up their potential. In this case, how to exploit the information complementarity of the event camera and the conventional camera is a key issue required for feature detection and tracking.

In the field of feature detection of event cameras, most of current researches are extension and fusion of image-based popular corner detection methods. A frame-based method of extending the Harris corner has been proposed, which requires computation of gradients and convolution for detection on binary frames obtained from event accumulation. Inspired by the frame-based FAST (free from estimated Segment Test) corner detection method, an event-based corner detection method called eFAST has been proposed, which detects on an Active event Surface (Surface of Active Events SAE) and only needs to perform comparison operation. SAE is a two-dimensional representation of the event stream that stores the timestamp of the most recent event at each pixel location. In order to improve the robustness of eFAST, an event feature detection method based on SAE filtering has been proposed, which is named Arc. This algorithm detects corners faster than eFAST and ehharris, while enhancing the repeatability of corner detection. FA-Harris provides a selection and refinement strategy that uses improved eFAST to select candidate points and filters with improved efaris. A method has been proposed for detection by three stages of filtering and low complexity Harris.

In the field of feature tracking for event cameras, researchers are working on implementing event-driven asynchronous feature tracking methods using event information. A probability-based correlation method between event streams and features has been proposed, in which event corner point tracking is described as an optimization problem matching the current view and feature template, and only a set of discrete tracking hypotheses need to be evaluated. An event feature tracking method based on a tree structure and having multiple data association possibilities has been proposed, in which each node is an event corner, and the matching mechanism for adding nodes is based on spatio-temporal constraints. Based on spatio-temporal constraints, constraints on the direction of the corner points have been added to the tree structure. This improvement can cut off branches and simplify the tree structure. A gradient descriptor based on a time plane with constant velocity has been proposed and used as a basis for matching two event corners in a tree structure. A first feature tracking method using both frame and event cameras has been proposed, in which feature blocks are initialized on a frame using corner and edge detectors, and then aligned using an event stream through a two-dimensional euclidean transform to achieve tracking.

However, the above method does not fully utilize the advantages of the event information. Specifically, the event information includes not only spatial and temporal information but also polarity information. These methods only utilize spatio-temporal information and do not utilize polarity information. Furthermore, these methods do not use the information of a conventional camera as a supplement.

A better combination of frames and event cameras is the asynchronous photometric feature tracking (EKLT) method using events and frames. This method detects features on the luminance frame and then uses the event stream for tracking. It realizes asynchronous tracking and makes full use of the polarity information of the event stream.

Disclosure of Invention

In view of this, an object of the present application is to provide an asynchronous feature tracking method and system based on events and frames, which can specifically solve the existing problems.

Based on the above purpose, the present application provides an asynchronous feature tracking method based on events and frames, including:

step 1, using a FAST corner detection method based on a decision tree on a frame, and using a non-maximum value to suppress to obtain initialized feature points;

step 2, extracting feature blocks on the image plane around the feature points;

step 3, establishing an observation model of the event generated by the event camera;

step 4, integrating the polarity information of the event stream by using an observation model to obtain an observation value of the luminosity increment image;

step 5, based on the principle of luminosity invariance of local feature blocks, obtaining a predicted value of a luminosity increment image through image gradient, optical flow and affine transformation;

step 6, establishing a loss function based on the difference value of the two normalized luminosity increment images, and minimizing the loss function by using a nonlinear optimization method to obtain the motion parameters of the feature block;

step 7, evaluating based on the historical value of the loss function, and judging the effectiveness of the motion parameters;

step 8, updating the position of the feature block by using the optical flow and the affine transformation;

step 9, after initializing new feature points, traversing all existing feature blocks, finding out the nearest initial feature point according to Euclidean distance, and ensuring that the distance is lower than a threshold value;

and step 10, taking the unmatched initialization feature points as centers to extract new feature blocks.

Further, the FAST corner detection method in step 1 is a frame-based corner detection method.

Further, the event camera in step 3 is a bionic visual sensor which asynchronously outputs discrete information, and the output information is called an event, and comprises image plane coordinates, a time stamp and a polarity.

Further, the polarity information of the event in the step 4 is integrated, and the polarity information of the event whose position falls within the feature block is accumulated.

Further, the principle of the luminosity invariance of the local feature block in step 5 is to assume that the local luminosity of the image plane is unchanged in a very short time, the image gradient is a difference value of the luminosity value in two directions of the image plane, the optical flow is a movement speed of a point in the image plane, and the affine transformation is a rotation and translation matrix of a two-dimensional plane.

Further, the euclidean distance in step 9 is a distance between two-dimensional points in the image plane.

In view of the above, the present application further provides an event and frame based asynchronous feature tracking system, including:

the initialization module initializes the characteristic blocks of the frames based on FAST corner detection;

the updating module updates the position of the feature block based on the optical flow of the event information integration and the affine transformation optimization method;

the verification module is used for verifying the effectiveness of the optical flow and the affine transformation based on an optimization result evaluation method of the historical loss function value;

and the association module determines the initialization of a new feature block based on the feature block association method of the nearest neighbor search.

In general, the advantages of the present application and the experience brought to the user are: the method can ensure the tracking precision and improve the tracking time. Compared with the existing method, the overall quality of the track is improved, the characteristic time is delayed, and the tracking accuracy is ensured. The present invention provides the necessary foundation for the future development of fully event driven visual odometers and other computer vision tasks.

Drawings

In the drawings, like reference characters designate like or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily to scale. It is appreciated that these drawings depict only some embodiments in accordance with the disclosure and are therefore not to be considered limiting of its scope.

FIG. 1 shows a flow diagram of a method for asynchronous event and frame based feature tracking according to an embodiment of the application.

Fig. 2 shows a schematic diagram of the detection method of the original FAST corner.

Fig. 3 shows a visualization diagram of event flow information.

Fig. 4 shows a visualization diagram of a data set.

FIG. 5 shows a schematic view of a trace visualization of a data set.

FIG. 6 is a graph showing a comparison of the mean tracking error of the AFTEF method and the EKLT method of the present application.

FIG. 7 is a graph showing a comparison of the characteristic times of the AFTEF method and the EKLT method of the present application.

FIG. 8 illustrates a block diagram of an event and frame based asynchronous feature tracking system according to an embodiment of the present application.

Fig. 9 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.

FIG. 10 is a schematic diagram of a storage medium provided by an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

FIG. 1 shows a flow diagram of a method for asynchronous event and frame based feature tracking according to an embodiment of the application. As shown in fig. 1, the asynchronous feature tracking method based on events and frames includes:

step 1, performing corner detection on a frame by using a FAST corner detection method based on a decision tree through the brightness values of pixel points on a circular arc shown in FIG. 2, and obtaining initialized feature points by using non-maximum value inhibition;

step 2, extracting feature blocks on the image plane around the feature points;

as shown in fig. 3, the events in the image plane and the time axis are discrete information, and include only spatial information, temporal information, and polarity information. Specifically, an event { u, pol, t } contains the two-dimensional position of the pixel u = { x, y }, the polarity pol ∈ { +1, -1} that represents the positive or negative of the brightness change, and the time stamp t of the triggering event. An event occurs when the logarithmic luminance change between t and t- Δ t for a pixel location is above a threshold value ± C (C > 0). q (u, t) = log (I (u, t)) is a logarithmic luminance image, and the logarithmic luminance incremental image Δ q (u, t) can be expressed as:

Δq(u，t)＝q(u，t)-q(u，t-Δt)＝pol*C (1)

where t- Δ t is the timestamp of the last event at the same pixel location.

when the number of events falling into P within the feature block reaches the adaptive threshold N _e The polarity of the event at each pixel location in the feature block is accumulated over a time interval Δ τ. As shown in equation (2), Δ q (u, t) is referred to as an observed value of a photometric incremental image:

where f (u, t) is the polarity of all events at the time stamp t for pixel location u. Note that the polarity is-1 or +1 and the generation of the event is based on the threshold C. The photometric delta image is then normalized. Adaptive threshold value N _e Is initialized to a constant value (e.g., 100) and subsequently based on image gradient

And recalculating after optimizing the optical flow v:

step 5, based on the principle of local luminosity invariance, obtaining a predicted value of a luminosity increment image through image gradient, optical flow and affine transformation;

consider the case where optical flow is unknown. The derivative of q (u, t) = const, q (u, t) assuming that the gradient and logarithmic luminance image in the feature block are constant over time is expressed as:

wherein,

is the luminance gradient at the pixel location and v is the optical flow. The taylor approximation of equation 4 can be expressed as:

substituting equation 4 into equation 5:

referred to as the prediction value of the photometric delta image, because the optical flow is an unknown value. In fact, the gradient will vary according to the motion parameter p:

W(u，p)＝R(p)*u+t(p) (7)

wherein (R, t) ∈ SE (2) is a rotation and translation quantity represented by a lie group, and p ∈ SE (2) is a corresponding lie algebra, and after affine transformation is performed on the gradient, a predicted value of the luminosity increment image can be written as:

step 6, establishing a loss function based on the difference value of the two normalized luminosity increment images, and minimizing the loss function by using a nonlinear optimization method to obtain the motion parameters (optical flow and affine transformation) of the feature block;

where the loss function V is written as:

the motion parameter after each optimization is p _last And v _last Corresponding to a final loss function value of V _last ：

And (3) taking the latest n suboptimal final loss function values of the feature blocks, and calculating an average value:

the quality of the optimization result is based on V _average And a threshold value c _threshold The size of the compared values. The threshold is a constant parameter preset empirically. If it exceeds the threshold c _threshold The optimization will fail and the state of the feature block is set to lost. Otherwise, the position of the feature block is updated.

the optimized motion parameters (optical flow v and affine transformation p) are used to update the feature block positions:

u′＝R(p) ^-1 *u-R(p) ^-1 *t(p) (12)

step 9, after initializing new feature points, traversing all existing feature blocks, finding the nearest initial feature point according to Euclidean distance, and ensuring that the distance is lower than a threshold value;

The algorithm verifies that (a) is a simple black and white scene, (b) and (c) are high texture scenes, and (d) and (e) are natural scenes in the data set scene shown in fig. 4. The schematic diagram of the trace obtained by tracking the feature block is shown in fig. 5, and the clustered short lines represent the trace of the feature block. FIG. 6 shows the comparison of the mean tracking error of the AFTEF method and the EKLT method of the present application, and FIG. 7 shows the comparison of the characteristic times of the AFTEF method and the EKLT method of the present application. As can be seen from the results of fig. 6 and 7, compared with the EKLT method, the AFTEF method proposed in the present application improves the feature time of the whole feature block by about 10% to 30% while ensuring the feature tracking accuracy.

The application embodiment provides an event and frame based asynchronous feature tracking system, which is used for executing the event and frame based asynchronous feature tracking method described in the above embodiment, as shown in fig. 8, and the system includes:

an initialization module 501, which initializes the feature blocks of the frame based on FAST corner detection;

an updating module 502 for updating the feature block position based on the optical flow of the event information integration and the affine transformation optimization method;

the verification module 503 is used for verifying the effectiveness of the optical flow and the affine transformation based on the optimization result evaluation method of the historical loss function value;

the association module 504 determines a new feature block initialization based on a feature block association method of nearest neighbor search.

The asynchronous feature tracking system based on events and frames provided by the above embodiment of the present application and the asynchronous feature tracking method based on events and frames provided by the embodiment of the present application have the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by the application program stored in the asynchronous feature tracking system based on events and frames.

The embodiment of the present application further provides an electronic device corresponding to the asynchronous feature tracking method based on events and frames provided by the foregoing embodiment, so as to execute the asynchronous feature tracking method based on events and frames. The embodiments of the present application are not limited.

Please refer to fig. 9, which illustrates a schematic diagram of an electronic device according to some embodiments of the present application. As shown in fig. 9, the electronic device 20 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected through the bus 202; the memory 201 stores a computer program that can be executed on the processor 200, and the processor 200 executes the computer program to perform the asynchronous feature tracking method based on events and frames provided by any of the foregoing embodiments of the present application.

The Memory 201 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.

Bus 202 can be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used for storing a program, and the processor 200 executes the program after receiving an execution instruction, and the method for tracking asynchronous features based on events and frames disclosed in any embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

The processor 200 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 200. The Processor 200 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the method in combination with the hardware thereof.

The electronic device provided by the embodiment of the present application and the asynchronous feature tracking method based on events and frames provided by the embodiment of the present application have the same beneficial effects as the method adopted, operated or implemented by the electronic device.

The embodiment of the present application further provides a computer-readable storage medium corresponding to the asynchronous feature tracking method based on event and frame provided in the foregoing embodiment, please refer to fig. 10, which illustrates a computer-readable storage medium being an optical disc 30, on which a computer program (i.e., a program product) is stored, where the computer program, when executed by a processor, executes the asynchronous feature tracking method based on event and frame provided in any of the foregoing embodiments.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, a phase change memory (PRAM), a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), other types of Random Access Memories (RAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash memory, or other optical and magnetic storage media, which are not described in detail herein.

The computer-readable storage medium provided by the above-mentioned embodiment of the present application and the event and frame-based asynchronous feature tracking method provided by the embodiment of the present application have the same beneficial effects as the method adopted, run or implemented by the application program stored in the computer-readable storage medium.

It should be noted that:

the algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. In addition, this application is not directed to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present application as described herein, and any descriptions of specific languages are provided above to disclose the best modes of the present application.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the application and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this application.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the application and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a virtual machine creation system according to embodiments of the present application. The present application may also be embodied as apparatus or system programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present application may be stored on a computer readable medium or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems can be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various changes or substitutions within the technical scope of the present application, and these should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An asynchronous feature tracking method based on events and frames is characterized by comprising the following steps:

step 2, extracting feature blocks from image planes around the feature points;

step 8, updating the position of the feature block by using the optical flow and affine transformation;

2. The event and frame based asynchronous feature tracking method of claim 1, wherein: the FAST corner detection method in step 1 is a frame-based corner detection method.

3. The event and frame based asynchronous feature tracking method of claim 1, wherein: the event camera in step 3 is a bionic visual sensor which asynchronously outputs discrete information, and the output information is called an event and comprises image plane coordinates, a time stamp and polarity.

4. The event and frame based asynchronous feature tracking method of claim 1, wherein: and integrating the polarity information of the events in the step 4, and accumulating the polarity information of the events of which the positions fall into the feature block.

5. The event and frame based asynchronous feature tracking method of claim 1, wherein: the principle of the local feature block luminosity invariance in the step 5 is that the local luminosity of the image plane is assumed to be unchanged in a very short time, the image gradient is a difference value of the luminosity values in two directions of the image plane, the optical flow is the movement speed of a point in the image plane, and the affine transformation is a rotation translation matrix of a two-dimensional plane.

6. The event and frame based asynchronous feature tracking method of claim 1, characterized by: the euclidean distance in step 9 is the distance between two-dimensional points on the image plane.

7. An asynchronous event and frame based feature tracking system, comprising:

and the association module determines the initialization of a new feature block based on a feature block association method of nearest neighbor search.

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor to implement the method according to any of claims 1-6.