CN112734797A

CN112734797A - Image feature tracking method and device and electronic equipment

Info

Publication number: CN112734797A
Application number: CN201911038964.4A
Authority: CN
Inventors: 鲍虎军; 章国锋; 杨镑镑; 王楠
Original assignee: Zhejiang Shangtang Technology Development Co Ltd
Current assignee: Zhejiang Shangtang Technology Development Co Ltd; Zhejiang Sensetime Technology Development Co Ltd
Priority date: 2019-10-29
Filing date: 2019-10-29
Publication date: 2021-04-30

Abstract

The application provides an image feature tracking method, an image feature tracking device and electronic equipment, wherein the method comprises the following steps: for an acquired image frame sequence, determining a feature point set of each current image frame, wherein the feature point set comprises: optical flow tracking points; performing descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points; and determining a feature tracking result of the current image frame according to the optical flow tracking point and the tracking information of the descriptor tracking point. In the process of determining the image feature tracking result, the advantages of optical flow method feature tracking and descriptor method feature tracking are effectively combined, and for image sequences shot in complex scenes such as rapid or violent motion scenes, scenes with insufficient light rays and the like, equipment with limited computing capacity can also accurately and stably track feature points representing the same object, so that the visual tracking effect is improved, and the visual tracking power consumption of the equipment is reduced.

Description

Image feature tracking method and device and electronic equipment

Technical Field

The application relates to the technical field of computer vision, in particular to an image feature tracking method and device and electronic equipment.

Background

Image feature tracking is an important functional module of a visual SLAM (simultaneous localization and mapping) system, and directly affects the efficiency and robustness of the SLAM system. In the related art, feature tracking may be performed when a camera rapidly photographs an image of a scene using an optical flow method or a descriptor method. Among them, the optical flow method can be used for efficiently tracking features in a fast motion, but the weak matching characteristic of the optical flow method enables tracking points to easily drift; and the descriptor method is used for carrying out feature tracking, although the obtained tracking result has higher matching precision and less drift, under the condition of quick motion, feature points are easy to lose, and the ideal tracking effect can be achieved only by using the descriptor method which usually needs to track more feature points and consumes more computing resources than the optical flow method.

In an actual application scenario of an electronic device such as an Augmented Reality (AR) of a mobile phone, if a user carries the electronic device to move relatively fast or if the imaging quality of an image sensor is limited, the efficiency and robustness of feature tracking cannot be taken into consideration at the same time, so that the AR experience of the user is poor.

Disclosure of Invention

The embodiment of the application provides an image feature tracking method and device and electronic equipment.

According to a first aspect of embodiments of the present application, there is provided an image feature tracking method, including:

for an acquired image frame sequence, determining a feature point set of each current image frame, wherein the feature point set comprises: optical flow tracking points;

performing descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points;

and determining a feature tracking result of the current image frame according to the optical flow tracking point and the tracking information of the descriptor tracking point.

In combination with any one of the method embodiments provided herein, the determining a set of feature points for each current image frame includes:

acquiring optical flow tracking information of a current image frame, and determining optical flow tracking points in the current image frame based on the optical flow tracking information;

extracting feature points of the current image frame to obtain feature extraction points in the current image frame;

determining a feature point set of the current image frame according to the optical flow tracking points and the feature extraction points;

wherein, the feature point set of the current image frame comprises two types of feature points: the optical flow tracking points and the feature extraction points.

In combination with any method embodiment provided by the present application, the performing descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points includes:

calculating a descriptor for each feature point in the feature point set to obtain a descriptor of a target feature point;

and matching the feature points based on the descriptors of the target feature points, and determining the tracking information of the descriptor tracking points.

In combination with any method embodiment provided by the present application, the performing feature point matching based on the descriptor of the target feature point, and determining the tracking information of the descriptor tracking point includes:

determining a search range center of a first target feature point in a current image frame for each first target feature point in a previous image frame, wherein the first target feature point is a feature point with a descriptor in the previous image frame;

determining descriptors of all second target feature points in a target search range, wherein the target search range is a range determined based on the center of the search range;

and performing feature point matching according to the descriptor of the first target feature point and the descriptors of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

In combination with any one of the method embodiments provided herein, the first target feature point has a corresponding target optical flow tracking point in the current image frame;

the determining a search range center of the first target feature point in the current image frame includes:

determining whether a three-dimensional position of the first target feature point has been initialized;

if the three-dimensional position of the first target feature point is initialized, determining a projection prediction position of the initialized three-dimensional point in the current image frame;

determining a first distance between the target optical flow tracking point and the projected predicted location;

if the first distance is smaller than or equal to a preset first distance threshold value, determining the target optical flow tracking point as the center of the search range;

and if the first distance is greater than the first preset distance threshold, determining the projection prediction position as the center of the search range.

In combination with any one of the method embodiments provided herein, the determining a search range center of the first target feature point in the current image frame further includes:

and if the three-dimensional position of the first target feature point is not initialized, determining the target optical flow tracking point as the center of the search range.

In combination with any one of the method embodiments provided herein, the first target feature point has no corresponding target optical flow tracking point in the current image frame;

and determining the projection prediction position as the center of the search range.

if the three-dimensional position of the first target feature point is not initialized, determining a homography matrix representing the mapping relation between the current image frame and the previous image frame;

and determining the center of the search range of the first target characteristic point according to the homography matrix.

In combination with any one of the method embodiments provided herein, the determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame includes:

determining a number of the optical flow tracking points in the set of feature points;

if the number of the optical flow tracking points is larger than a preset number threshold, determining pixel coordinate information of each optical flow matching point pair according to optical flow tracking information of the optical flow tracking points;

determining the homography matrix according to the pixel coordinate information of each optical flow matching point pair;

wherein the optical flow matching point pairs are feature point pairs matched in the previous image frame and the current image frame by an optical flow tracking method.

In combination with any one of the method embodiments provided by the present application, the describing the tracking information of the sub-tracking point includes: the method comprises the following steps of (1) tracking point positions and related tracking information, wherein the related tracking information is used for representing a determination mode of a search range center in the process of determining descriptor tracking points;

the determining a feature tracking result of the current image frame according to the tracking information of the optical flow tracking point and the descriptor tracking point comprises:

determining an initial feature tracking result of the current image frame according to the optical flow tracking point and the tracking information of the descriptor tracking point;

and removing the characteristic points of the initial characteristic tracking result according to the related tracking information of each tracking point to obtain a target characteristic tracking result.

In combination with any one of the method embodiments provided by the present application, if a first target feature point in a previous image frame is respectively matched to an optical flow tracking point and a descriptor tracking point in the current image frame;

the determining an initial feature tracking result of the current image frame according to the tracking information of the optical flow tracking points and the descriptor tracking points comprises:

determining a second distance between the optical flow tracking point and the descriptor tracking point aiming at the same first target feature point;

if the second distance is smaller than or equal to a preset second distance threshold value, determining the optical flow tracking point as a target tracking point of the first target feature point;

and if the second distance is greater than the preset second distance threshold, determining the descriptor tracking point as a target tracking point of the first target characteristic point.

In combination with any method embodiment provided by the present application, the performing feature point elimination on the initial feature tracking result according to the related tracking information of each tracking point to obtain a target feature tracking result includes:

determining the score of each tracking point in the initial characteristic tracking result according to the relevant tracking information of each tracking point and a preset weight assigning rule;

sequencing the tracking points according to the sequence of the tracking point scores from high to low to obtain a tracking point ordered list, wherein the tracking point ordered list comprises: the corresponding relation between the tracking point index and the tracking point score;

and eliminating the tracking points with close distances in the current image frame according to the tracking point sorting list to obtain the target characteristic tracking result.

In combination with any one of the method embodiments provided by the present application, the sorting the scores of the tracking points in order of assigned score values from high to low includes:

if the scores of the tracking points are equal, updating the scores of the tracking points by using the times of tracking the tracking points in the front image frame;

sorting the plurality of tracking points based on the updated tracking point scores.

According to a second aspect of the embodiments of the present application, there is provided a scene creation method, including:

acquiring an image frame sequence;

performing image feature point tracking on each current image frame in the image frame sequence by using any image feature tracking method embodiment provided by the application to obtain an image feature tracking result;

estimating pose information and three-dimensional point cloud information of the terminal equipment according to an image feature tracking result of the image frame sequence;

and creating an augmented reality scene according to the pose information and the three-dimensional point cloud information of the terminal equipment.

According to a third aspect of embodiments of the present application, there is provided an image feature tracking apparatus including:

a feature point determination module configured to determine, for the acquired sequence of image frames, a set of feature points for each current image frame, the set of feature points including: optical flow tracking points;

the tracking information determining module is configured to perform descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points;

a feature tracking module configured to determine a feature tracking result of the current image frame according to tracking information of the optical flow tracking points and the descriptor tracking points.

In combination with any apparatus embodiment provided by the present application, the feature point determining module includes:

an optical flow tracking sub-module configured to acquire optical flow tracking information of a current image frame and determine optical flow tracking points in the current image frame based on the optical flow tracking information;

a feature point extraction submodule configured to perform feature point extraction on the current image frame to obtain feature extraction points in the current image frame;

a feature point set determination submodule configured to determine a feature point set of the current image frame from the optical flow tracking points and the feature extraction points;

In combination with any apparatus embodiment provided herein, the tracking information determining module includes:

a descriptor determining submodule configured to calculate a descriptor for each feature point in the feature point set, and obtain a descriptor of a target feature point;

and the descriptor tracking sub-module is configured to perform feature point matching based on the descriptor of the target feature point and determine the tracking information of the descriptor tracking point.

In combination with any one of the apparatus embodiments provided herein, the descriptor tracking sub-module includes:

a search center determining unit configured to determine, for each first target feature point in a previous image frame, a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame;

a descriptor determination unit configured to determine a descriptor of each second target feature point within a target search range, the target search range being a range determined based on the search range center;

and the matching unit is configured to perform feature point matching according to the descriptor of the first target feature point and the descriptors of the second target feature points to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

In combination with any apparatus embodiment provided herein, the first target feature point has a corresponding target optical flow tracking point in the current image frame;

the search center determining unit includes:

a position initialization determination subunit configured to determine whether a three-dimensional position of the first target feature point has been initialized;

a projection position determination subunit configured to determine, in a case where the three-dimensional position of the first target feature point has been initialized, a projection prediction position of the initialized three-dimensional point in the current image frame;

a first distance determination subunit configured to determine a first distance between the target optical flow tracking point and the projected predicted position;

a first search center determining subunit configured to determine the target optical flow tracking point as the search range center if the first distance is equal to or less than a preset first distance threshold;

a second search center determining subunit configured to determine the projected predicted position as the search range center if the first distance is greater than the first preset distance threshold.

In combination with any apparatus embodiment provided by the present application, the search center determining unit is configured to determine the target optical flow tracking point as the search range center if the three-dimensional position of the first target feature point is not initialized.

In combination with any apparatus embodiment provided herein, the first target feature point has no corresponding target optical flow tracking point in the current image frame;

the search center determining unit includes:

a third search center determining subunit configured to determine the projected predicted position as the search range center.

In combination with any one of the apparatus embodiments provided in the present application, the search center determining unit further includes:

a homography matrix determination subunit configured to determine a homography matrix representing a mapping relationship between the current image frame and the previous image frame in a case where the three-dimensional position of the first target feature point is not initialized;

a fourth search center determining subunit configured to determine a search range center of the first target feature point according to the homography matrix.

In combination with any one of the apparatus embodiments provided herein, the homography matrix determination subunit includes:

a number determination unit configured to determine the number of the optical flow tracking points in the feature point set;

the pixel coordinate determination unit is configured to determine pixel coordinate information of each optical flow matching point pair according to optical flow tracking information of the optical flow tracking points if the number of the optical flow tracking points is larger than a preset number threshold;

a matrix determination unit configured to determine the homography matrix from the pixel coordinate information of each of the optical flow matching point pairs;

In combination with any one of the apparatus embodiments provided herein, the tracking information describing the sub-tracking point includes: the method comprises the following steps of (1) tracking point positions and related tracking information, wherein the related tracking information is used for representing a determination mode of a search range center in the process of determining descriptor tracking points;

the feature tracking module includes:

an initial tracking result determining sub-module configured to determine an initial feature tracking result of the current image frame according to the optical flow tracking points and the tracking information of the descriptor tracking points;

and the target tracking result determining submodule is configured to remove the characteristic points of the initial characteristic tracking result according to the related tracking information of each tracking point to obtain a target characteristic tracking result.

In combination with any of the apparatus embodiments provided herein, if a first target feature point in a previous image frame is respectively matched to an optical flow tracking point and a descriptor tracking point in the current image frame;

the initial tracking result determining sub-module includes:

a second distance determination unit configured to determine, for the same first target feature point, a second distance between the optical flow tracking point and the descriptor tracking point;

a first target determination unit configured to determine the optical flow tracking point as a target tracking point of the first target feature point when the second distance is equal to or less than a preset second distance threshold;

a second target determination unit configured to determine the descriptor tracking point as a target tracking point of the first target feature point if the second distance is greater than the preset second distance threshold.

In combination with any one of the apparatus embodiments provided in the present application, the target tracking result determining sub-module includes:

the score determining unit is configured to determine a score of each tracking point in the initial feature tracking result according to relevant tracking information of each tracking point and a preset weight assigning rule;

a sorting unit configured to sort the tracking points in order of the tracking point scores from high to low to obtain a tracking point sorted list, where the tracking point sorted list includes: the corresponding relation between the tracking point index and the tracking point score;

and the tracking result determining unit is configured to remove the tracking points with close distances in the current image frame according to the tracking point sorting list to obtain the target feature tracking result.

In combination with any apparatus embodiment provided herein, the sorting unit includes:

the score updating subunit is configured to update the scores of the tracking points by using the times of tracking the tracking points in the front image frame if the scores of the tracking points are equal;

a ranking subunit configured to rank the plurality of tracking points based on the updated tracking point scores.

According to a fourth aspect of embodiments of the present application, there is provided a scene creation apparatus, including:

an image acquisition module configured to acquire a sequence of image frames;

the image feature tracking module is configured to track image feature points of each current image frame in the image frame sequence by using any image feature tracking device provided by the application to obtain an image feature tracking result;

an information estimation module configured to estimate pose information and three-dimensional point cloud information of a terminal device according to an image feature tracking result of the image frame sequence;

and the scene creating module is configured to create an augmented reality scene according to the pose information and the three-dimensional point cloud information of the terminal equipment.

According to a fifth aspect of embodiments herein, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method of any one of the above.

According to a sixth aspect of embodiments of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the above when executing the program.

According to a fifth aspect of embodiments of the present application, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method of any one of the first and second aspects described above.

According to a sixth aspect of embodiments of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the first and second aspects when executing the program.

By adopting the image feature tracking method provided by the embodiment of the application, in the process of carrying out image feature tracking on the current image frame, the feature point set of the current image frame can be determined based on the optical flow tracking information; and finally, determining an image feature tracking result of the current image frame according to tracking information respectively corresponding to the optical flow tracking point and the descriptor tracking point. In the process of determining the image feature tracking result, the method and the device effectively integrate the advantages of optical flow method feature tracking and descriptor method feature tracking, and for image sequences shot in complex scenes such as rapid or violent motion scenes, scenes with insufficient light rays and the like, the device with limited computing power can also accurately and stably track feature points representing the same object, so that the visual tracking effect is improved, and the visual tracking power consumption of the device is reduced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a flow chart of an image feature tracking method shown herein according to an exemplary embodiment;

FIG. 2 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 3 is a schematic diagram illustrating an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

FIG. 4 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 5 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 6 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 7 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 8 is a schematic diagram illustrating an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram illustrating an application scenario of an image feature tracking method according to an exemplary embodiment of the present application;

FIG. 10 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 11-1 is a schematic diagram illustrating an application scenario of an image feature tracking method according to an exemplary embodiment of the present application;

FIG. 11-2 is a schematic diagram illustrating an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

FIG. 12 is a flow chart of another image feature tracking method illustrated herein in accordance with an exemplary embodiment;

FIG. 13 is a schematic diagram illustrating an application scenario of another image feature tracking method according to an exemplary embodiment of the present application;

FIG. 14 is a flowchart of a scene creation method shown herein in accordance with an illustrative embodiment;

FIG. 15 is a block diagram of an image feature tracking device shown in accordance with an exemplary embodiment of the present application;

FIG. 16 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 17 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 18 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 19 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 20 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment of the present application;

FIG. 21 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 22 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 23 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 24 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 25 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 26 is a block diagram of another image feature tracking device illustrated in accordance with an exemplary embodiment;

FIG. 27 is a schematic diagram illustrating an electronic device according to an exemplary embodiment of the present application;

fig. 28 is a schematic structural diagram of another electronic device shown in the present application according to an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is merely one type of associative relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

The application provides an image feature tracking method which can be applied to application scenes such as intelligent driving, man-machine interaction, security monitoring and Augmented Reality (AR).

Image feature point tracking is a fundamental problem in computer vision, where it is necessary to first find a combination of adjacent image matching points, and from these matching point pairs, calculate the pose of the camera, i.e. the rotation and translation information of the camera with respect to the initial position.

In the embodiment of the present application, the related execution main body includes: computer system and the camera of real-time acquisition image. The camera sends the scene image data acquired in real time to the computer system, so that the computer system processes the scene image data, determines corresponding feature point pairs of a current image frame and a previous image frame, calculates the pose change of the camera according to the information of the corresponding feature point pairs, and facilitates the computer system to perform subsequent data processing according to the pose change of the camera, for example, to construct an augmented reality scene at the current moment.

The computer system may be a computer system in an electronic device such as a personal computer system, a vehicle-mounted device, or a mobile terminal, or may be a computer system installed in a server, a server cluster, or a cloud platform. The camera may be a camera of a terminal device, and the terminal device may be an electronic device such as a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, and a wearable device. In a specific implementation process, the camera and the computer system are independent from each other and are mutually connected to jointly implement the image feature tracking method provided by the embodiment of the application. The following describes the image feature tracking method provided in the present application in detail by taking a computer system as an example.

The embodiment of the application provides a method for tracking image features by fusing an optical flow method and a descriptor method, which can be applied to an SLAM system.

Before the image feature tracking method provided by the present application is introduced, feature point matching between two frames of images, i.e., image feature tracking, based on an optical flow method and a descriptor method is introduced first.

The image feature tracking is performed by using an optical flow method, wherein two continuous image frame data and a feature point array detected in a previous image frame are input into a preset optical flow tracking function, such as cv:: calcOptical FlowPyrLK function, and the optical flow tracking function returns a new set of feature point positions, which represent the positions of corresponding points of the feature points of the previous image frame in the current image frame.

Image feature tracking with respect to the descriptor method: respectively extracting feature point information in two frames of images, wherein each feature point information comprises: a Keypoint (Keypoint) and a Descriptor (Descriptor); and performing image feature matching according to the feature point information in the two frames of images.

Wherein, the key point usually represents the position information of the characteristic point in the image; the descriptor is used for describing the information of key points and surrounding pixels, the change of the image characteristics in the scale and the direction caused by the change of the visual angle can be eliminated through the descriptor, and the characteristic point matching can be accurately carried out among the images. According to the design characteristics of the descriptors, the two key points with similar appearances have similar descriptors, so that the matching of the descriptors can accurately judge which feature points in the two frames of images are the same point.

Based on this, referring to fig. 1, a flowchart of an image feature tracking method is shown according to an exemplary embodiment, which may include:

in step 11, for the acquired image frame sequence, a feature point set of each current image frame is determined, where the feature point set includes: optical flow tracking points and feature extraction points;

the computer system can acquire the image frame sequence shot by the camera in real time, and the current image frame belongs to any image in the image frame sequence.

In the embodiment of the application, one feature point set can be determined for each current image frame. The feature point set of the current image frame is a set composed of a plurality of feature points.

The feature points are features that can reflect the same object in the image taken when the camera moves and rotates, that is, the angle of view changes. Namely, the same object in the image shot by the camera under different visual angles can be found out through the characteristic points, and the pose change of the camera is determined according to the position change of the characteristic points in different images.

In the embodiment of the present application, the feature point set of the current image frame includes two types of feature points: optical flow tracking points and feature extraction points.

The optical flow tracking points refer to feature points tracked in a current image frame by an optical flow method. According to the related knowledge, optical flow (optical flow) describes the motion of a corresponding projection point after a three-dimensional motion point is projected to a two-dimensional image, and can be regarded as the instantaneous displacement of the two-dimensional projection point, namely, the optical flow is the displacement of the same point in two adjacent frames. Based on the position of a feature point in the previous image frame, the position of the feature point in the current image frame can be calculated by using the optical flow information of the feature point; in the embodiment of the present application, the feature points determined in the current image frame by the optical flow method are referred to as optical flow tracking points.

The optical flow tracking points can be classified into two types according to the source of the optical flow tracking information:

a first type of optical flow tracking point determined by externally provided optical flow tracking information; for example, the optical flow tracking point information comes from a front-end optical flow tracker which can independently perform the extraction of the feature points and the pure optical flow tracking.

The second type of optical flow tracking points are determined by extracting feature points of two adjacent images, namely a previous image frame and a current image frame, and performing optical flow tracking on the basis of the feature points by using a computer system, and are suitable for the condition that no optical flow tracking information is provided outside.

The feature extraction points refer to feature points obtained by a computer system performing feature extraction on a current image frame by using a related feature point extraction algorithm. The feature point extraction algorithm may include: SIFT (Scale Invariant Feature Transform), surf (speedup Robust features) algorithm, fast (features From accessed Segment test) algorithm, and the like, which can extract image Feature points.

In step 12, descriptor matching is performed on each feature point in the feature point set to obtain tracking information of descriptor tracking points;

in the embodiment of the application, secondary tracking is attempted to be performed on the optical flow tracking point by using a descriptor method, and the descriptor method is performed on the corresponding feature point of the optical flow tracking point in the previous image frame at the same time, so that the accuracy of tracking the feature point is higher, and the tracking drift probability is reduced.

In this application, the tracking information describing the child tracking point includes: describing the position of the sub-tracking point in the current image frame and the related tracking information; the related tracking information is used for describing a determination mode of a search range center in the sub-tracking point determination process.

Referring to fig. 2, which is a flow chart illustrating another image feature tracking method according to an exemplary embodiment, the step 12 may include:

in step 121, calculating a descriptor for each feature point in the feature point set to obtain a descriptor of a target feature point;

in the embodiment of the present application, for each feature point in the feature point set, whether the feature point belongs to an optical flow tracking point or a feature extraction point, the computer system calculates a descriptor for each feature point by using an algorithm that can calculate a feature point descriptor, such as a BRIEF algorithm, an orb (organized FAST and Rotated BRIEF) algorithm, a SIFT algorithm, a SURF algorithm, and the like.

It should be noted here that in the process of calculating the descriptor of the feature point, the computer system calculates the descriptor for the position of each feature point in the feature point set. Since the descriptor of the feature point is used to describe the information of the key point and its surrounding pixels, and can eliminate the change of the scale and direction of the image caused by the change of the viewing angle, a good descriptor is usually required to have invariance, robustness and distinguishability. Wherein, the invariance means that the characteristic does not change along with the zooming rotation of the image; robustness refers to insensitivity to noise, illumination, or some other small deformation; differentiable means that descriptors of each feature point are unique and exclusive, and similarity between each feature point and each other is reduced as much as possible. In view of the above characteristics of descriptors, in the process of performing descriptor computation by traversing feature points, not every feature point can compute a corresponding descriptor.

In the embodiment of the present application, a feature point at which a descriptor can be calculated is referred to as a target feature point. The target feature points may belong to optical flow tracking points or feature extraction points.

In addition, in the algorithms mentioned in the embodiments of the present application, it is not limited to implement one function, and for some algorithms such as the above-mentioned SIFT algorithm and SURF algorithm, the computer system may implement two functions of feature point extraction and descriptor calculation simultaneously by using these algorithms.

In step 122, feature point matching is performed based on the descriptor of the target feature point, and the tracking information of the descriptor tracking point in the feature point set is determined.

The descriptor tracking point of the feature point set represents a feature point which can be tracked by a descriptor method based on a target feature point of a previous image frame in the feature point set of the current image frame.

In the process of matching the feature points by the descriptor method, matching needs to be performed based on the feature point descriptor of the previous image frame and the feature point descriptor of the current image frame.

Referring to fig. 3, which is a flow chart of another method for image feature tracking according to an exemplary embodiment, the step 122 may include:

step 1221, for each first target feature point in a previous image frame, determining a search range center of the first target feature point in the current image frame;

the first target feature point is a feature point of a descriptor that can be calculated in a previous image frame. In the process of feature matching by the descriptor method, for a feature point in a previous image frame, in order to match a corresponding point in a current image frame, a search range needs to be determined in the current image frame first, so as to perform feature matching in the search range. Then, the key to determining the search range is to determine the position of the center of the search range.

In the embodiment of the present application, the center of the search range of a first target feature point in the current image frame may be determined in at least one of the following manners:

the method I is suitable for an application scene that the first target feature point has a corresponding target optical flow tracking point in the current image frame.

Referring to fig. 4, which illustrates another method flow diagram for image feature tracking according to an exemplary embodiment, the step 1221 may include:

in step 1201, determining whether the three-dimensional position of the first target feature point has been initialized;

in this embodiment, before determining the search range center of the first target feature point, the computer system may first determine whether the first target feature point has been initialized to a corresponding three-dimensional point. The initialization timing of the three-dimensional point is related to a Visual Odometer (VO) policy in the SLAM system.

In step 1202, if the three-dimensional position of the first target feature point is initialized, determining a projected predicted position of the initialized three-dimensional point in the current image frame;

in an embodiment of the present application, if the three-dimensional position of the first target feature point has been initialized, that is, a three-dimensional point has been initialized, a projected predicted position of the three-dimensional point in the current image frame may be determined.

In step 1203, determining a first distance between the target optical flow tracking point and the projected predicted location;

in an embodiment of the present application, if there are a target optical flow tracking point Pg corresponding to the first target feature point and a projection prediction position Pt in the current image frame at the same time, a distance between the two positions, which may be referred to as a first distance in this embodiment, is calculated and is denoted as D1.

In step 1204, if the first distance is smaller than or equal to a preset first distance threshold, determining the target optical flow tracking point as the center of the search range;

after the first distance D1 is determined, in an embodiment of the present application, the first distance may be compared with a preset first distance threshold, and a search range center of the first target feature point in the current image frame may be determined according to the comparison result.

Assuming that the first distance is a pixel distance, the preset first distance threshold may be a preset number of pixels, such as 5 pixels.

In an embodiment of the present application, if the first distance is smaller than or equal to a preset first distance threshold, the target optical flow tracking point Pg may be determined as a search range center of the first target feature point. Otherwise, the following step 1205 is performed.

In step 1205, if the first distance is greater than the first preset distance threshold, the projected predicted position is determined as the center of the search range.

The above-mentioned steps 1202 to 1205 can be understood as an implementation of step 1221, namely, how to determine the center of the search range of the first target feature point in the current image frame and improve the accuracy of determining the center of the search range in the case that the first target feature point has a corresponding target optical flow tracking point and the three-dimensional position of the first target feature point is initialized.

Referring to fig. 5, which is a flowchart illustrating another method for tracking image features according to an exemplary embodiment, after step 1201, step 1221 may further include:

in step 1206, if the three-dimensional position of the first target feature point is not initialized, determining the target optical flow tracking point as the search range center.

In this embodiment, it can be understood as a second implementation case of step 1221, that is, how to determine the center of the search range of the first target feature point in the current image frame in a case where the first target feature point has a corresponding target optical flow tracking point but the three-dimensional position is not initialized.

In the embodiment of the application, if the first target feature point has a corresponding target optical flow tracking point in the current image frame, in the process of performing feature matching by using the descriptor method, the target optical flow tracking point can be used as prior information to determine the center of a search range, so that the purpose of quickly determining the center of the search range is achieved, and the efficiency of performing feature matching by using the descriptor method is improved.

The method II is suitable for an application scene that the first target feature point is not matched with the target optical flow tracking point in the current image frame.

Referring to fig. 6, which is a flow chart illustrating another method for image feature tracking according to an exemplary embodiment, the step 1221 may include:

in step 1001, determining whether the three-dimensional position of the first target feature point has been initialized;

this step 1001 is similar to step 1201 described above and will not be described further here.

In step 1002, if the three-dimensional position of the first target feature point is initialized, determining a projection prediction position of the initialized three-dimensional point in the current image frame;

this step 1002 is similar to the above step 1202, and is referred to each other, and is not described herein again.

In step 1003, the projected predicted position is determined as the search range center.

The above embodiment may be understood as the third implementation case of step 1221, that is, in the case that there is no corresponding target optical flow tracking point at the first target feature point, but its three-dimensional position is initialized, the projected predicted position of the three-dimensional position in the current image frame may be determined as the search range center of the first target feature point.

For the case that the determination result of the step 1001 is no, refer to another flowchart of a method for tracking image features shown in fig. 7 according to an exemplary embodiment, and after the step 1001, the step 1221 may further include:

in step 1004, if the three-dimensional position of the first target feature point is not initialized, determining a homography matrix representing a mapping relationship between the current image frame and the previous image frame;

the homography matrix describes a mapping relation from a two-dimensional point set on one image to a two-dimensional point set on the other image. The condition for this mapping to hold is: two-dimensional points in the image all fall on a plane corresponding to three-dimensional points in the real world; however, in practice, the object photographed by the camera is not a large plane, but a real scene including many three-dimensional objects, and a certain error is generated when mapping with the homography matrix. However, this error is relatively small for two adjacent images in an image sequence such as a continuous video, and therefore, the error can be used to make an estimate of the potential location of the matching point, i.e., to predict the center of the search range of the first target feature point in the current image frame.

Regarding how to determine the homography matrix, in an embodiment of the present application, if the number of optical-flow tracking points in the feature point set exceeds a predetermined number threshold, the homography matrix may be determined according to pixel coordinates of optical-flow tracking point pairs in two adjacent frames of images.

Referring to FIG. 8, another method flow diagram of image feature tracking is shown, according to an exemplary embodiment, where the above step 1004 may include:

in step 1014, determining the number of optical flow tracking points in the set of feature points;

in step 1024, if the number of the optical flow tracking points is greater than a preset number threshold, determining pixel coordinate information of each optical flow matching point pair according to the optical flow tracking information of the optical flow tracking points;

the preset number threshold is the minimum number of matching point pairs required for ensuring the validity of the homography matrix; if the number of the optical flow tracking points is less than or equal to the threshold value, the calculated homography matrix is likely not to accurately reflect the mapping relationship between the two frames of images.

The optical flow matching point pairs are feature point pairs tracked in the previous image frame and the current image frame by an optical flow tracking method.

In step 1034, determining the homography matrix according to the pixel coordinate information of each optical flow matching point pair;

namely, feature point pairs matched in two adjacent frame images by adopting an optical flow tracking method are determined according to optical flow tracking point information in a current image frame, and then a homography matrix is calculated according to pixel coordinates of all the feature point pairs.

In another embodiment of the present application, if the number of optical flow tracking points in the feature point set does not exceed the preset number threshold, for example, 15 pixel points; assuming that the number of optical-flow tracking points in the feature point set is 13, accordingly, the pixel coordinate information of 13 optical-flow matching point pairs can be determined in the previous image frame and the current image frame, in this case, the homography matrix determined only according to the pixel coordinates of the 13 optical-flow matching point pairs is often unreliable, i.e., the homography matrix determined according to the optical-flow tracking point information is often unreliable. At the moment, the terminal device such as a smart phone can fit a homography matrix by using data sensed by other sensors sensing the pose of the device, such as Inertial Measurement Units (IMUs) and other sensors.

In step 1005, a search range center of the first target feature point is determined according to the homography matrix.

The foregoing embodiment may be understood as the fourth implementation case of the foregoing step 1221, that is, the case where the first target feature point is not matched to the corresponding target optical flow tracking point in the current image frame, and the corresponding three-dimensional position is not initialized, the search range center of the first target feature point may be determined by using the above homography matrix.

In step 1222, descriptors of the second target feature points in a target search range are determined, wherein the target search range is a range determined based on the center of the search range;

in the embodiment of the present application, a feature point of the descriptor, referred to as a second target feature point, may be calculated in the current image frame.

In the case that the search range center of the first target feature point is determined, the target search range of the first target feature point may be determined in the current image frame according to a preset strategy, for example, with the search range center as a circle center and a preset number of pixels as a radius, so as to subsequently perform feature matching on the descriptor of the first target feature point and the descriptor of each second target feature point in the target search range.

In step 1223, feature point matching is performed according to the descriptor of the first target feature point and the descriptors of the second target feature points, so as to obtain tracking information of the descriptor tracking point corresponding to the first target feature point in the current image frame.

In the process of carrying out feature matching by the descriptor method, the matching of features is carried out aiming at the feature point descriptor. The specific matching process is as follows: calculating the distance between the descriptor of the first target feature point and the descriptors of the second target feature points in the target search range to obtain a plurality of distance values;

the feature descriptor is usually a vector, and the distance between two feature descriptors can reflect the similarity of two feature points, that is, whether the two feature points represent the same object. Different distance measures may be selected depending on the description subtype. If the descriptor is a floating point type descriptor, the Euclidean distance of the descriptor can be used; for binary descriptors (BRIEF), their Hamming distance can be used, where the Hamming distance between two different binary strings refers to the number of different bits of the two binary strings.

Comparing the minimum distance value with a preset distance threshold value; if the minimum distance value is smaller than the preset distance threshold, determining a second target characteristic point corresponding to the minimum distance value as a descriptor tracking point corresponding to the first target characteristic point; otherwise, determining that the descriptor tracking point corresponding to the first target feature point is not matched in the current image frame.

That is, the descriptor matching result of the first target feature point includes two cases: matching corresponding feature points in second target feature points of a current image frame; and secondly, not matching corresponding feature points in the second target feature points of the current image frame.

Illustratively, see FIG. 9 for a schematic illustration of image feature tracking according to an exemplary embodiment.

For a feature point in the previous image frame, the mark is P1i, wherein 1i represents the serial number of the feature point in the t frame image, and i is a positive integer from 1 to n; for feature point P1i, a corresponding descriptor may be computed.

Firstly, aiming at a characteristic point P1i in a previous image frame, determining a search range center in a current image frame, namely a t +1 th image frame, and marking the search range center as P2 j; then, determining a target search range by using the search range center P2j, for example, a circle area determined by using the feature point P2j as a center and using a preset number of pixels, for example, 10 pixels, as a radius; and matching the descriptors of P1i with the descriptors of the second target feature points in the target search range, for example, the descriptors of 8 second target feature points, one by one to obtain descriptor matching results.

And by analogy, determining descriptor tracking points included in the feature point set of the current image frame in the manner described above, and determining tracking information of each descriptor tracking point. Wherein, the tracking information describing the sub-tracking point may include: the position of the descriptor tracking point, the related tracking information and the like, wherein the related tracking information is used for indicating a method for determining the center of the search range in the tracking process by adopting the descriptor method; the method can comprise the following steps: optical flow method, three-dimensional projection method, homography matrix method.

So far, the determination of the tracking information of the optical flow tracking points and the description sub tracking points included in the feature point set of the current image frame is finished.

In the process of determining the descriptor tracking point, the center of the search range can be determined quickly and accurately based on the optical flow tracking information as prior information, the search range of the descriptor tracking can be reduced, and the tracking effect is effectively improved.

In step 13, determining a feature tracking result of the current image frame according to the optical flow tracking point and the tracking information of the descriptor tracking point.

In an embodiment of the present application, the tracked feature points in the feature point set of the current image frame are determined by the computer system using an optical flow method and a descriptor method, respectively.

Referring to fig. 10, which is a flowchart illustrating another image feature tracking method according to an exemplary embodiment, the step 13 may include:

in step 131, determining an initial feature tracking result of the current image frame according to the optical flow tracking point and the tracking information of the descriptor tracking point;

in an embodiment of the present application, for a first target feature point in a previous image frame, an optical-flow tracking point and a descriptor tracking point may be respectively matched in a current image frame. In this case, one of the two tracking points may be selected as the target tracking point of the first target feature point.

Referring to fig. 11-1, which is a flow chart illustrating another image feature tracking method according to an exemplary embodiment, the above step 131 may include:

in step 1311, for the same first target feature point, a second distance between the optical flow tracking point and the descriptor tracking point is determined;

referring to fig. 11-2, a schematic view of another image feature tracking method scene is shown according to an exemplary embodiment, assuming that a first target feature point P1i in a previous image frame is matched to a corresponding optical flow tracking point P2g in a current image frame; and also to the corresponding descriptor tracking point P2 m. In the embodiment of the present application, the distance between the optical flow tracking point P2g and the descriptor tracking point P2m may be calculated, and may be referred to as a second distance D2.

In step 1312, determining whether the second distance is greater than a preset second distance threshold;

in this embodiment of the application, the second distance may be compared with a preset second distance threshold, and the target tracking point of the first target feature point may be determined based on a comparison result.

In step 1313, if the second distance is smaller than or equal to the preset second distance threshold, determine the optical flow tracking point as a target tracking point of the first target feature point;

in step 1314, if the second distance is greater than the preset second distance threshold, the descriptor tracking point is determined as the target tracking point of the first target feature point.

Assuming that the second distance D2 belongs to a pixel distance, the predetermined second distance threshold is a predetermined number of pixels, such as 3 pixels; if D2 is less than or equal to 3, determining the optical flow tracking point P2g as the target tracking point of the first target feature point P1i in the current image frame; the precision of the tracking point position information determined by the optical flow tracking method can be in a sub-pixel level, and the precision of the tracking point position information determined by the description sub-method is in a pixel level; therefore, when the positions of two kinds of tracking points for the same first target feature point in the current image frame are close to each other, the optical flow tracking point may be preferentially determined as the target tracking point to improve the accuracy of the position of the tracking point of the first target feature point in the current image frame.

On the contrary, if the second distance D2 is greater than 3, the above-described descriptor tracking point P2m is determined as the target tracking point of the first target feature point P1i in the current image frame. In the embodiment of the application, when the distances between the two tracking points are relatively long, that is, when the position deviation of the feature points tracked by adopting the optical flow method and the descriptor method is relatively large, the tracking points determined by adopting the descriptor method have higher precision, so that the descriptor tracking points are determined as the target tracking points of the first target feature points, and possible drift of the optical flow tracking points can be avoided.

And analogizing in turn, obtaining matching point pair information between the current image frame and the previous image frame, and further determining an initial feature tracking result of the current image frame, wherein the initial feature tracking result represents which feature points in the feature point set of the current image frame belong to tracked points, namely tracking points.

In step 132, feature points of the initial feature tracking result are removed according to the related tracking information of each tracking point, so as to obtain a target feature tracking result.

In the embodiment of the present application, the initial feature tracking result generally includes many tracking points, and for example, it is assumed that the step 11 determines that the feature point set of the current image frame includes 100 feature points, and 80 tracking points are determined through the step 131. Among the 80 tracking points, a tracking point with a short distance may be included.

For the above situation, feature points of the initial feature tracking result may be eliminated to obtain effective tracking points of the current image frame, so as to reduce the calculation amount of subsequent image feature tracking.

In an embodiment of the present application, the computer system may remove a tracking point closer to a current image frame by using a correlation method, such as a Brute-force matching (Brute-force matching), a cross matching, a K-nearest neighbor matching method, and the like, to obtain an effective tracking point of the current image frame, so as to reduce a calculation amount in a subsequent feature tracking process.

In another embodiment of the present application, the computer system may further determine the target tracking point of the current image frame by using a multiple weight assignment method based on the initial feature tracking result.

Referring to FIG. 12, which is a flow chart illustrating another method of image feature tracking according to an exemplary embodiment, the step 132 may include:

in step 1321, determining a score of each tracking point in the initial feature tracking result according to the relevant tracking information of each tracking point and a preset weight assigning rule;

in the embodiment of the application, for each target tracking point in the current image frame, different weights may be given according to different related tracking information. Wherein the related tracking information may include at least one of:

the tracked mode comprises the following steps: an optical flow tracking mode and a descriptor tracking mode;

the search range determining mode comprises the following steps: optical flow tracking, three-dimensional point projection, homography matrix;

the tracked number of times represents the number of times tracked in the sequence of leading image frames.

In this application, the preset weight assignment rule may include: the corresponding relationship between the related tracking information and the preset assigned value is, for example, as shown in the following table one:

watch 1

Related tracking information	Assigning a value
		Descriptor-optic flow prior	5 points of
Descriptor-projection	5 points of
		Descriptor-homography matrix	3 points of
Optical flow method	1 minute (1)

As shown in table one, if the tracking information related to a tracking point is a descriptor-optical flow prior, it indicates that the tracking point is a descriptor tracking point, and the search range center determined by the optical flow tracking information may correspond to step 1204 or step 1206 above.

Similarly, if the tracking information related to a tracking point is descriptor-projection, it indicates that the tracking point is a descriptor tracking point and the search range center determined by predicting the position by using three-dimensional point projection may correspond to step 1205 or step 1003.

If the related tracking information of a tracking point is a descriptor-homography matrix, it indicates that the tracking point is a descriptor tracking point and the search range center determined by the homography matrix may correspond to step 1005 above.

If the related tracking information of one tracking point is an optical flow method, the tracking point is represented as an optical flow tracking point.

For simple illustration, it is assumed that the initial tracking result includes 5 tracking points, which are respectively expressed as: the P21, the P22, the P23, the P24 and the P25 are assigned according to the preset weight assignment rule according to the relevant tracking information, and the obtained scores are respectively as follows: 5.3, 1, 5, 3; the correspondence between the tracking point index and the tracking point score can be as shown in table two:

watch two

In an actual application scenario, the number of the tracking points in the initial tracking result may be many, for example, 80, and the computer system may determine the score of each tracking point according to the preset weighting rule.

In step 1322, the tracking points are sorted in the order from high to low according to the score of the tracking point, and a sorted list of tracking points is obtained, where the sorted list includes: tracking the corresponding relation between the point index information and the score;

as in the above example, the ordered list of tracking points obtained may be as shown in table three:

watch III

Tracking point indexing	Score value
		P21	5 points of
P24	5 points of
		P22	3 points of
P25	3 points of
		P23	1 minute (1)

In another embodiment of the present application, if at least two tracking points obtain the same score value by using the preset weighting rule, as shown in table three, P21 and P24, and P22 and P25, in order to effectively distinguish the sorting information thereof, the score value of each tracking point may be updated by using the number of times that each tracking point is tracked in the preceding image frame, and the plurality of tracking points are sorted based on the updated score value of the tracking point.

For example, taking the above P21 and P24 as examples, assuming that P21 is tracked 5 times in a preceding preset number, such as 5 image frames, the score of P21 may be updated to 5.5; p24 was tracked 3 times in a pre-set number of previous frames, such as 5 image frames, and the score of P24 could be updated to 5.3. And adjusting the arrangement sequence of the P21 and the P24 in the ordered list by using the updated score. Likewise, the arrangement order of P22 and P25 in the tracking point ordered list can be effectively distinguished according to the method.

In step 1323, the tracking points with close distances in the current image frame are removed according to the tracking point ranking list, and the target feature tracking result is obtained.

In the embodiment of the application, the computer system may remove tracking points with similar distances from the current image frame according to the sorting information of the tracking point indexes in the tracking point sorting list, so as to obtain the target feature tracking result.

Still as in the above example, it is assumed that the table three is fine-tuned according to the tracking times, and the updated ordered list of tracking points is obtained as shown in table four:

watch four

Then the computer system may sequentially remove tracking points with similar distances from the current image frame by using a mask with a preset size according to the sorted index information.

Referring to fig. 13, still as an example, according to another application scenario diagram of the image feature tracking method shown in an exemplary embodiment, in the current image frame 100, according to the ordered list of tracking points shown in table four above, the computer system first uses a mask with a radius r and with P21 as a center to determine whether there are other tracking points within the mask coverage, and as shown in the figure, a tracking point P22 occurs within the mask coverage of a tracking point P21; the explanation P22 is closer to P21, so the tracking point P22 can be eliminated. Then, with the tracking point P24 of the rank 2 as the center, the mask is used to perform feature point elimination, and so on, and the feature point elimination is performed in the order from high to low according to the index rank, so as to obtain the target tracking result of the current image frame, as shown in the picture 200.

According to the method and the device, after the initial feature tracking result of the current image frame is determined by combining the optical flow tracking method and the descriptor tracking method, the feature points which are relatively close to each other can be removed according to the feature point sorting list, so that a reasonable number of tracking feature points are extracted from the current image frame, the extraction precision of image features can be improved, the calculated amount of subsequent image feature tracking can be reduced, and the feature tracking efficiency is improved.

In summary, by using the image feature tracking method provided by the embodiment of the present application, in the process of performing image feature tracking on the current image frame, the feature point set of the current image frame may be determined based on the optical flow tracking information; and finally, determining an image feature tracking result of the current image frame according to tracking information respectively corresponding to the optical flow tracking point and the descriptor tracking point. In the process of determining the image feature tracking result, the method and the device effectively integrate the advantages of optical flow method feature tracking and descriptor method feature tracking, and for image sequences shot in complex scenes such as rapid or violent motion scenes, scenes with insufficient light rays and the like, the device with limited computing power can also accurately and stably track feature points representing the same object, so that the visual tracking effect is improved, and the visual tracking power consumption of the device is reduced.

Based on the image feature tracking method described in any of the above embodiments, the present application also provides a scene creation method, which can be applied to a visual SLAM system of a terminal device.

Referring to fig. 14, a flowchart of a scene creation method is shown according to an example embodiment, which may include:

in step 21, a sequence of image frames is acquired;

in step 22, performing image feature point tracking on each current image frame in the image frame sequence by using the image feature tracking method to obtain an image feature tracking result;

in this embodiment, the SLAM system may perform feature tracking on the current image frame acquired in real time by using any of the above image feature tracking method embodiments, to obtain an image feature tracking result.

In step 23, estimating pose information and three-dimensional point cloud information of the terminal device according to an image feature tracking result of the image frame sequence;

in an embodiment of the present application, a method of a VIO (Visual-Inertial odometer) in an SLAM system may be adopted, and real-time pose information and three-dimensional point cloud information of a terminal device are estimated through Bundle Adjustment (Bundle Adjustment) according to an image feature tracking result of the image frame sequence and motion amount or motion change amount data sensed by Inertial components mounted on the terminal device, such as an IMU, a magnetometer, and other sensors.

And in step 24, creating an augmented reality scene according to the pose information and the three-dimensional point cloud information of the terminal equipment.

One core engine of AR applications is the three-dimensional positioning capability of terminal devices such as smart phones. Only if the computer system of the terminal device accurately estimates the six-degree-of-freedom information, namely rotation and translation information, of the smart phone according to the image frame sequence acquired in real time and accurately estimates the position and the pose of the smart phone, the accurate placement position of the virtual object in the real scene can be ensured, and therefore the AR experience effect of the terminal device is enhanced.

In the AR effect, the placement of virtual objects mainly depends on the accuracy of the output camera pose and the accuracy of the three-dimensional point position of the visual SLAM system. The camera pose is determined based on the image feature tracking result of adjacent image frames, namely in an SLAM system, feature points are continuously tracked through an image sequence shot in real time, and the camera pose and the position of a three-dimensional point are estimated through a Bundled Adjustment (BA) method in combination with data measured by an inertial sensor carried by a target object. In general, the better the feature point tracking, the more accurate the estimated three-dimensional point position and camera pose are.

By adopting the image feature tracking method provided by the embodiment of the application, the image frame sequence shot under the condition that the user carries the terminal equipment to move rapidly or the ambient light is not good also has higher tracking efficiency and precision, so the vision SLAM system can output the camera pose with higher precision, the pose output is more stable, better AR effect can be realized, and the AR experience of the terminal equipment is improved.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the data processing apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to part of the description of the method embodiment.

While, for purposes of simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, the present disclosure is not limited by the order of acts or acts described, as some steps may, in accordance with the present disclosure, occur in other orders and concurrently. It will be understood by those skilled in the art that the order of description of the steps in the specific implementation of the above method is not meant to imply a strict order of execution or any limitation on the implementation, and the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

Further, those skilled in the art should also appreciate that the embodiments described in the specification are exemplary embodiments and that acts and modules referred to are not necessarily required by the disclosure.

Corresponding to the embodiment of the application function implementation method, the disclosure also provides an embodiment of an application function implementation device and a corresponding terminal.

Referring to fig. 15, a block diagram of an image feature tracking device according to an exemplary embodiment may include:

a feature point determining module 31 configured to determine, for the acquired image frame sequence, a feature point set of each current image frame, where the feature point set includes: optical flow tracking points;

a tracking information determination module 32 configured to perform descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points;

a feature tracking module 33 configured to determine a feature tracking result of the current image frame according to the tracking information of the optical flow tracking points and the descriptor tracking points.

Referring to fig. 16, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, and on the basis of the embodiment of the apparatus shown in fig. 15, the feature point determining module 31 may include:

an optical-flow tracking sub-module 311 configured to acquire optical-flow tracking information of a current image frame and determine optical-flow tracking points in the current image frame based on the optical-flow tracking information;

a feature point extraction sub-module 312 configured to perform feature point extraction on the current image frame to obtain feature extraction points in the current image frame;

a feature point set determination submodule 313 configured to determine a feature point set of the current image frame from the optical flow tracking points and the feature extraction points;

Referring to fig. 17, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, on the basis of the embodiment of the apparatus shown in fig. 15 or fig. 16, the tracking information determining module 32 may include:

a descriptor determining submodule 321 configured to calculate a descriptor for each feature point in the feature point set, and obtain a descriptor of a target feature point;

and a descriptor tracking sub-module 322 configured to perform feature point matching based on the descriptor of the target feature point, and determine tracking information of the descriptor tracking point.

Referring to fig. 18, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, and based on the embodiment of the apparatus shown in fig. 17, the descriptor tracking sub-module 322 may include:

a search center determining unit 3221 configured to determine, for each first target feature point in a previous image frame, a search range center of the first target feature point in the current image frame, wherein the first target feature point is a feature point having a descriptor in the previous image frame;

a descriptor determining unit 3222 configured to determine a descriptor of each second target feature point within a target search range, the target search range being a range determined based on the search range center;

a matching unit 3223, configured to perform feature point matching according to the descriptor of the first target feature point and the descriptors of the respective second target feature points, and obtain tracking information of a descriptor tracking point corresponding to the first target feature point in the current image frame.

In an embodiment of the present application, the method further includes, for a case that the first target feature point has a corresponding target optical flow tracking point in the current image frame;

referring to fig. 19, which is a block diagram of another image feature tracking apparatus according to an exemplary embodiment, on the basis of the apparatus embodiment shown in fig. 18, the search center determining unit 3221 may include:

a position initialization determination subunit 301 configured to determine whether the three-dimensional position of the first target feature point has been initialized;

a projection position determining subunit 302, configured to determine, in a case where the three-dimensional position of the first target feature point has been initialized, a projection prediction position of the initialized three-dimensional point in the current image frame;

a first distance determining subunit 303 configured to determine a first distance between the target optical flow tracking point and the projected predicted position;

a first search center determining subunit 304, configured to determine, in a case where the first distance is equal to or less than a preset first distance threshold, the target optical flow tracking point as the search range center;

a second search center determining subunit 305 configured to determine the projected predicted position as the search range center if the first distance is greater than the first preset distance threshold.

In another apparatus embodiment of the present application, the search center determining unit may be configured to determine the target optical flow tracking point as the search range center in a case where a three-dimensional position of the first target feature point is not initialized.

In another apparatus embodiment of the present application, for a case where the first target feature point does not have a corresponding target optical flow tracking point in the current image frame;

referring to fig. 20, which is a block diagram of another image feature tracking apparatus according to an exemplary embodiment, on the basis of the apparatus embodiment shown in fig. 18, the search center determining unit 3221 may include:

a third search center determining subunit 306 configured to determine the projected predicted position as the search range center.

Referring to fig. 21, which is a block diagram of another image feature tracking apparatus according to an exemplary embodiment, on the basis of the apparatus embodiment shown in fig. 18, the search center determining unit 3221 may include:

a homography matrix determination subunit 307 configured to determine a homography matrix representing a mapping relationship between the current image frame and the previous image frame in a case where the three-dimensional position of the first target feature point is not initialized;

a fourth search center determining subunit 308 configured to determine a search range center of the first target feature point according to the homography matrix.

Referring to fig. 22, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, and on the basis of the embodiment of the apparatus shown in fig. 21, the homography matrix determination subunit 307 may include:

a number determination unit 3071 configured to determine the number of the optical-flow tracking points in the feature point set;

a pixel coordinate determination unit 3072 configured to determine pixel coordinate information of each optical flow matching point pair according to optical flow tracking information of the optical flow tracking points if the number of the optical flow tracking points is greater than a preset number threshold;

a matrix determination unit 3073 configured to determine the homography matrix from the pixel coordinate information of each of the optical-flow matching point pairs;

In an embodiment of the present application, the tracking information describing the sub-tracking point obtained by the tracking information determining module 32 may include: the method comprises the following steps of (1) tracking point positions and related tracking information, wherein the related tracking information is used for representing a determination mode of a search range center in the process of determining descriptor tracking points;

referring to fig. 23, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, and based on the embodiment of the apparatus shown in fig. 15, the feature tracking module 33 may include:

an initial tracking result determining sub-module 331 configured to determine an initial feature tracking result of the current image frame according to the optical flow tracking points and tracking information of the descriptor tracking points;

and the target tracking result determining submodule 332 is configured to perform feature point elimination on the initial feature tracking result according to the related tracking information of each tracking point to obtain a target feature tracking result.

In an embodiment of the present application, if a first target feature point in a previous image frame is respectively matched to an optical flow tracking point and a descriptor tracking point in the current image frame;

referring to fig. 24, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, on the basis of the embodiment of the apparatus shown in fig. 23, the initial tracking result determining sub-module 331 may include:

a second distance determining unit 3311 configured to determine, for the same first target feature point, a second distance between the optical flow tracking point and the descriptor tracking point;

a first target determining unit 3312 configured to determine the optical-flow tracking point as a target tracking point of the first target feature point in a case where the second distance is equal to or less than a preset second distance threshold;

a second target determining unit 3313 configured to determine the descriptor tracking point as a target tracking point of the first target feature point if the second distance is greater than the preset second distance threshold.

Referring to fig. 25, a block diagram of another image feature tracking apparatus according to an exemplary embodiment is shown, on the basis of the embodiment of the apparatus shown in fig. 23 or fig. 24, the target tracking result determining sub-module 332 may include:

a score determining unit 3321 configured to determine a score of each tracking point in the initial feature tracking result according to relevant tracking information of each tracking point and a preset weight assigning rule;

a sorting unit 3322 configured to sort the tracking points in order of the tracking point scores from high to low to obtain a tracking point sorted list, where the tracking point sorted list includes: the corresponding relation between the tracking point index and the tracking point score;

a tracking result determining unit 3323 configured to remove the tracking points with close distances in the current image frame according to the tracking point ranking list, and obtain the target feature tracking result.

In another embodiment of the present invention, the sorting unit 3322 may include:

The present application also provides a scene creation apparatus, referring to a block diagram of the scene creation apparatus shown in fig. 26 according to an exemplary embodiment, the scene creation apparatus may include:

an image acquisition module 41 configured to acquire a sequence of image frames;

a feature tracking module 42 configured to perform image feature point tracking on each current image frame in the image frame sequence by using any one of the image feature tracking apparatuses described above, so as to obtain an image feature tracking result;

an information estimation module 43 configured to estimate pose information and three-dimensional point cloud information of a terminal device according to an image feature tracking result of the image frame sequence;

and a scene creating module 44 configured to create an augmented reality scene according to the pose information and the three-dimensional point cloud information of the terminal device.

For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.

In some embodiments, functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Corresponding to the image feature tracking method described above, an embodiment of the present application further provides a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application, shown in fig. 27. Similarly, corresponding to the above-mentioned scene creating method, an embodiment of the present application further provides a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application, shown in fig. 28. Referring to fig. 27 or fig. 28, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program to form the intelligent driving control device on the logic level. Of course, besides the software implementation, the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.

As will be appreciated by one skilled in the art, one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present specification further provide a computer-readable storage medium, on which a computer program may be stored, where the computer program, when executed by a processor, implements the steps of the image feature tracking method or the scene creation method provided in any one of fig. 1 to 14 in the present specification.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computers suitable for executing computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Moreover, a computer may be embedded in another device, e.g., a mobile telephone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., an internal hard disk or a removable disk), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above description is only for the purpose of illustrating the preferred embodiments of the one or more embodiments of the present disclosure, and is not intended to limit the scope of the one or more embodiments of the present disclosure, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the one or more embodiments of the present disclosure should be included in the scope of the one or more embodiments of the present disclosure.

Claims

1. An image feature tracking method, comprising:

2. The method of claim 1, wherein determining the set of feature points for each current image frame comprises:

3. The method according to claim 1 or 2, wherein the performing descriptor matching on each feature point in the feature point set to obtain tracking information of descriptor tracking points comprises:

4. A method for creating a scene, comprising:

acquiring an image frame sequence;

tracking image feature points of each current image frame in the image frame sequence by using the method of any one of claims 1 to 3 to obtain an image feature tracking result;

5. An image feature tracking apparatus, comprising:

6. The apparatus of claim 5, wherein the feature point determining module comprises:

7. The apparatus of claim 5 or 6, wherein the tracking information determining module comprises:

8. A scene creation apparatus, characterized by comprising:

an image acquisition module configured to acquire a sequence of image frames;

a feature tracking module configured to perform image feature point tracking on each current image frame in the image frame sequence by using the apparatus according to any one of claims 5 to 7 to obtain an image feature tracking result;

9. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-4.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 4 when executing the program.