CN111179309A - Tracking method and device - Google Patents

Tracking method and device Download PDF

Info

Publication number
CN111179309A
CN111179309A CN201911319416.9A CN201911319416A CN111179309A CN 111179309 A CN111179309 A CN 111179309A CN 201911319416 A CN201911319416 A CN 201911319416A CN 111179309 A CN111179309 A CN 111179309A
Authority
CN
China
Prior art keywords
frame image
feature points
initial
image
current frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911319416.9A
Other languages
Chinese (zh)
Other versions
CN111179309B (en
Inventor
周毅
高江涛
陈建冲
杨旭
孙炼杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lenovo Beijing Ltd
Original Assignee
Lenovo Beijing Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lenovo Beijing Ltd filed Critical Lenovo Beijing Ltd
Priority to CN201911319416.9A priority Critical patent/CN111179309B/en
Publication of CN111179309A publication Critical patent/CN111179309A/en
Application granted granted Critical
Publication of CN111179309B publication Critical patent/CN111179309B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The method comprises the steps of obtaining feature points of a current frame image of a target object, updating the feature points of the current frame image according to initial information of the target object when the number of the feature points of the current frame image is smaller than a number threshold value, wherein the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image, and continuously tracking the target object according to the updated feature points of the current frame image.

Description

Tracking method and device
Technical Field
The present application relates to the field of digital image processing technology, and relates to, but is not limited to, a tracking method and apparatus.
Background
With the rapid development of multimedia technology and the continuous improvement of computer performance, dynamic image processing technology is increasingly favored by people, obtains fruitful results, and is widely applied to the fields of traffic management, military target tracking, biomedicine and the like.
In the identification and tracking of moving objects, the original method is to extract and match characteristic points of each frame of shot image, and the defects of large calculation amount and low processing speed exist. In order to avoid the above drawbacks, an existing solution is to perform recognition and tracking based on an optical flow method, specifically, to extract feature points of a moving target, and calculate a pose based on a feature point extraction and matching method.
The prior scheme has the following defects: in the existing identification and tracking based on the optical flow method, the feature points are only gradually reduced until the pose cannot be calculated, and the accuracy of the calculated pose is lower and lower in the process of reducing the feature points. Especially, when the camera moves from one direction to another direction, the contents of two continuous frame images are completely different, the feature points for identification and tracking are completely lost, the pose cannot be calculated any more, so that the identification and tracking are failed, or the error of the calculated pose causes the identification and tracking error.
Disclosure of Invention
In view of the above, embodiments of the present application provide a tracking method and apparatus to solve the problems in the prior art.
The technical scheme of the embodiment of the application is realized as follows:
in a first aspect, an embodiment of the present application provides a tracking method, including:
acquiring feature points of a current frame image of a target object;
when the number of the feature points of the current frame image is smaller than a number threshold value, updating the feature points of the current frame image according to initial information of the target object, wherein the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image;
and continuously tracking the target object according to the updated feature points of the current frame image.
In a second aspect, an embodiment of the present application provides a tracking apparatus, including:
the acquisition module is used for acquiring the characteristic points of the current frame image of the target object;
the updating module is used for updating the feature points of the current frame image according to initial information of the target object when the number of the feature points of the current frame image is smaller than a number threshold, wherein the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image;
and the tracking module is used for continuously tracking the target object according to the updated feature points of the current frame image.
In a third aspect, an embodiment of the present application provides a tracking device, including:
a memory for storing executable instructions;
and the processor is used for realizing the method provided by the embodiment of the application when executing the executable instructions stored in the memory.
In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for causing a processor to implement a method provided by an embodiment of the present application when the processor executes the executable instructions.
In the tracking method provided by the embodiment of the application, a tracking device obtains feature points of a current frame image of a target object, when the number of the feature points of the current frame image is smaller than a number threshold, the feature points of the current frame image are updated according to initial information of the target object, the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image, and finally the target object is continuously tracked according to the updated feature points of the current frame image; therefore, in the process of tracking the target object, when the feature points of the current frame image are reduced to the point that the current pose cannot be accurately calculated, the reduced feature points of the current frame image can be supplemented according to the initial information, so that the number of the feature points of the current frame image is more than the number threshold, the accuracy of the current pose calculated according to the feature points of the current frame image is ensured, and the success rate of tracking the target object is improved.
Drawings
Fig. 1A is a schematic diagram of a network architecture of a tracking method according to an embodiment of the present application;
fig. 1B is a schematic diagram of another network architecture of the tracking method according to the embodiment of the present application;
fig. 2 is a schematic structural diagram of a tracking device provided in an embodiment of the present application;
fig. 3 is a schematic flowchart of an implementation of the tracking method according to the embodiment of the present application;
fig. 4 is a schematic flowchart of another implementation of the tracking method according to the embodiment of the present application;
FIG. 5 is a schematic flowchart of another implementation of the tracking method according to an embodiment of the present disclosure;
fig. 6 is a schematic flowchart of another implementation flow of the tracking method according to the embodiment of the present application.
Detailed Description
In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.
In the following description, references to the terms "first \ second \ third" are only to distinguish similar objects and do not denote a particular order, but rather the terms "first \ second \ third" are used to interchange specific orders or sequences, where appropriate, so as to enable the embodiments of the application described herein to be practiced in other than the order shown or described herein.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.
Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be used for the following explanation.
1) The optical flow is due to movement of the foreground objects themselves in the scene, movement of the camera, or both.
2) The principle of tracking a moving object by an optical flow method is as follows: each pixel in the image is first given a velocity vector (optical flow, including magnitude and direction), thus forming an optical-flow field. If no moving object exists in the image, the optical flow field is continuous and uniform; if the moving object exists in the image, the optical flow of the moving object is different from that of the image, and the optical flow field is not continuous and uniform any more, so that the moving object and the position thereof can be detected, and the tracking of the moving object is realized.
3) The intrinsic parameters may also be referred to as intrinsic parameters. The intrinsic parameters refer to parameters related to the characteristics of the acquisition device itself, such as the focal length, pixel size, and the like of the acquisition device. Specifically, the internal parameters of the acquisition device may include: 1/dx, 1/dy, u0, v0, f, where dx and dy represent how many length units a pixel in the x direction and the y direction occupies, respectively, i.e., the size of the actual physical value represented by a pixel, and dx and dy are key to implementing the conversion between the image physical coordinate system and the pixel coordinate system. u0, v0 indicates the number of horizontal and vertical pixels of the phase difference between the center pixel coordinate of the image and the origin pixel coordinate of the image, and f is the focal length. In some embodiments, the intrinsic parameter information may also include distortion parameters, which in turn further include radial distortion coefficients and tangential distortion coefficients. Radial distortion occurs during the transformation of the camera coordinate system to the image physical coordinate system. Tangential distortion occurs during the camera fabrication process due to the non-parallel plane of the photosensing elements and the lens.
4) The extrinsic parameters may also be referred to as extrinsic parameters. The external parameters may refer to pose information of the acquisition device, and the external parameters can determine the position and orientation of the acquisition device in a certain three-dimensional space. The external parameters may include the position, rotation angle, etc. of the acquisition device. In particular, the external parameters of the acquisition device may include three-axis rotation parameter information and three-axis translation parameter information. The size of the matrix R of the rotation parameter information of the three axes is 3 × 3, and the 3 × 4 matrix formed by combining the translation parameters T (Tx, Ty, Tz) of the three axes and R, T is the external parameter of the acquisition device.
5) Point clouds, a point data set of the product appearance surface obtained by a measuring instrument in reverse engineering is also called point clouds, the number of points obtained by using a three-dimensional coordinate measuring machine is small, the distance between the points is large, and the point clouds are called sparse point clouds; point clouds obtained using, for example, three-dimensional laser scanners or photographic scanners are relatively large and dense, called dense point clouds. The point cloud data, the scanning data are recorded in the form of points, and each point comprises three-dimensional coordinates.
For better understanding of the embodiments of the present application, the following method and the existing disadvantages in the related art will be described first.
In the related art, one method for identifying and tracking a three-dimensional object is to shoot each frame of image of the moving object, extract feature points for each frame of image to determine the moving object, wherein the extracted feature points have large data quantity, low identification speed and high requirement on processing equipment, and the tracking cost is increased.
In the related art, another method for identifying and tracking the three-dimensional object is to track the feature points based on an optical flow method after calculating the initial pose based on a feature point extraction and matching method, so that feature point extraction does not need to be performed on each frame of image, and the processing speed is improved. However, when the method is adopted, the feature points capable of being identified and tracked in the tracking process are less and less due to the continuous motion of the moving object. And when the feature points are reduced to the state that the pose can not be calculated, returning to re-extract the feature points, so as to realize the identification and tracking of the moving object. Therefore, the method still needs to extract the feature points for many times, and when the feature points are reduced to the extent that the pose cannot be calculated, the feature points are extracted again, which may cause failure in identification and tracking, or cause error in identification and tracking due to error in the pose calculated again.
Based on the above problems, the embodiment of the present application provides a tracking method, which can supplement reduced feature points according to initial information when the feature points are reduced to a position and pose that cannot be accurately calculated, so that the number of the feature points is always greater than a number threshold, and the accuracy of the position and pose calculated according to the feature points is ensured, thereby improving the success rate of tracking a target object.
An exemplary application of the apparatus implementing the embodiment of the present application is described below, and the apparatus provided in the embodiment of the present application may be implemented as a terminal device. In the following, exemplary applications covering terminal devices when the apparatus is implemented as a terminal device will be explained.
Referring to fig. 1A, fig. 1A is a schematic diagram of a network architecture of a tracking method provided in an embodiment of the present application, as shown in fig. 1A, the network architecture at least includes a terminal 101, a server 102, a network 103, and a tracked target object 104. The terminal 101 may be a mobile terminal with wireless communication capability, such as a mobile phone (mobile phone), a tablet computer, and a notebook computer, and may also be AR glasses. The terminal 101 comprises at least a capturing device by which a user can capture an image of the tracked target object 104. The terminal 101 is shown in fig. 1A in the form of AR glasses for example. The server 102 may refer to one server, or may be a server cluster composed of a plurality of servers, a cloud computing center, or the like, and is not limited herein. The terminal 101 establishes a communication connection with the server 102 through a network 103, wherein the network 103 may be a wide area network or a local area network, or a combination of the two, and uses a wireless link to realize data transmission.
In the network architecture, a current frame image of a target object 104 may be acquired by a terminal 101, the current frame image of the target object 104 is sent to a server 102, then the server 102 acquires feature points of the current frame image, compares the number of the acquired feature points of the current frame image with a preset number threshold, updates the feature points of the current frame image based on initial information of the target object 104 stored in the server when the number of the feature points of the current frame image is smaller than the number threshold, so as to obtain updated feature points of the current frame image, then the server 102 sends the updated feature points of the current frame image to the terminal 101, and the terminal 101 continues to track the target object 104 based on the updated feature points of the current frame image.
Fig. 1B is a schematic diagram of another network architecture of the tracking method according to the embodiment of the present application, as shown in fig. 1B, the network architecture includes: a terminal 111 and a tracked target object 114. The terminal 111 comprises at least a capturing device by which a user can capture an image of the tracked target object 114. The terminal 111 is also shown exemplarily in fig. 1B in the form of AR glasses. The terminal 111 collects a current frame image of the target object 114, then obtains feature points of the current frame image, compares the number of the obtained feature points of the current frame image with the size of a preset number threshold, when the number of the feature points of the current frame image is smaller than the number threshold, updates the feature points of the current frame image based on the initial information of the target object 114 stored by the terminal, obtains updated feature points of the current frame image, and continuously tracks the target object 114 based on the updated feature points of the current frame image.
In the network architecture shown in fig. 1B, the network architecture shown in fig. 1A is generally adopted because the requirements for the computing efficiency, the storage space, and the like of the terminal 111 are high.
The apparatus provided in the embodiments of the present application may be implemented as hardware or a combination of hardware and software, and various exemplary implementations of the apparatus provided in the embodiments of the present application are described below.
Other exemplary configurations of the tracking device 200 are contemplated in view of the exemplary configuration of the tracking device 200 shown in FIG. 2, and thus the configurations described herein should not be considered limiting, e.g., some of the components described below may be omitted, or components not described below may be added to accommodate the particular needs of certain applications.
The tracking device 200 shown in fig. 2 includes: at least one processor 210, memory 240, at least one network interface 220, and a user interface 230. Each of the components in the tracking device 200 are coupled together by a bus system 250. It will be appreciated that the bus system 250 is used to enable communications among the components. The bus system 250 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 250 in fig. 2.
The user interface 230 may include a display, a keyboard, a mouse, a touch-sensitive pad, a touch screen, and the like.
The memory 240 may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM). The volatile Memory may be a Random Access Memory (RAM). The memory 240 described in embodiments herein is intended to comprise any suitable type of memory.
The memory 240 in embodiments of the present application is capable of storing data to support tracking the operation of the device 200. Examples of such data include: any computer program for operating on the tracking device 200, such as an operating system and application programs. The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application program may include various application programs.
As an example of the method provided by the embodiment of the present application implemented by software, the method provided by the embodiment of the present application may be directly embodied as a combination of software modules executed by the processor 210, the software modules may be located in a storage medium located in the memory 240, and the processor 210 reads executable instructions included in the software modules in the memory 240, and completes the method provided by the embodiment of the present application in combination with necessary hardware (for example, including the processor 210 and other components connected to the bus 250).
By way of example, the Processor 210 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor or the like.
The following describes embodiments of a tracking method, a tracking device, and a tracking device, with reference to application scenario diagrams shown in fig. 1A and fig. 1B.
The present embodiment provides a tracking method, which is applied to a tracking apparatus, which may be the server 102 as shown in fig. 1A, or may also be the terminal 111 as shown in fig. 1B.
Fig. 3 is a schematic flow chart of an implementation of the tracking method according to the embodiment of the present application, as shown in fig. 3, the method includes the following steps:
in step S301, feature points of the current frame image of the target object are acquired.
After the tracking device determines a tracked target object, firstly, an initial frame image acquired by the acquisition device is acquired, characteristic points of the initial frame image are extracted, then, the initial pose of the acquisition device when acquiring the initial frame image is determined according to the extracted characteristic points, and then, the extracted characteristic points are tracked based on an optical flow method.
The initial frame image is a first frame image of a target object acquired during tracking, the feature points of the frame image are used as a reference, the feature points of the previous frame image are used for acquiring the feature points of the current frame image based on an optical flow tracking principle, the current frame image is used as a new previous frame image, the feature points of the new current frame image are continuously acquired by using the feature points of the new previous frame image, and the feature points of the subsequent frame image are acquired in a circulating manner.
In the actual implementation process, if the current terminal moves from one direction to another direction, the content difference between the current frame image and the previous frame image is large, even the content is completely different. At this time, when the feature point of the next frame image is obtained according to the feature point of the previous frame image, the obtained feature point is almost rarely or even completely lost, so that the tracking cannot be continued; or the acquired feature points are erroneous, resulting in tracking errors. To solve this problem, in this embodiment, before step S302, it is determined whether the tracking is successful. Specifically, whether the current pose can be calculated is judged according to the acquired feature points of the current frame image, if the current pose can be calculated, the tracking success is determined, at this time, whether the number of the feature points of the current frame image is less than a number threshold is further judged, and when the number of the feature points of the current frame image is less than the number threshold, the step S302 is performed. If the current pose cannot be calculated, it is indicated that tracking fails, and at this time, the initial frame image of the target object needs to be obtained again for tracking.
Step S302, when the number of the feature points of the current frame image is smaller than the number threshold, updating the feature points of the current frame image according to the initial information of the target object.
If it is determined before step S302 that whether tracking is successful or not, when the number of the feature points of the current frame image is smaller than the number threshold, it indicates that the current pose cannot be accurately calculated according to the feature points of the current frame image, that is, the accuracy of the current pose calculated according to the feature points of the current frame image is low. If it is not determined whether the tracking is successful before step S302, when the number of the feature points of the current frame image is smaller than the number threshold, it indicates that the current pose cannot be calculated according to the feature points of the current frame image, or the current pose can be calculated according to the feature points of the current frame image, but the accuracy of the calculated current pose is low. At this time, it is necessary to acquire initial information of the target object, and perform an update operation on the feature point of the current frame image according to the initial information of the target object. Here, the initial information includes initial pose information of the acquisition device when acquiring the initial frame image and initial three-dimensional feature points corresponding to the feature points of the initial frame image.
In some embodiments, the obtaining process of the initial information of the target object may be implemented by:
in step S31, the feature points of the initial frame image are acquired.
In this embodiment, after the tracking device receives the initial frame image acquired by the acquisition device, the feature points of the initial frame image are extracted. The algorithm used for extracting the Features is an existing extraction algorithm, and for example, Feature extraction algorithms such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), FAST corner detection, and BRIEF Features (ORB) may be used. In some embodiments, the feature points of the initial frame image may also be artificially marked.
Each extracted feature point has two-dimensional spatial information which can represent the position of each feature point in the initial frame image. In practical applications, the two-dimensional spatial information may be coordinate information of each feature point in the initial frame image, for example, the two-dimensional spatial information of one extracted feature point may be (20, 30), and the two-dimensional spatial information may indicate that the feature point is a pixel point in row 20 and column 30.
Step S32, determining initial pose information of the acquisition device when acquiring the initial frame image and an initial three-dimensional feature point corresponding to the feature point of the initial frame image according to the feature point of the initial frame image and training data.
Here, the training data includes two-dimensional feature points extracted from training images of the target object acquired at different angles and three-dimensional feature points corresponding to the two-dimensional feature points.
In this embodiment, the feature point has two-dimensional space information and three-dimensional space information, where the two-dimensional space information of the feature point represents a position of the feature point in the frame image, and the three-dimensional information of the feature point represents a three-dimensional coordinate of the feature point in reality. For clarity, when a feature point in a frame image is specified, the feature point is called a two-dimensional feature point, and when a feature point in a space is specified, the feature point is called a three-dimensional feature point.
The tracking device stores training data of a tracked target object in advance, wherein the training data is acquired by the tracking device according to pre-acquired training images of a plurality of different angles of the target object. In the implementation process, the training images of the target object at a plurality of different angles acquired by the acquisition device are firstly acquired, and then the three-dimensional space model of the target object is established according to the depth information of the training images at the plurality of different angles. And performing feature extraction on each training image to obtain two-dimensional feature points of each training image, and combining the established three-dimensional space model to obtain three-dimensional feature points corresponding to each two-dimensional feature point, thereby obtaining training data. The training data includes two-dimensional feature points and three-dimensional feature points corresponding to the two-dimensional feature points.
After the tracking device obtains the feature points of the initial frame image, the initial two-dimensional feature points corresponding to the feature points are matched in the training data, and then the corresponding initial three-dimensional feature points are determined according to the initial two-dimensional feature points. And the tracking device determines the initial pose information when the acquisition device acquires the initial frame image by utilizing a PNP (Positive-negative-N-Point) algorithm according to the two-dimensional characteristic Point and the three-dimensional characteristic Point of the initial frame image and the internal parameters of the acquisition device.
After the initial information of the target object is obtained according to the steps S31 and S32, the current three-dimensional feature point of the current frame image is determined according to the initial three-dimensional feature point, the current three-dimensional feature point is mapped to the current frame image to obtain a mapping point, and finally the feature point of the current frame image is updated according to the mapping point, so that the supplement of the reduced feature points is realized, the number of the current frame feature points is more than a number threshold, the accuracy of the pose calculated according to the feature point of the current frame image is ensured, and the success rate of tracking the target object is improved.
And step S303, continuing to track the target object according to the updated feature points of the current frame image.
And after supplementing the feature points of the current frame image, continuing tracking the target object based on the optical flow method until the tracking is finished.
In the tracking method provided by the embodiment of the application, a tracking device obtains feature points of a current frame image of a target object, when the number of the feature points of the current frame image is smaller than a number threshold, the feature points of the current frame image are updated according to initial information of the target object, the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image, and finally the target object is continuously tracked according to the updated feature points of the current frame image; therefore, in the process of tracking the target object, when the feature points of the current frame image are reduced to the point that the current pose cannot be accurately calculated, the reduced feature points of the current frame image can be supplemented according to the initial information, so that the number of the feature points of the current frame image is more than the number threshold, the accuracy of the current pose calculated according to the feature points of the current frame image is ensured, and the success rate of tracking the target object is improved.
In some embodiments, the step S301 "acquiring feature points of the current frame image of the target object" may be implemented by:
step S3011, obtains feature points of the previous frame image of the target object.
And according to the optical flow tracking principle, acquiring the feature points of the second frame image through the feature points of the initial frame image, then acquiring the feature points of the third frame image according to the feature points of the second frame image, and repeating the steps to obtain the feature points of the current frame image. Therefore, when the feature point of the current frame image of the target object is to be acquired, the feature point of the previous frame image of the target object is acquired first.
Step S3012, determine the corresponding point of the feature point of the previous frame image in the current frame image.
In step S3012, firstly, an optical flow change between the previous frame image and the current frame image is determined according to the previous frame image and the current frame image, and then, a corresponding point of the feature point of the previous frame image in the current frame image is determined according to the optical flow change between the previous frame image and the current frame image, so as to obtain the feature point of the current frame image.
Step S3013, determine the corresponding point as a feature point of the current frame image.
The tracking method provided by the embodiment adopts an optical flow method for tracking, realizes the purpose of acquiring the feature points of the current frame image on the basis of the feature points of the previous frame image of the target object, can acquire the feature points of the current frame image without extracting the feature points of the current frame image, and improves the processing speed of the current frame image in the tracking process.
In some embodiments, the feature points of the current frame image can be directly matched with the three-dimensional feature points in the initial information according to only the feature points in the current frame image, but because the feature points of the current frame image are less than the number threshold, when the feature points in a small number are matched with the three-dimensional feature points in the initial information, the accuracy of successful matching is low, the accuracy of the updated feature points of the current frame image is influenced, the accuracy of the current pose calculated according to the updated feature points of the current frame image is reduced, and the success rate of tracking the target object is influenced.
In this embodiment, the feature point of the previous frame of image may be updated first, and then the feature point of the current frame of image may be obtained again according to the updated feature point of the previous frame of image. When the method is used to obtain the feature points of the current frame image, the step S3013 "determining the corresponding points as the feature points of the current frame image" may be implemented by the following steps:
step S3013a, acquiring first pose information of the acquisition device when the previous frame of image is acquired.
In step S3013a, when the tracking device is implemented, first, the tracking device determines a three-dimensional feature point corresponding to the two-dimensional feature point of the previous frame of image according to the feature point of the previous frame of image and the training data, and then determines first pose information of the acquisition device when the previous frame of image is acquired according to the two-dimensional feature point, the three-dimensional feature point, and the internal parameter of the acquisition device.
Step S3013b, updating the feature points of the previous frame of image according to the first pose information, the initial pose information, and the initial three-dimensional feature points.
In step S3013b, when the tracking device is implemented, a transformation model from the initial pose information to the first pose information is determined, the initial three-dimensional feature point is transformed into a first three-dimensional feature point according to the transformation model, the first three-dimensional feature point is mapped to the previous frame of image, a mapping point is obtained, and finally, the feature point of the previous frame of image is updated according to the mapping point.
Firstly, screening the mapping points according to the size of the previous frame of image and/or the first posture information to obtain the mapping points matched with the previous frame of image; and then determining the mapping point matched with the previous frame image as the updated characteristic point of the previous frame image.
According to the size difference between the previous frame image and the initial frame image, or according to the difference between the first pose information corresponding to the previous frame image and the initial pose information corresponding to the initial frame image, or according to the difference between the previous frame image and the initial frame image, and the difference between the first pose information corresponding to the previous frame image and the initial pose information corresponding to the initial frame image, after the first three-dimensional feature point is mapped to the previous frame image, the mapping point which does not accord with the size and/or pose information of the previous frame image is removed, and the mapping point which is matched with the previous frame image is obtained, so that the accuracy of the extracted feature point of the previous frame image is ensured.
Step S3013c, updating the feature point of the current frame image according to the updated feature point of the previous frame image and the first pose information.
In the process of tracking the target object, if the feature points of the current frame image are reduced to be less than the number threshold, the current pose of the acquisition device cannot be calculated according to the feature points of the current frame image, or the current pose can be calculated according to the feature points of the current frame image, but the calculation accuracy is low. In this embodiment, the feature points of the previous frame image are updated according to the three-dimensional feature points in the initial information, so that the matching accuracy can be improved, the feature points of the current frame image are obtained again according to the updated feature points of the previous frame image, and the accuracy of the updated feature points of the current frame image can be ensured, so that the accuracy of the current pose calculated according to the updated feature points of the current frame image is improved, and the success rate of tracking the target object is improved.
On the basis of the embodiment shown in fig. 3, an embodiment of the present application further provides a tracking method, and fig. 4 is a schematic flow chart of another implementation of the tracking method provided in the embodiment of the present application, as shown in fig. 4, the method includes:
step S401, a feature point of the current frame image of the target object is acquired.
Step S401, step S403, and step S404 in this embodiment specifically refer to descriptions of corresponding parts in step S301 to step S303 in the embodiment shown in fig. 3, and are not described again in this embodiment.
Step S402, determining whether the number of feature points of the current frame image is less than a number threshold.
When the number of the feature points of the current frame image is smaller than the number threshold, it indicates that the current pose cannot be accurately calculated according to the feature points of the current frame image, the feature points of the current frame image need to be updated, and the process proceeds to step S403. When the number of the feature points of the current frame image is greater than or equal to the number threshold, it indicates that the accuracy of the current pose calculated according to the feature points of the current frame image is high, and the feature points do not need to be updated, and the process proceeds to step S405.
Step S403, updating the feature points of the current frame image according to the initial information of the target object.
Here, the initial information includes initial pose information of the acquisition device when acquiring the initial frame image and initial three-dimensional feature points corresponding to the feature points of the initial frame image.
And S404, continuously tracking the target object according to the updated feature points of the current frame image.
And when the number of the feature points of the current frame image is greater than or equal to the number threshold, determining second pose information of the acquisition device when the current frame image is acquired according to the feature points of the current frame image, determining the current position of the target object by combining the initial pose information and the coordinates of the target object in the current frame image, and executing the step S405 and the step S406.
Step S405, determining second position information of the acquisition device and coordinates of the target object in the current frame image when the current frame image is acquired according to the feature points of the current frame image.
When the feature points of the current frame image are not updated, that is, the number of the feature points of the current frame image is greater than or equal to the number threshold, or when the feature points of the current frame image are the feature points of the updated current frame image, the second pose information corresponding to the current frame image is accurately calculated according to the feature points of the current frame image on the basis of the above embodiment.
In the actual implementation process, the two-dimensional feature points corresponding to the training data and the three-dimensional feature points corresponding to the two-dimensional feature points are determined according to the feature points of the current frame image, and then the second attitude information of the acquisition device when the current frame image is acquired is determined according to the two-dimensional feature points, the three-dimensional feature points and the internal parameters of the acquisition device.
And determining the coordinates of the target object in the current frame image according to the two-dimensional space information of all the characteristic points of the current frame image, namely the pixel positions of all the characteristic points in the current frame image, wherein the coordinates are plane coordinates.
Step S406, determining the position of the target object according to the initial pose information, the second pose information and the coordinates of the target object in the current frame image.
And determining the current position and orientation of the acquisition device in the world coordinate according to the second position information when the acquisition device acquires the current frame image and the initial position information when the acquisition device acquires the initial frame image obtained in the step S405. And determining the current position of the target object according to the known current position and orientation of the acquisition device and the coordinates of the acquired target object in the current frame image.
In some embodiments, as shown in fig. 4, after step S406, the method further comprises:
in step S407, it is determined whether an end tracking condition is reached.
When the end tracking condition is reached, returning to execute the step S405; when the end tracking condition is not met, the current frame image is taken as the previous frame image, and the step S401 is executed.
Step S408, stopping tracking and sending the position of the target object.
In this embodiment, when the end tracking condition is not reached, the tracking device always tracks the target object based on the optical flow method. In the tracking process, once the tracking ending condition is met, the tracking is stopped, and the current position of the target object when the tracking is stopped is sent to the terminal.
In a possible implementation, the end tracking condition may be: the tracking time reaches a preset tracking time, and/or an operation instruction which is sent by the terminal and used for indicating the end of tracking is received, and/or the arrival of the target object at a preset position is determined. For example, when the time period from the start of tracking to the current time reaches the preset tracking time period, the tracking is not continued, and the tracking apparatus determines that the end-of-tracking condition is reached. For another example, when the time period from the start of tracking to the current time reaches the preset tracking time period, or the target object reaches the preset position, the tracking is not continued, and the tracking apparatus determines that the end-of-tracking condition is reached.
Based on the foregoing embodiments, an embodiment of the present application further provides a tracking method, which is applied to a network architecture shown in fig. 1A, and fig. 5 is a schematic diagram of another implementation flow of the tracking method provided in the embodiment of the present application, and as shown in fig. 5, the method includes:
step S501, the terminal collects training images of a target object at a plurality of different angles.
In a static state of the target object, the position and the angle of the terminal are changed, the image of the target object at a plurality of different angles is acquired by the acquisition device of the terminal, and the image is uploaded to the server to be used as a training image for training, so that the feature point data of the moving target object at a plurality of different angles is obtained.
Step S502, the terminal sends a plurality of training images with different angles to the server.
Step S503, the server respectively extracts the two-dimensional feature points of the training images at the different angles and the three-dimensional feature points corresponding to the two-dimensional feature points.
When the step S503 is implemented, feature extraction is performed on the training images at each angle, so as to obtain two-dimensional feature points of all the training images, and then three-dimensional feature points corresponding to the two-dimensional feature points are obtained according to the two-dimensional position information of the two-dimensional feature points, in combination with the depth information of the feature points and the internal parameters of the acquisition device.
And each extracted two-dimensional feature point has two-dimensional space information, and the two-dimensional space information can represent the position of each two-dimensional feature point in the training image. In practical applications, the two-dimensional spatial information of the two-dimensional feature points may be coordinate information of the feature points in the training image. The three-dimensional spatial information of the three-dimensional feature points may be world coordinates of the feature points in reality.
Step S504, the server determines the two-dimensional feature points and the three-dimensional feature points corresponding to the two-dimensional feature points as training data.
Since the same feature point of the target object may exist in a plurality of training images, the two-dimensional feature point corresponding to the feature point in the training images is actually the same point. Therefore, a plurality of two-dimensional feature points of the same feature point of the target object in different training images are fused to obtain two-dimensional feature points of training data, then three-dimensional feature points corresponding to the two-dimensional feature points are determined according to the two-dimensional feature points, and the two-dimensional feature points and the three-dimensional feature points corresponding to the two-dimensional feature points are determined as the training data.
Step S505, the terminal acquires an initial frame image of the target object.
After the training data is obtained, the terminal acquires the initial frame image of the target object through the acquisition device of the terminal, and starts to track the target object.
Step S506, the terminal sends the initial frame image to the server.
Step S507, the server obtains the feature points of the initial frame image according to the initial frame image.
The server adopts the existing feature extraction algorithm to extract the feature points of the initial frame image to obtain the feature points of the initial frame image. For example, feature extraction algorithms such as Scale-Invariant feature transform (SIFT), Speeded Up Robust Features (SURF), FAST corner detection and BRIEF Features (ORB) may be utilized. In some embodiments, the feature points of the initial frame image may also be artificially marked.
Step S508, the server determines, according to the feature points of the initial frame image and the training data, initial pose information of the acquisition device when acquiring the initial frame image and initial three-dimensional feature points corresponding to the feature points of the initial frame image.
After the tracking device obtains the feature points of the initial frame image, the initial two-dimensional feature points corresponding to the feature points are matched in the training data, and then the corresponding initial three-dimensional feature points are determined according to the initial two-dimensional feature points. And the tracking device determines initial pose information when the acquisition device acquires the initial frame image according to the two-dimensional characteristic points and the three-dimensional characteristic points of the initial frame image and the internal parameters of the acquisition device.
And after the terminal sends the initial frame image to the server, continuing to acquire the subsequent frame image of the target object and sending the subsequent frame image to the server. The server starts tracking the target object on the received frame image based on the optical flow method.
In step S509, the server acquires the feature point of the previous frame image of the target object.
And the server acquires the characteristic points of the second frame image according to the characteristic points of the initial frame image, acquires the characteristic points of the third frame image according to the characteristic points of the second frame image, and circularly acquires the characteristic points of the previous frame image.
Step S510, the terminal acquires a current frame image of the target object.
And step S511, the terminal sends the current frame image to a server.
In step S512, the server determines the corresponding point of the feature point of the previous frame image in the current frame image.
And the server determines the optical flow change between the previous frame image and the current frame image according to the previous frame image and the current frame image, and then determines the corresponding point of the feature point of the previous frame image in the current frame image according to the optical flow change of the previous frame image and the current frame image, so as to obtain the feature point of the current frame image.
In step S513, the server determines the corresponding point as a feature point of the current frame image.
In step S514, the server determines whether the number of feature points of the current frame image is less than a number threshold.
When the number of feature points of the current frame image is smaller than the number threshold, the process proceeds to step S515. When the number of feature points of the current frame image is greater than or equal to the number threshold, the process proceeds to step S523.
Step S515, the server determines a previous frame three-dimensional feature point corresponding to the feature point of the previous frame image according to the feature point of the previous frame image and the training data.
Step S516, the server determines the first pose information of the acquisition device when acquiring the previous frame of image according to the feature point of the previous frame of image, the three-dimensional feature point of the previous frame and the internal parameters of the acquisition device.
And step S517, the server determines a transformation model from the initial pose information to the first pose information.
Step S518, the server transforms the initial three-dimensional feature point into a first three-dimensional feature point according to the transformation model.
Here, the first three-dimensional feature point is a three-dimensional feature point corresponding to the previous frame image.
Since the tracked target object is invariant, the three-dimensional model of the target object is invariant. The transformation relation from the initial pose information to the first pose information of the acquisition device corresponds to the transformation relation from the initial three-dimensional feature point of the initial frame image to the first three-dimensional feature point of the previous frame image, so that the initial three-dimensional feature point can be transformed into the first three-dimensional feature point according to the transformation model.
In step S519, the server maps the first three-dimensional feature point to the previous frame of image to obtain a mapped point.
And step S520, the server performs screening operation on the mapping points according to the size of the previous frame of image and/or the first posture information to obtain the mapping points matched with the previous frame of image.
And after the first three-dimensional characteristic point is mapped to the previous frame of image, eliminating the mapping points which do not accord with the size and/or pose information of the previous frame of image to obtain the mapping points which are matched with the previous frame of image so as to ensure the accuracy of the characteristic points of the previous frame of image.
In step S521, the server determines that the mapping point matched with the previous frame of image is the updated feature point of the previous frame of image.
The mapping points after the screening operation are used as the feature points after the last frame of image is updated, and the feature points lost in the last frame of image are supplemented.
In step S522, the server continues to track the target object according to the updated feature point of the current frame image.
When the number of the feature points of the current frame image is smaller than the number threshold, the feature points of the previous frame image are updated, the feature points of the current frame image are obtained again according to the updated feature points of the previous frame image, the feature points of the current frame image are supplemented, the number of the feature points of the current frame image is larger than the number threshold, the accuracy of the current pose calculated according to the feature points of the current frame image is ensured, and the success rate of tracking the target object is improved.
Step S523, the server determines, according to the feature points of the current frame image, second position information of the acquisition device and coordinates of the target object in the current frame image when the current frame image is acquired.
Step S524, the server determines the position of the target object according to the initial pose information, the second pose information, and the coordinates of the target object in the current frame image.
The number of the feature points of the current frame image is greater than or equal to the number threshold, and the second position and attitude information corresponding to the current frame image can be accurately calculated according to the feature points of the current frame image, so that the current position of the target object can be determined, and real-time tracking is realized.
In step S525, the server determines whether an end tracking condition is reached.
When the tracking condition is reached, the flow proceeds to step S526. When the tracking condition is not met, the current frame image is taken as the previous frame image, and the step S510 is executed again.
In step S526, the server sends an instruction to stop tracking to the terminal.
Step S527, the server sends the position of the target object to the terminal.
In step S528, the terminal stops tracking.
In this embodiment, when the tracking end condition is not reached, the server always tracks the target object based on the optical flow method. In the tracking process, once the tracking ending condition is met, the tracking is stopped, and the instruction for stopping the tracking and the current position of the target object when the tracking is stopped are sent to the terminal. And the terminal stops tracking according to the received instruction.
In the tracking method provided by the embodiment, in the process of tracking the target object, when the feature points of the current frame image are reduced to the point that the current pose cannot be accurately calculated, the feature points of the previous frame image can be updated according to the three-dimensional feature points in the initial information, and then the feature points of the current frame image are obtained again according to the updated feature points of the previous frame image, so that the feature points reduced in the current frame image are supplemented, the number of the feature points of the current frame image is more than the number threshold, the accuracy of the updated feature points of the current frame image is ensured, the accuracy of the current pose calculated according to the updated feature points of the current frame image is improved, and the success rate of tracking the target object is further improved.
Based on the foregoing embodiments, the present application further provides a tracking method, which is applied to the network architecture shown in fig. 1A. In the tracking method provided by the embodiment of the application, firstly, a server acquires training data of a target object according to training images of a plurality of different angles acquired by an acquisition device of a terminal. After tracking is started, an initial frame image is obtained, feature points in the initial frame image are extracted, each frame image in the tracking process is continuously obtained, feature points of a current frame image are obtained according to the feature points of the previous frame image, whether the pose can be calculated or not is judged according to the obtained feature points of the current frame image, if the current pose cannot be calculated according to the feature points of the current frame image, tracking failure is determined, and tracking is restarted after the tracking failure. If the current pose can be calculated according to the feature points of the current frame image, the tracking success is determined, whether the feature points of the current frame image are less than the number threshold is further judged, and if the feature points of the current frame image are less than the number threshold, the tracking is continued after the feature points of the current frame image are supplemented.
Fig. 6 is a schematic flowchart of another implementation flow of the tracking method according to the embodiment of the present application, and as shown in fig. 6, the method includes:
step S601, calculating the pose based on the feature point extraction and matching method.
After acquiring an initial frame image acquired by an acquisition device of the terminal, the server extracts feature points in the initial frame image and calculates the initial pose of the acquisition device.
Since the training image of the target object is an image matching the initial frame image, the corresponding three-dimensional feature points of the two-dimensional feature points in the training data matching the feature points of the initial frame image may be used as the three-dimensional feature points of the initial frame image. After the two-dimensional characteristic points, the three-dimensional characteristic points and the internal parameters of the acquisition device which are matched with the characteristic points of the initial frame image are known, the PNP algorithm is utilized to determine the initial pose of the acquisition device when the acquisition device acquires the initial frame image, wherein the initial pose can comprise the position, the rotation direction and the like of the acquisition device when the initial frame image is acquired.
Step S602, the feature points are tracked based on the optical flow method.
In practical implementation, each frame of image in the tracking process is acquired, the feature points of the current frame of image are acquired according to the feature points of the previous frame of image, the current frame of image is taken as the previous frame of image, and the next frame of image is continuously acquired.
Step S603, determining whether the tracking is successful.
If the current terminal's collection device moves from one position to another position, the content difference of the current frame image and the last frame image that is gathered is great, even the content is totally different. At this time, when the feature point of the next frame image is obtained according to the feature point of the previous frame image, the obtained feature point is almost rarely or even completely lost, so that the tracking cannot be continued; or the acquired feature points are erroneous, resulting in tracking errors. To solve this problem, in this embodiment, before step S604, it is determined whether the tracking is successful. Specifically, it is determined whether the current pose can be calculated according to the feature points of the obtained current frame image, and if the current pose can be calculated, it is determined that the tracking is successful, and the process proceeds to step S604. If the current pose cannot be calculated, it indicates that tracking fails, at this time, the initial frame image of the target object needs to be obtained again, and tracking is performed, that is, the step S601 is executed.
In step S604, it is determined whether the number of feature points is less than the set number threshold.
When the optical flow method is adopted for tracking, the acquired feature points are less and less due to the movement of the target. When the feature points of the current frame image are less than the number threshold, although the current pose can be calculated, the accuracy of the calculated current pose cannot be ensured. In this embodiment, when the feature points of the current frame image are reduced to the number threshold, the process proceeds to step S605. If not, the process returns to step S602.
Step S605 supplements the feature points.
When the number of the feature points of the current frame image is smaller than the number threshold, the feature points of the current frame image are updated according to the initial information of the target object, and then the target object is continuously tracked according to the updated feature points of the current frame image, that is, the step S602 is executed again.
In the tracking method provided by the embodiment, in the process of tracking the target object, when the feature points of the current frame image are reduced to the point that the current pose cannot be accurately calculated, the feature points of the previous frame image can be updated according to the three-dimensional feature points in the initial information, and then the feature points of the current frame image are obtained again according to the updated feature points of the previous frame image, so that the feature points reduced in the current frame image are supplemented, the number of the feature points of the current frame image is more than the number threshold, the accuracy of the updated feature points of the current frame image is ensured, the accuracy of the current pose calculated according to the updated feature points of the current frame image is improved, and the success rate of tracking the target object is further improved.
Continuing with the exemplary structure of the tracking device 700 provided by the embodiments of the present application as implemented as software modules, in some embodiments, as shown in fig. 2, the software modules stored in the tracking device 700 of the memory 140 may include: a first obtaining module 701, an updating module 702, and a tracking module 703, wherein:
a first obtaining module 701, configured to obtain a feature point of a current frame image of a target object;
an updating module 702, configured to update the feature points of the current frame image according to initial information of the target object when the number of the feature points of the current frame image is smaller than a number threshold, where the initial information includes initial pose information of an acquisition device when acquiring an initial frame image and initial three-dimensional feature points corresponding to the feature points of the initial frame image;
and the tracking module 703 is configured to continue tracking the target object according to the updated feature point of the current frame image.
In other embodiments, the first obtaining module 701 further includes:
the first obtaining submodule is used for obtaining the characteristic points of the last frame of image of the target object;
the first determining submodule is used for determining the corresponding point of the characteristic point of the previous frame image in the current frame image;
and the second determining submodule is used for determining the corresponding point as the characteristic point of the current frame image.
In other embodiments, the update module 702 further includes:
the second acquisition submodule is used for acquiring the first position and orientation information of the acquisition device when the previous frame of image is acquired;
the first updating submodule is used for updating the feature points of the previous frame of image according to the first pose information, the initial pose information and the initial three-dimensional feature points;
and the second updating submodule is used for updating the feature point of the current frame image according to the updated feature point of the previous frame image and the first pose information.
In other embodiments, the apparatus further comprises:
the second acquisition module is used for acquiring the characteristic points of the initial frame image;
and the first determining module is used for determining initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional characteristic points corresponding to the characteristic points of the initial frame image according to the characteristic points of the initial frame image and training data. The training data comprises two-dimensional feature points extracted according to training images acquired by the target object at different angles and three-dimensional feature points corresponding to the two-dimensional feature points.
In other embodiments, the second obtaining sub-module further includes:
the first determining unit is used for determining a previous frame three-dimensional feature point corresponding to the feature point of the previous frame image according to the feature point of the previous frame image and the training data;
and the second determining unit is used for determining the first position and orientation information of the acquisition device when the previous frame of image is acquired according to the feature point of the previous frame of image, the three-dimensional feature point of the previous frame and the internal parameters of the acquisition device.
In other embodiments, the first update sub-module further includes:
a third determination unit configured to determine a transformation model from the initial pose information to the first pose information;
a first transformation unit configured to transform the initial three-dimensional feature point into a first three-dimensional feature point according to the transformation model;
the first mapping unit is used for mapping the first three-dimensional feature point to the previous frame of image to obtain a mapping point;
and the first updating unit is used for updating the characteristic points of the previous frame of image according to the mapping points.
In other embodiments, the first updating unit further includes:
the first screening subunit is configured to perform a screening operation on the mapping points according to the size of the previous frame of image and/or the first pose information to obtain mapping points matched with the previous frame of image;
and the first determining subunit is used for determining the mapping point matched with the previous frame image as the updated feature point of the previous frame image.
In other embodiments, the apparatus further comprises:
the second determining module is used for determining second position and attitude information of the acquisition device and coordinates of the target object in the current frame image when the current frame image is acquired according to the feature points of the current frame image when the number of the feature points of the current frame image is greater than or equal to a number threshold;
and the third determining module is used for determining the position of the target object according to the initial pose information, the second pose information and the coordinates of the target object in the current frame image.
In other embodiments, the apparatus further comprises:
a fourth determination module for determining whether an end tracking condition is reached;
and the first sending module is used for stopping tracking and sending the position of the target object when the tracking ending condition is determined to be reached.
Here, it should be noted that: the above description of the tracking device embodiment is similar to the above description of the method and has the same advantageous effects as the method embodiment. For technical details not disclosed in the embodiments of the tracking device of the present application, a person skilled in the art shall refer to the description of the embodiments of the method of the present application for understanding.
Embodiments of the present application provide a storage medium having stored therein executable instructions, which when executed by a processor, will cause the processor to perform the methods provided by embodiments of the present application, for example, the methods as illustrated in fig. 3, 4 and 5.
In some embodiments, the storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.
In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.
The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims (10)

1. A tracking method, comprising:
acquiring feature points of a current frame image of a target object;
when the number of the feature points of the current frame image is smaller than a number threshold value, updating the feature points of the current frame image according to initial information of the target object, wherein the initial information comprises initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image;
and continuously tracking the target object according to the updated feature points of the current frame image.
2. The method of claim 1, wherein the obtaining of the feature points of the current frame image of the target object comprises:
acquiring a characteristic point of a previous frame of image of a target object;
determining the corresponding point of the feature point of the previous frame image in the current frame image;
and determining the corresponding point as a characteristic point of the current frame image.
3. The method of claim 2, wherein the updating the feature points of the current frame image according to the initial information of the target object comprises:
acquiring first position and orientation information of the acquisition device when the previous frame of image is acquired;
updating the feature points of the previous frame of image according to the first pose information, the initial pose information and the initial three-dimensional feature points;
and updating the feature points of the current frame image according to the updated feature points of the previous frame image and the first posture information.
4. The method of claim 3, further comprising:
acquiring characteristic points of an initial frame image;
determining initial pose information of an acquisition device when the initial frame image is acquired and initial three-dimensional feature points corresponding to the feature points of the initial frame image according to the feature points of the initial frame image and training data, wherein the training data comprise two-dimensional feature points extracted according to training images acquired by the target object at different angles and three-dimensional feature points corresponding to the two-dimensional feature points.
5. The method of claim 4, the acquiring first pose information of the acquisition device at the time of acquiring the previous frame of image, comprising:
determining a previous frame three-dimensional feature point corresponding to the feature point of the previous frame of image according to the feature point of the previous frame of image and the training data;
and determining first position and attitude information of the acquisition device when the previous frame of image is acquired according to the feature point of the previous frame of image, the three-dimensional feature point of the previous frame and the internal parameters of the acquisition device.
6. The method of claim 3, said updating feature points of the previous frame of image according to the first pose information, initial pose information, and initial three-dimensional feature points, comprising:
determining a transformation model from the initial pose information to the first pose information;
transforming the initial three-dimensional feature points into first three-dimensional feature points according to the transformation model;
mapping the first three-dimensional feature point to the previous frame of image to obtain a mapping point;
and updating the characteristic points of the previous frame of image according to the mapping points.
7. The method of claim 6, the updating feature points of the previous frame of image according to the mapping points, comprising:
screening the mapping points according to the size of the previous frame of image and/or the first posture information to obtain mapping points matched with the previous frame of image;
and determining the mapping point matched with the previous frame image as the updated feature point of the previous frame image.
8. The method of claim 1, further comprising:
when the number of the feature points of the current frame image is greater than or equal to the number threshold, determining second attitude information of the acquisition device and coordinates of the target object in the current frame image when the current frame image is acquired according to the feature points of the current frame image;
and determining the position of the target object according to the initial pose information, the second pose information and the coordinates of the target object in the current frame image.
9. The method of claim 8, further comprising:
determining whether an end tracking condition is reached;
when it is determined that the end tracking condition is reached, stopping tracking, and transmitting the position of the target object.
10. A tracking device, comprising:
a memory for storing executable instructions;
a processor for implementing the method of any one of claims 1 to 9 when executing executable instructions stored in the memory.
CN201911319416.9A 2019-12-19 2019-12-19 Tracking method and device Active CN111179309B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911319416.9A CN111179309B (en) 2019-12-19 2019-12-19 Tracking method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911319416.9A CN111179309B (en) 2019-12-19 2019-12-19 Tracking method and device

Publications (2)

Publication Number Publication Date
CN111179309A true CN111179309A (en) 2020-05-19
CN111179309B CN111179309B (en) 2024-05-24

Family

ID=70657562

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911319416.9A Active CN111179309B (en) 2019-12-19 2019-12-19 Tracking method and device

Country Status (1)

Country Link
CN (1) CN111179309B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112396654A (en) * 2020-11-17 2021-02-23 闪耀现实(无锡)科技有限公司 Method and device for determining pose of tracking object in image tracking process
CN112837424A (en) * 2021-02-04 2021-05-25 脸萌有限公司 Image processing method, device, equipment and computer readable storage medium
CN113450389A (en) * 2021-09-01 2021-09-28 智道网联科技(北京)有限公司 Target tracking method and device and electronic equipment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107481315A (en) * 2017-06-29 2017-12-15 重庆邮电大学 A kind of monocular vision three-dimensional environment method for reconstructing based on Harris SIFT BRIEF algorithms
CN109509211A (en) * 2018-09-28 2019-03-22 北京大学 Positioning simultaneously and the feature point extraction and matching process and system built in diagram technology
CN109753940A (en) * 2019-01-11 2019-05-14 京东方科技集团股份有限公司 Image processing method and device
CN109887003A (en) * 2019-01-23 2019-06-14 亮风台（上海）信息科技有限公司 A kind of method and apparatus initialized for carrying out three-dimensional tracking
US20190244050A1 (en) * 2017-07-07 2019-08-08 Tencent Technology (Shenzhen) Company Limited Method, device and storage medium for determining camera posture information

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107481315A (en) * 2017-06-29 2017-12-15 重庆邮电大学 A kind of monocular vision three-dimensional environment method for reconstructing based on Harris SIFT BRIEF algorithms
US20190244050A1 (en) * 2017-07-07 2019-08-08 Tencent Technology (Shenzhen) Company Limited Method, device and storage medium for determining camera posture information
CN109509211A (en) * 2018-09-28 2019-03-22 北京大学 Positioning simultaneously and the feature point extraction and matching process and system built in diagram technology
CN109753940A (en) * 2019-01-11 2019-05-14 京东方科技集团股份有限公司 Image processing method and device
CN109887003A (en) * 2019-01-23 2019-06-14 亮风台（上海）信息科技有限公司 A kind of method and apparatus initialized for carrying out three-dimensional tracking

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112396654A (en) * 2020-11-17 2021-02-23 闪耀现实(无锡)科技有限公司 Method and device for determining pose of tracking object in image tracking process
CN112837424A (en) * 2021-02-04 2021-05-25 脸萌有限公司 Image processing method, device, equipment and computer readable storage medium
CN112837424B (en) * 2021-02-04 2024-02-06 脸萌有限公司 Image processing method, apparatus, device and computer readable storage medium
CN113450389A (en) * 2021-09-01 2021-09-28 智道网联科技(北京)有限公司 Target tracking method and device and electronic equipment

Also Published As

Publication number Publication date
CN111179309B (en) 2024-05-24

Similar Documents

Publication Publication Date Title
CN110310326B (en) Visual positioning data processing method and device, terminal and computer readable storage medium
CN107292949B (en) Three-dimensional reconstruction method and device of scene and terminal equipment
CN108805917B (en) Method, medium, apparatus and computing device for spatial localization
CN111797650B (en) Obstacle identification method, obstacle identification device, computer equipment and storage medium
CN111598993B (en) Three-dimensional data reconstruction method and device based on multi-view imaging technology
CN110176032B (en) Three-dimensional reconstruction method and device
CN109683699B (en) Method and device for realizing augmented reality based on deep learning and mobile terminal
CN111179309B (en) Tracking method and device
US11940774B2 (en) Action imitation method and robot and computer readable storage medium using the same
JP7422105B2 (en) Obtaining method, device, electronic device, computer-readable storage medium, and computer program for obtaining three-dimensional position of an obstacle for use in roadside computing device
WO2021139176A1 (en) Pedestrian trajectory tracking method and apparatus based on binocular camera calibration, computer device, and storage medium
CN110986969B (en) Map fusion method and device, equipment and storage medium
WO2015085779A1 (en) Method and system for calibrating surveillance cameras
CN110866497B (en) Robot positioning and mapping method and device based on dotted line feature fusion
WO2021027692A1 (en) Visual feature library construction method and apparatus, visual positioning method and apparatus, and storage medium
CN112927363B (en) Voxel map construction method and device, computer readable medium and electronic equipment
CN112556685B (en) Navigation route display method and device, storage medium and electronic equipment
CN111127524A (en) Method, system and device for tracking trajectory and reconstructing three-dimensional image
CN111325798B (en) Camera model correction method, device, AR implementation equipment and readable storage medium
CN112819892B (en) Image processing method and device
CN111986214A (en) Construction method of pedestrian crossing in map and electronic equipment
CN112967340A (en) Simultaneous positioning and map construction method and device, electronic equipment and storage medium
CN110348351B (en) Image semantic segmentation method, terminal and readable storage medium
CN111862150A (en) Image tracking method and device, AR device and computer device
CN112085842B (en) Depth value determining method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant