CN110059651B - Real-time tracking and registering method for camera - Google Patents
Real-time tracking and registering method for camera Download PDFInfo
- Publication number
- CN110059651B CN110059651B CN201910334821.1A CN201910334821A CN110059651B CN 110059651 B CN110059651 B CN 110059651B CN 201910334821 A CN201910334821 A CN 201910334821A CN 110059651 B CN110059651 B CN 110059651B
- Authority
- CN
- China
- Prior art keywords
- image
- key frame
- camera
- feature point
- points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
Abstract
The invention relates to a camera real-time tracking and registering method in an augmented reality system, which comprises the following steps: selecting two input images under an unknown scene, initializing the scene by using a European reconstruction method, designing a key frame online selection method, updating the key frame in real time, and realizing the reconstruction of the unknown scene based on local beam set adjustment; expressing the key frame by taking the neighborhood of the feature point on the key frame as basic data, training a classifier by a random classification tree algorithm, and performing the learning and identification tasks of the key frame; designing an SIFT feature point matching algorithm based on GPU acceleration, and matching feature points; and designing a system initialization algorithm based on two-stage feature matching according to the feature point matching result, solving the initial pose of the camera, and finishing system initialization based on the initial pose state. The invention reduces the calculation amount, improves the calculation speed and can meet the requirement of real-time tracking registration of an augmented reality system.
Description
Technical Field
The invention relates to the field of computer vision and computer augmented reality, in particular to a camera real-time tracking and registering method in an augmented reality system.
Background
With the continuous enhancement of the computing performance of mobile platforms, computer vision research is rapidly developed and applied to many fields such as identification, navigation, retrieval and the like. The augmented reality system is one of important application fields of computer vision, virtual digital information generated by a computer is fused and displayed with a real environment through special display equipment, and practical auxiliary information outside the real environment, such as characters, videos, three-dimensional animations and the like, can be provided for people. The augmented reality system can analyze environmental information and environmental characteristics in a real scene by utilizing a computer vision technology, draws auxiliary information generated by a computer at a specified position and helps people to better understand the scene, wherein the real-time tracking and registration of the camera pose in the augmented reality system is a core module which ensures that the auxiliary information is accurately and stably displayed at the specified position.
At present, most of the known mobile platform augmented reality applications adopt a method based on artificial calibration to complete three-dimensional registration of a camera. Such systems mostly adopt methods of extracting feature points for comparison or matching gray values. Due to the characteristics of relative simplicity and rules of the identification image, a better matching result can be obtained quickly, so that quick tracking registration is realized. However, the artificially set identification image is often not coordinated with the actual environment information, and the perception experience of the user to the real world is damaged to a certain extent. Meanwhile, the method has a limited practical range in a complex large scene. Therefore, the camera tracking registration method based on the natural features has practical research value, and the conventional algorithm based on the natural features generally has the characteristics of low speed, good real-time performance and the like due to high calculation cost.
Disclosure of Invention
The invention aims to provide a camera real-time tracking and registering method in an augmented reality system, which is used for solving the problem of registration failure caused by jitter and characteristic point reduction in the characteristic point matching and tracking processes.
The invention discloses a camera real-time tracking and registering method in an augmented reality system, which comprises the following steps: selecting two input images under an unknown scene, initializing the scene by using a European reconstruction method, designing a key frame online selection method, updating the key frame in real time, and realizing the reconstruction of the unknown scene based on local beam set adjustment; expressing the key frame by taking the neighborhood of the feature point on the key frame as basic data, training a classifier by a random classification tree algorithm, and performing the learning and identification tasks of the key frame; designing an SIFT feature point matching algorithm based on GPU acceleration, and matching feature points; and designing a system initialization algorithm based on two-stage feature matching according to the feature point matching result, calculating initial pose information of the camera by using the feature point matching result in one stage, performing feature point extraction and matching operation again after image scaling processing in the other stage, and finishing system initialization according to the initial pose information in the first stage.
The invention relates to a camera real-time tracking registration method in an augmented reality system, which is characterized in that based on the feature point matching result, a system initialization algorithm based on two-stage feature matching is designed, the initial pose of a camera is obtained, and the system initialization is completed based on the initial pose state; and dividing the scene structure reconstruction and the camera tracking into two relatively independent tasks, and completing real-time tracking and registration based on multithreading. Compared with the existing tracking registration algorithm based on natural features, the method has the advantages that the calculation amount is reduced, the calculation speed is increased, and the real-time tracking registration requirement of an augmented reality system can be met.
Drawings
FIG. 1 is a flowchart of a method for real-time tracking and registering a camera during a tracking and updating process;
FIG. 2 is a schematic diagram of texture storage in two independent processes in feature point detection;
FIG. 3 is a diagram of feature point matching and tracking results;
FIG. 4 is a flow chart of a multi-threaded camera real-time tracking registration process;
FIG. 5 is a graph comparing performance of the present invention in a scene reconstruction phase using a local bundle set adjustment method with a conventional method;
fig. 6a and 6b are graphs of registration effect under different illumination conditions.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
Fig. 1 is a flowchart of a work flow of a real-time camera tracking and registering method in a tracking and updating process, and as shown in fig. 1, the work flow of the real-time camera tracking and registering method in an iterative updating process specifically includes:
firstly, at the system starting stage, two input images under an unknown scene are selected, and scene initialization is completed by utilizing an European reconstruction method.
And secondly, designing a key frame online selection method to complete the real-time updating of the key frame.
The online key frame selecting method comprises the following steps:
if the following two rules are satisfied by one frame of input image, the input image is taken as candidate key frame data.
a. Presetting a camera pose calculation allowable threshold, wherein the pose calculation result of the camera must be accurate to the threshold;
b. a key frame interval time threshold is preset, and the input time interval between a new key frame and a previous key frame must exceed the threshold.
And after one frame of input image is selected as candidate key frame data according to the rule, if any one of the following two rules is met, the frame of input image is taken as new key frame data and added into the system to complete the three-dimensional reconstruction of the new feature point.
c. Taking a baseline distance between two key frames used for scene initialization as a baseline distance threshold, wherein the baseline distance between a candidate key frame and an existing nearest key frame is smaller than the threshold;
d. a threshold value of the rotation angle of the key frame image along any axis is preset, and the rotation component between the candidate key frame and the existing nearest key frame along any coordinate axis is larger than the threshold value.
According to the key frame selection method, once a new key frame image is added into the current system, a feature point matching set between the current key frame and the existing key frame closest to the key frame is established by directly utilizing epipolar constraint and a normalized cross-correlation algorithm. And then the triangulation is directly utilized to calculate the three-dimensional position of the new feature point for the subsequent camera pose calculation process.
Third, using the feature points around the key frameAnd expressing the key frame by the image neighborhood to obtain basic data of key frame image training and learning. The method specifically comprises the following steps: given a set of keyframes { K ] acquired by online reconstructioniSuppose for each KiThe image neighborhood f corresponding to the jth feature point is obtainedi,j. Set { fi,jConstitute the basic data needed to train the classifier.
And fourthly, after the basic data are obtained, training and learning are carried out on the basic data by utilizing the improved random classification tree method of the invention, and the tasks of on-line learning and identification of the key frame are completed. The specific process is as follows:
when a new key frame SiAfter adding to the system, we first obtain a certain number of feature point neighborhoods { f over the keyframei,1,fi,2,...,fi,NAnd then, all image neighborhoods are subjected to positive inversion operation according to the main directions of the corresponding SIFT descriptors, so that the negative influence of rotation on recognition and training is avoided. After the positive conversion operation, a Gaussian filter is adopted to carry out smooth operation on the image neighborhood so as to avoid the influence of image noise on the training result to the maximum extent.
Each processed image neighborhood is directly tested according to the internal nodes on each tree
To leaf nodes of each tree, where θiFor the compensation value, the compensation value is used to compensate the influence of noise on the gray value difference between two pixels in the image neighborhood, in this embodiment, the value is θiAnd (4) completely randomly taking a value, wherein the value range is 0-z.
Then according to the formula
And replacing the posterior probability stored in the classification leaf node by the number of image neighborhoods. When deleting a class, the number of image neighborhoods corresponding to the class on each leaf node is directly set to 0.
For the recognition process, the features on the input image are first combinedImage neighborhood around a feature point { f1,f2,...,fNAnd performing correction and smoothing operations, sequentially transforming the transformed image neighborhood images to leaf nodes of each tree, and finally returning the type with the maximum image neighborhood quantity and the following formula as an identification result to the tracking system to finish initialization.
And fifthly, accelerating to complete an SIFT feature point matching algorithm based on the GPU. The specific process comprises the following steps:
the data structure in the traditional SIFT feature point matching algorithm is converted into a texture format which is convenient for GPU processing, and 4 color channels of the texture format are respectively used for storing image pixel gray scale, Gaussian difference image, gradient vector and gradient direction.
Meanwhile, in order to save storage space, a list structure is adopted to store the feature point information, and feature points on different scale spaces are stored separately. Thus, after the vital sign point extraction operation, a feature point storage list can be created, and the system then constructs a feature point list with a single main direction to separately store feature points with a plurality of main directions.
For scale-space image generation based on gaussian filtering, the embodiment divides it into two independent tasks of filtering in the horizontal direction and the vertical direction to be executed in parallel. By writing the gray value in the temporary register back to the original color channel, two independent filtering processes can read and write the same texture space at the same time, so that the parallelism of the program is effectively improved to improve the operation speed.
Fig. 2 is a schematic diagram illustrating texture storage in two independent processes in feature point detection, and as shown in fig. 2, in the feature point extraction operation, the process of comparing the pixel gray level with the current layer and several adjacent pixels on the upper and lower layers of gaussian difference images is divided into two independent processes of intra-layer comparison and inter-layer comparison to save texture storage space.
The first process is used to compare adjacent pixels in the layer and store the maximum or minimum values in the auxiliary texture memory, and the channel is responsible for the calculation of the gradient vector and the gradient direction.
The second process is used to perform inter-layer comparison, which only needs to compare with 2 elements in the upper and lower layers because non-maximum or minimum elements have been eliminated by the intra-layer comparison.
The condition that a certain pixel is maximum (minimum) is that the pixel has a maximum (minimum) value between layers and is larger (smaller) than the maximum values (minimum) values on both the upper and lower layers.
And sixthly, completing the on-line tracking of the feature points by an optical flow method after completing the matching of the feature points. The embodiment also adopts the following strategy to quickly eliminate wrong tracking points, thereby ensuring the accuracy and the stability of the system.
First, adopt Td,dThe test accelerates the 3-point RANSAC mismatching point elimination algorithm, randomly selects a fourth pair of corresponding points while randomly selecting three pairs of corresponding points, the fourth pair of corresponding points is firstly used for carrying out a reprojection test, if a projection matrix generated by the random three points is effective on the fourth pair of corresponding points (the reprojection error is less than a predefined threshold value), and carries out reprojection operation on other corresponding points, so that the method has the advantages of reducing unnecessary reprojection operation caused by an incorrect projection matrix and saving the operation time so that the algorithm can carry out random sampling inspection for more times.
Secondly, after randomly extracting three pairs of corresponding points, firstly judging whether the three points are approximately positioned on a straight line or not, or judging whether the distance between any two points in the three points is smaller than a threshold value defined in advance, and if the distance between any two points in the three points is consistent with any one of the two points, abandoning the current set and turning to the next sampling. By doing so, the corresponding point set with invalid or poor calculation precision can be eliminated in advance, and unnecessary projection matrix calculation and the re-projection operation caused by the calculation are saved.
Finally, in order to ensure the real-time performance of the system, the upper limit of the computation time of the RANSAC algorithm needs to be defined (the embodiment specifies 50 milliseconds), so that the system cannot enter a dead loop to influence the acquisition and display of subsequent images under the condition that a scene is blocked or is not within the sight range.
After eliminating outliers in the feature point matching set using the above method, all remaining correct matching point sums are used
To find the initial position of the camera. Wherein n is the number of correct matching points, X is the position of the characteristic point in a world coordinate system, and X is the projection of the three-dimensional point on the image.
Meanwhile, the homography transformation and the reprojection technology are adopted to recover lost points in the tracking process so as to ensure that enough characteristic points are provided to complete the calculation of the pose of the camera.
For a certain feature point x on the reference image, the neighborhood is in the reference image IrWith the current picture IcThe transformation relationship between can be expressed as
Wherein
The homography transformation relationship between the current and reference images can be calculated by four pairs of corresponding points. x is the number ofi＝[xi yi 1]TIs the ith pixel in the neighborhood of the feature point x.
In order to recover the characteristic points which are moved out of the sight line range and lost in the tracking process, homography transformation is firstly calculated by using four adjacent points, and then the neighborhood of the lost point on the reference image is transformed to the current viewpoint. And then acquiring the general position of the lost point on the current image by utilizing a projection matrix P corresponding to the current camera, and searching the characteristic point which is most similar to the transformed neighborhood as the position of the lost point on the current image by taking the position as the center and 3 pixels as the radius by utilizing normalized cross correlation.
In the above process, if the normalized cross-correlation coefficient is smaller than a threshold value set in advance (defined as 0.6 in the present embodiment), it is considered that the point is blocked or out of the user's sight range and discarded. The following transformation is simultaneously performed on the pixels in the neighborhood of the image so as to avoid the negative influence of illumination on the recovery of the lost point to the maximum extent.
Wherein InormIs normalized pixel gray value, I is original pixel gray value, Iavg，ImaxAnd I andminrespectively the average, maximum and minimum of the gray levels of the pixels within the original neighborhood.
Fig. 3 is a diagram of the feature point matching and tracking results, as shown in fig. 3,
and seventhly, designing a system initialization algorithm based on two-stage feature matching according to the feature point matching result, solving the initial pose of the camera, and completing system initialization based on the initial pose state. The method specifically comprises the following steps:
firstly, a certain number of feature points with the highest matching degree on the current image and the identification image are selected, the number of the feature points is 20-40 pairs, and the feature points are utilized
Solving the pose parameter of the camera, wherein rho () is Tukey bilateral function, | | xj-ξ([R|T],Xj) And | is the reprojection error.
The calculation method of ρ () is as follows:
and c is the standard deviation of the reprojection error, and the point with the projection error exceeding the threshold value is taken as a mismatching point to be excluded, so that the method can effectively avoid the negative influence of mismatching on the calculation result while acquiring the camera parameters.
After the initial pose is obtained, the current image is scaled to the same scale as the identified image according to the translation ratio of the camera along the Z axis, and feature point extraction operation is performed. According to the initial pose, more matching points between the current image and the identified image can be obtained. In this groupOn the basis of the above, reuse
And calculating an external parameter once to serve as an initial camera pose to complete system initialization.
Eighthly, fig. 4 is a flow chart of a multithread camera real-time tracking registration work, and as shown in fig. 4, on the basis of the research content, the camera on-line tracking and scene reconstruction are divided into two independent threads to run on a dual-core machine, so that the real-time tracking and virtual information position registration of the camera in an unknown environment are completed. Wherein the camera tracking thread is used for completing pose calculation, key frame identification and reinitialization of the camera. The reconstruction thread is used for completing online reconstruction of the scene and online learning of the key frames.
In the scene reconstruction and registration processes, when a new key frame is added into the system, the local beam set adjustment is directly utilized to reconstruct a new feature point, and a reconstruction result is directly returned to a tracking thread to complete a camera tracking task. And the new keyframes will be used to train a random classification tree for camera re-initialization.
Fig. 5 is a performance comparison graph of the local bundle set adjustment method adopted in the scene reconstruction stage of the present invention and the conventional method, and as shown in fig. 5, the local bundle set adjustment method adopted in the present embodiment has higher registration accuracy compared to the conventional method.
When the "local beamset adjustment" is done and no new key frames are added to the system, the "global beamset adjustment" will be performed to minimize reconstruction errors. This process can be interrupted by the arrival of new key frames, which is done to allow new feature points to be added to the tracking system most quickly to ensure flexibility in camera tracking.
Fig. 6 is a graph of registration effect under different lighting conditions, as shown in fig. 6, the final display result of completing online registration in this embodiment under different lighting conditions.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (9)
1. A camera real-time tracking and registering method in an augmented reality system is characterized by comprising the following steps:
selecting two input images under an unknown scene, initializing the scene by using a European reconstruction method, designing a key frame online selection method, updating the key frame in real time, and realizing the reconstruction of the unknown scene based on local beam set adjustment;
expressing the key frame by taking the neighborhood of the feature point on the key frame as basic data, training a classifier by a random classification tree algorithm, and performing the learning and identification tasks of the key frame;
designing an SIFT feature point matching algorithm based on GPU acceleration, matching feature points, and completing on-line tracking of the feature points by an optical flow method after completing the feature point matching;
designing a system initialization algorithm based on two-stage feature matching according to the feature point matching result, calculating initial pose information of a camera by using the feature point matching result in one stage, performing feature point extraction and matching operation again after image scaling processing in the other stage, and finishing system initialization according to the initial pose information in the first stage;
dividing the on-line tracking and scene reconstruction of the camera into two independent threads to run on a dual-core machine, thereby completing the real-time tracking and virtual information position registration of the camera in an unknown environment;
the SIFT feature point matching algorithm is completed based on GPU acceleration, and comprises the following steps:
transforming a data structure in an SIFT feature point matching algorithm into a texture format which is convenient to process by a GPU, and respectively storing image pixel gray scale, Gaussian difference image, gradient vector and gradient direction by 4 color channels in the texture format;
storing feature point information by adopting a list structure, and separately storing feature points on different scale spaces;
for scale space image generation based on Gaussian filtering, two independent tasks of filtering in the horizontal direction and the vertical direction are divided to be executed in parallel, the gray value in the temporary register is written back to the original color channel, and the two independent filtering processes read and write the same texture space at the same time;
in the feature point extraction operation, the process of comparing the pixel gray level with the current layer and a plurality of adjacent pixels on the upper and lower layers of Gaussian difference images is divided into two independent processes of in-layer comparison and interlayer comparison;
the first process is to complete the comparison of adjacent pixels in the layer, and simultaneously store the maximum or minimum value into an auxiliary texture memory, and perform the calculation of the gradient vector and the gradient direction;
the second process is used to perform the inter-layer comparison.
2. The method for tracking and registering a camera in real time in an augmented reality system according to claim 1, wherein the method for selecting the key frame online comprises:
if one frame of input image meets the following two rules, the input image is taken as candidate key frame data;
a. presetting a camera pose calculation allowable threshold, wherein the pose calculation result of the camera is accurate to the allowable threshold;
b. presetting a key frame interval time threshold, wherein the input time interval between a new key frame and a previous key frame exceeds the time threshold;
after one frame of input image is selected as candidate key frame data according to rules a and b, if any one of the rules c and d is met, the frame of input image is taken as new key frame data and added into the system to complete the three-dimensional reconstruction of a new feature point;
c. taking a baseline distance between two keyframes used for scene initialization as a baseline distance threshold, wherein the baseline distance between a candidate keyframe and an existing nearest keyframe is less than the baseline distance threshold;
d. and presetting a rotation angle threshold value of the key frame image along any axis, wherein the rotation component between the candidate key frame and the existing nearest key frame along any coordinate axis is greater than the rotation angle threshold value.
3. The method as claimed in claim 2, wherein for the selection of the key frame, a new key frame image is added to the current system, and then a feature point matching set between the current key frame and the existing key frame closest to the current key frame is established directly by using epipolar constraint and normalized cross-correlation algorithm, and further a triangulation is used to calculate the three-dimensional position of the new feature point for the subsequent camera pose calculation process.
4. The method for tracking and registering a camera in real time in an augmented reality system according to claim 1, wherein the key frame is expressed by an image neighborhood around a feature point on the key frame, and basic data for training and learning of the image of the key frame is acquired, and the method comprises the following steps: given a set of keyframes { K ] acquired by online reconstructioniSuppose for each KiThe image neighborhood f corresponding to the jth feature point is obtainedi,jSet { f }i,jConstitute the basic data needed to train the classifier.
5. The method for tracking and registering a camera in real time in an augmented reality system according to claim 1, wherein after the basic data is acquired, the basic data is trained and learned by using a random classification tree method, and the tasks of on-line learning and identification of key frames are completed, including:
when a new key frame SiAfter adding to the system, a certain number of feature point neighborhoods { f) on the new keyframe are obtainedi,1,fi,2,...,fi,NPerforming positive conversion operation on all image neighborhoods according to the main direction of the corresponding SIFT descriptors, and performing smoothing operation on the image neighborhoods by adopting a Gaussian filter after the positive conversion operation;
testing each processed image neighborhood according to the internal nodes on each tree
To leaf nodes of each tree, where θiThe compensation value is used for compensating the influence of noise on the gray value difference between two pixels in the image neighborhood;
according to the formula
Replacing the posterior probability stored in the classification leaf node by the image neighborhood number, and directly setting the image neighborhood number corresponding to each leaf node to be 0 when deleting one type;
for the identification process, first, the image neighborhood { f ] around the feature point on the input image is1,f2,...,fNPerforming positive rotation and smoothing operations, further sequentially converting the converted image neighborhood images to leaf nodes of each tree, and finally returning the type with the maximum image neighborhood quantity and the following formula as an identification result to the tracking system to finish initialization;
6. the method as claimed in claim 5, wherein the real-time tracking and registering method is applied to θiAnd (4) completely randomly taking a value, wherein the value range is 0-z.
7. The method for real-time tracking and registering of a camera in an augmented reality system according to claim 1, wherein after completing feature point matching, the online tracking of the feature points is completed by an optical flow method, comprising:
first, adopt Td,dTesting and accelerating a 3-point RANSAC mismatching point elimination algorithm, randomly selecting a fourth pair of corresponding points while randomly selecting three pairs of corresponding points, wherein the fourth pair of corresponding points is firstly used for carrying out a reprojection test, and if a projection matrix generated by the random three points is effective to the fourth pair of corresponding points, carrying out reprojection operation on other corresponding points;
secondly, after three pairs of corresponding points are randomly extracted, judging whether the three points are approximately positioned on a straight line or whether the distance between any two points of the three points is smaller than a threshold value defined in advance, if the distance between any two points of the three points is smaller than the threshold value defined in advance, abandoning the current set and turning to the next sampling; defining the upper limit of the operation time of the RANSAC algorithm;
after excluding outliers in the feature point matching set, all remaining correct matching point sums are used
Obtaining the initial position of the camera, wherein n is the number of correct matching points, X is the position of the characteristic point in a world coordinate system, and X is the projection of the three-dimensional point on the image;
recovering lost points in the tracking process by adopting homography transformation and reprojection technology;
for a certain feature point x on the reference image, the neighborhood is in the reference image IrWith the current picture IcThe transformation relationship between is expressed as
Wherein
Calculating the homography transformation relation between the current image and the reference image through four pairs of corresponding points; x is the number ofi＝[xi yi 1]TIs the ith pixel in the neighborhood of the feature point x;
calculating homography transformation by using adjacent four points, further transforming the neighborhood of the lost point on the reference image to the current viewpoint, then obtaining the general position of the lost point on the current image by using a projection matrix P corresponding to the current camera, and searching the characteristic point which is most similar to the transformed neighborhood as the position of the lost point on the current image by using normalized cross correlation by taking the general position as the center and 3 pixels as the radius;
if the normalized cross-correlation coefficient is smaller than a preset threshold value, the point is considered to be blocked or out of the sight range of the user and is discarded.
8. The method for tracking and registering a camera in real time in an augmented reality system according to claim 1, wherein the method for obtaining the initial pose of the camera based on a two-stage feature matching system initialization algorithm and completing the system initialization based on the initial pose state comprises the following steps:
firstly, a certain number of feature points with the highest matching degree on the current image and the identification image are selected and utilized
Solving pose parameters of the camera, wherein rho () is a Tukey bilateral function,
is the reprojection error;
the calculation method of ρ () is as follows:
wherein c is the standard deviation of the reprojection error, and points with the projection error exceeding the threshold value are excluded as mismatching points;
after the initial pose is obtained, the current image is scaled to the same scale as the identification image according to the translation ratio of the camera along the Z axis, the feature point extraction operation is carried out, more matching points between the current image and the identification image are obtained according to the initial pose, and the matching points are reused
And calculating an external parameter once to serve as an initial camera pose to complete system initialization.
9. The method as claimed in claim 1, wherein the camera on-line tracking and scene reconstruction are divided into two independent threads to run on a dual-core machine, so as to complete the camera real-time tracking and virtual information position registration in an unknown environment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910334821.1A CN110059651B (en) | 2019-04-24 | 2019-04-24 | Real-time tracking and registering method for camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910334821.1A CN110059651B (en) | 2019-04-24 | 2019-04-24 | Real-time tracking and registering method for camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110059651A CN110059651A (en) | 2019-07-26 |
| CN110059651B true CN110059651B (en) | 2021-07-02 |
Family
ID=67320518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910334821.1A Active CN110059651B (en) | 2019-04-24 | 2019-04-24 | Real-time tracking and registering method for camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110059651B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114429495B (en) * | 2022-03-14 | 2022-08-30 | 荣耀终端有限公司 | Three-dimensional scene reconstruction method and electronic equipment |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030044048A1 (en) * | 2001-06-18 | 2003-03-06 | Zhengyou Zhang | Incremental motion estimation through local bundle adjustment |
| CN101287142A (en) * | 2008-05-16 | 2008-10-15 | 清华大学 | Method for converting flat video to tridimensional video based on bidirectional tracing and characteristic points correction |
| CN101763647A (en) * | 2010-02-02 | 2010-06-30 | 浙江大学 | Real-time video camera tracking method based on key frames |
| CN102201115A (en) * | 2011-04-07 | 2011-09-28 | 湖南天幕智能科技有限公司 | Real-time panoramic image stitching method of aerial videos shot by unmanned plane |
| CN102768767A (en) * | 2012-08-06 | 2012-11-07 | 中国科学院自动化研究所 | Online three-dimensional reconstructing and locating method for rigid body |
| CN103021017A (en) * | 2012-12-04 | 2013-04-03 | 上海交通大学 | Three-dimensional scene rebuilding method based on GPU acceleration |
| CN103247075A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Variational mechanism-based indoor scene three-dimensional reconstruction method |
| CN104881869A (en) * | 2015-05-15 | 2015-09-02 | 浙江大学 | Real time panorama tracing and splicing method for mobile platform |
| CN106688013A (en) * | 2014-09-19 | 2017-05-17 | 高通股份有限公司 | System and method of pose estimation |
| CN107301618A (en) * | 2017-06-21 | 2017-10-27 | 华中科技大学 | Based on the GPU basis matrixs accelerated parallel and homography matrix method of estimation and system |
| CN107369183A (en) * | 2017-07-17 | 2017-11-21 | 广东工业大学 | Towards the MAR Tracing Registration method and system based on figure optimization SLAM |
| CN107527366A (en) * | 2017-08-23 | 2017-12-29 | 上海视智电子科技有限公司 | A kind of camera tracking towards depth camera |
| WO2018169110A1 (en) * | 2017-03-17 | 2018-09-20 | 주식회사 언리얼파크 | Markerless augmented reality apparatus and method for expressing three-dimensional object |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103871063B (en) * | 2014-03-19 | 2017-04-19 | 中国科学院自动化研究所 | Image registration method based on point set matching |
| CN104392457B (en) * | 2014-12-11 | 2017-07-11 | 中国测绘科学研究院 | Incline the tie point automatic matching method and device of image |
| CN106203277B (en) * | 2016-06-28 | 2019-08-20 | 华南理工大学 | Fixed lens based on SIFT feature cluster monitor video feature extraction method in real time |
-
2019
- 2019-04-24 CN CN201910334821.1A patent/CN110059651B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030044048A1 (en) * | 2001-06-18 | 2003-03-06 | Zhengyou Zhang | Incremental motion estimation through local bundle adjustment |
| CN101287142A (en) * | 2008-05-16 | 2008-10-15 | 清华大学 | Method for converting flat video to tridimensional video based on bidirectional tracing and characteristic points correction |
| CN101763647A (en) * | 2010-02-02 | 2010-06-30 | 浙江大学 | Real-time video camera tracking method based on key frames |
| CN102201115A (en) * | 2011-04-07 | 2011-09-28 | 湖南天幕智能科技有限公司 | Real-time panoramic image stitching method of aerial videos shot by unmanned plane |
| CN102768767A (en) * | 2012-08-06 | 2012-11-07 | 中国科学院自动化研究所 | Online three-dimensional reconstructing and locating method for rigid body |
| CN103021017A (en) * | 2012-12-04 | 2013-04-03 | 上海交通大学 | Three-dimensional scene rebuilding method based on GPU acceleration |
| CN103247075A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Variational mechanism-based indoor scene three-dimensional reconstruction method |
| CN106688013A (en) * | 2014-09-19 | 2017-05-17 | 高通股份有限公司 | System and method of pose estimation |
| CN104881869A (en) * | 2015-05-15 | 2015-09-02 | 浙江大学 | Real time panorama tracing and splicing method for mobile platform |
| WO2018169110A1 (en) * | 2017-03-17 | 2018-09-20 | 주식회사 언리얼파크 | Markerless augmented reality apparatus and method for expressing three-dimensional object |
| CN107301618A (en) * | 2017-06-21 | 2017-10-27 | 华中科技大学 | Based on the GPU basis matrixs accelerated parallel and homography matrix method of estimation and system |
| CN107369183A (en) * | 2017-07-17 | 2017-11-21 | 广东工业大学 | Towards the MAR Tracing Registration method and system based on figure optimization SLAM |
| CN107527366A (en) * | 2017-08-23 | 2017-12-29 | 上海视智电子科技有限公司 | A kind of camera tracking towards depth camera |
Non-Patent Citations (2)
| Title |
|---|
| "基于GPU的SIFT特征提取算法研究";王瑞 等;《现代电子技术》;20101231(第15期);第41-44页 * |
| "未知环境重建及摄像机位姿估计算法研究";张栩;《中国优秀硕士学位论文全文数据库 信息科技辑》;20120215(第2期);正文第7-19、25-41页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110059651A (en) | 2019-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107481279B (en) | Monocular video depth map calculation method | |
| CN106570507B (en) | Multi-view-angle consistent plane detection and analysis method for monocular video scene three-dimensional structure | |
| CN107329962B (en) | Image retrieval database generation method, and method and device for enhancing reality | |
| CN111899334B (en) | Visual synchronous positioning and map building method and device based on point-line characteristics | |
| CN106991388B (en) | Key point positioning method | |
| CN110688947B (en) | Method for synchronously realizing human face three-dimensional point cloud feature point positioning and human face segmentation | |
| CN109974743B (en) | Visual odometer based on GMS feature matching and sliding window pose graph optimization | |
| CN111063021A (en) | Method and device for establishing three-dimensional reconstruction model of space moving target | |
| CN113298934B (en) | Monocular visual image three-dimensional reconstruction method and system based on bidirectional matching | |
| CN112750133A (en) | Computer vision training system and method for training a computer vision system | |
| CN113724379B (en) | Three-dimensional reconstruction method and device for fusing image and laser point cloud | |
| CN111797688A (en) | Visual SLAM method based on optical flow and semantic segmentation | |
| CN112001859A (en) | Method and system for repairing face image | |
| CN111709893A (en) | ORB-SLAM2 improved algorithm based on information entropy and sharpening adjustment | |
| CN116310098A (en) | Multi-view three-dimensional reconstruction method based on attention mechanism and variable convolution depth network | |
| CN113076891B (en) | Human body posture prediction method and system based on improved high-resolution network | |
| CN110059651B (en) | Real-time tracking and registering method for camera | |
| CN111402429B (en) | Scale reduction and three-dimensional reconstruction method, system, storage medium and equipment | |
| Long et al. | Detail preserving residual feature pyramid modules for optical flow | |
| CN110009683B (en) | Real-time on-plane object detection method based on MaskRCNN | |
| CN107341151B (en) | Image retrieval database generation method, and method and device for enhancing reality | |
| CN116128919A (en) | Multi-temporal image abnormal target detection method and system based on polar constraint | |
| CN115456870A (en) | Multi-image splicing method based on external parameter estimation | |
| CN114612545A (en) | Image analysis method and training method, device, equipment and medium of related model | |
| Kitt et al. | Trinocular optical flow estimation for intelligent vehicle applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |