CN101976464B - Multi-plane dynamic augmented reality registration method based on homography matrix - Google Patents
Multi-plane dynamic augmented reality registration method based on homography matrix Download PDFInfo
- Publication number
- CN101976464B CN101976464B CN 201010535231 CN201010535231A CN101976464B CN 101976464 B CN101976464 B CN 101976464B CN 201010535231 CN201010535231 CN 201010535231 CN 201010535231 A CN201010535231 A CN 201010535231A CN 101976464 B CN101976464 B CN 101976464B
- Authority
- CN
- China
- Prior art keywords
- homography matrix
- point
- real
- augmented reality
- registration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention discloses a multi-plane dynamic augmented reality registration method based on a homography matrix, which belongs to the field of computer augmented reality application, and mainly deals with the coordinate mapping relation between the real world and the virtual world. The method comprises the following steps: firstly, utilizing color information and shape information to automatically perform the initial coordinate mapping relation; then carrying out dynamic virtual environmental registration according to the real-time homography matrix calculation; and finally triggering relocation registration according to the shape information. The invention is characterized in that three-dimensional registration is realized by automatically recognizing the known positions in the real environment according to the special attribute of scene features, thus manual operation is not needed for realizing initialization, and automatic initialization is realized; and in terms of the problem that the homography matrix method can realize three-dimensional registration only on the plane of the physical world, the method of the invention utilizes the color information and the shape information to carry out multi-plane dynamic augmented reality registration.
Description
Technical field
The invention belongs to computing machine augmented reality and computer vision field, specifically the present invention relates to the dynamic augmented reality process registration in a kind of many planes based on homography matrix, is the basic technology that augmented reality is used.
Background technology
Augmented reality system has vast potential for future development, is subjected to people's attention day by day, and it has a wide range of applications in medical research, dissection training, exact instrument manufacturing, assembling and fields such as maintenance, military training, engineering design and tele-robotic.Owing to will realize virtual and real object perfect adaptation, dummy object must be merged in the real world position accurately, must detect the position of observer in scene, the angle of observation in real time in the augmented reality system, or even travel direction, so that be used for which kind of dummy object of help system decision demonstration, and rebuild coordinate system according to observer's visual field, promptly three-dimensional registration technology is the basis of augmented reality technology.Three-dimensional registration technology essence is exactly to calculate the mapping relations of the coordinate system of real world and virtual world, represents with matrix.And different matrixes has also determined the difference that three-dimensional registration technology method is chosen, and wherein matrix table is shown with homography matrix, fundamental matrix and essential matrix.
The homography matrix method is the simplest three-dimensional register method, because this method only need utilize 4 pairs of known points just can calculate the corresponding relation of the coordinate system of real world and virtual world, but the point that this method necessarily requires to choose is on the same plane and is the point of non-colinear, traditional like this homography matrix computing method can only be at the angular field of view of video camera in same plane or seek the situation of the special point on the same plane, for example use the angle point that detects on the gridiron pattern (referring to Z.Zhang, " A Flexible New Technique for Camera Calibration; " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, no.11,2000, pp.1330-1334).In addition, based on the calculating of homography matrix, for multilevel three-dimensional registration, main now employing identification shape and texture carry out multilevel identification and registration.Method for the identification shape, often when shape is by partial occlusion, understand registration failure (referring to http://www.hitl.washington.edu/artoolkit/), method for the identification texture, more complicated needs study (referring to Jacky Baltes.Camera Calibration Using Rectangular Textures.Springer Berlin Publish.2001) in advance.
Summary of the invention
The objective of the invention is to overcome the deficiencies in the prior art, provide a kind of many planes dynamic augmented reality process registration based on homography matrix, utilizing recognition feature point auto-initiation on the basis of colouring information, avoided general register method initialization to need manual shortcoming, utilize shape information to carry out reorientation simultaneously and realize multilevel augmented reality drafting, and utilize tracking technique real-time update homography matrix result of calculation, reach dynamic augmented reality effect.
The technical solution used in the present invention: the dynamic augmented reality process registration in a kind of many planes based on homography matrix, its characteristics are that step is as follows: at first carry out the detection and the identification of scene characteristic, the unique point of utilizing colouring information to identify is carried out the initialization of homography matrix and is calculated; According to the real-time calculating homography matrix of the tracking of unique point, utilize the result who calculates to judge the mapping relations of real world and virtual world, thereby carry out the drafting of dummy model then; Identification according to shape information at last comes trigger re-positioning.
The detection of described scene characteristic and the method for identification are as follows:
(1) color space conversion: each two field picture of storage is become hsv color space form by RGB color space format conversion, solve the variation of the color codomain that RGB color image storage format causes the susceptibility of illumination;
(2) feature point detection: the method for utilizing the Harris Corner Detection detects angle point feature in the image according to the first order difference of gray scale and filtering;
(3) unique point identification: for the feature angle point that detects, calculate color value in its neighborhood, and utilize sort algorithm to find out wherein specifically defined 4 maximum points of color value in the neighborhood, utilize this 4 points to calculate initial homography matrixes.
Described real-time calculating homography matrix method is as follows: utilize the conplane unique point of the real-time tracking of KLT tracking, and according to the result of calculation and the formula of initial homography matrix
Wherein
Be the i frame homography matrix result who calculates according to the 0th frame result, last real-time renewal homography matrix result, thereby the real-time coordinate Mapping relation that obtains real world and virtual world.
Described identification according to the shape information on the special sign plate comes the method for trigger re-positioning as follows: every two field picture all can detect the shape information that the present invention has defined, switch when detecting special shape information, the capital triggering system is carried out reorientation, promptly reinitialize, re-register, realize the coordinate Mapping relation of real world and virtual world.
The conplane unique point method of described real-time follow-up is as follows: in order to reduce calculated amount, adopt the method compressed picture of pyramidal compression, dwindle the hunting zone; At the unique point after following the tracks of owing to produce the mistake that unique point on error and the on-plane surface all can cause homography matrix thereafter to calculate in the motion, so adopt the RANSAC method to reject frontier point and Characteristics of Fault point.
The present invention's beneficial effect compared with prior art is:
(1) auto-initiation calculating is finished in detection and the identification that utilizes colouring information to carry out unique point, avoids manual initialization; Color space selects the HSV space to alleviate the susceptibility of color for illumination; At the influence of change of scale, adopt neighborhood to calculate and sort algorithm eliminating noise, eliminate the influence of change of scale at last.
(2) for dynamic augmented reality system, KLT track algorithm and RANSAC algorithm are used to real-time tracking and choose the real-time update that conplane unique point is carried out homography matrix, realize that with the mapping relations of the coordinate system that obtains real world and virtual world correct augmented reality draws effect.
(3) can only register on single plane for homography matrix, simple colouring information is used to differentiate different planes.And the color codomain may be because the influence of illumination change and cause recognition failures, and employing shape information (character shape among simple shape or the ARToo1kit) is further differentiated positional information, comes trigger re-positioning, carries out multilevel registration effect.
Description of drawings
Fig. 1 is an overall process synoptic diagram of the present invention;
Fig. 2 is a scale affects synoptic diagram of the present invention;
Fig. 3 is an identification plate synoptic diagram of the present invention;
Fig. 4, Fig. 5 plays up synoptic diagram for last drafting of the present invention.
Embodiment
Below in conjunction with accompanying drawing and embodiment the present invention is described in further detail:
The invention process process comprises three key steps: auto-initiation, real-time update homography matrix, reorientation.As shown in Figure 1.
Step 1 is an auto-initiation, mainly is divided into three phases:
The present invention is in order to overcome the problem of traditional three-dimensional register initialization, the mapping relations of the coordinate system that obtains initialized real world and virtual world have been, must in real world, find some features physical coordinates and with these selected corresponding image coordinate of feature, set up the initialized relation of hinting obliquely at of the coordinate system of real world and virtual world.General traditional way is the physical coordinates of some features in the self-defined real world, manual then on initialized two field picture, choose with real world in the corresponding point of feature chosen, obtaining its image coordinate, opening relationships, the initialization matrix that obtains like this is more accurate; The method that also has is that the image coordinate of utilizing shape or texture information to obtain some special features is carried out initialization, and initialization does not at this time need manually, but its shortcoming is also arranged, pre-service more complicated, and the feature out of true on the special texture.The present invention has taked to carry out auto-initiation with the information of color, and utilizes probability and sort algorithm to solve the problem of change of scale, and concrete step is as follows:
First stage: memory image
The present invention has used industrial camera to improve frame per second and resolution, after this industrial camera needs hardware initialization, with itself data structure storage is the IplImage data structure of opencv, and the color channel number is 3, and every two field picture is stored as the presentation format R (k) of RGB color space.
Second stage: converted image form
Color space is responsive for light application ratio, particularly traditional RGB color space.RGB image R (k) with storage is converted to hsv color spatial image V (k) thus.
Three phases: feature point detection identification
The image of the hsv color space representation after obtaining changing carries out the Harris Corner Detection to image.Single order differential transform according to gray scale detects the angle point in the image.Because angle point is the point that two dimensional image brightness changes curvature maximum value on violent point or the image border curve, promptly is the point of crossing in strong variations zone, so codomain is inaccurate under the color.The present invention takes to detect the color of the neighborhood point around the angle point, determines that a threshold value discerns needed initialized known point.
Determine that neighborhood is the core of looking for known point.Detected point under different yardsticks is different but dimensional variation has caused identical neighborhood, and then has caused a lot of noises occurring.As shown in Figure 2, be red color in the abcd square frame, getting neighborhood is 20 pixels, the neighborhood scope is approximately circle 1 when distance is for 1.2m; Under same vicinity, the neighborhood scope is approximately circle 2 when distance is 1m; This has just caused occurring removing a, b, and c outside the d point, also has a lot of noises.
Use method for normalizing to solve the problem of dimensional variation.Method for normalizing is exactly to determine total amount and component.Total amount is exactly the maximal value of sum that detects the point of designated color in all angle points in the neighborhood, and component is exactly the sum that detects the point of designated color in the angle point neighborhood.The maximal value of taking out the sum detect with sort algorithm is as total amount value, obtains value after the normalization with component value divided by total amount value then.What the present invention used is the mapping relations that homography matrix removes to calculate the coordinate system between real world and the virtual world, only needs 4 pairs of match points to come update calculation homography matrix result.4 points that identify are exactly preceding 4 points maximum after the normalization.As Fig. 2, shown in Figure 3, be respectively multi-form identification plate with designated color, can find out 4 points accurately.
Step 2: real-time update homography matrix.
Calculating homography matrix, in fact is exactly the process of asking for camera interior and exterior parameter.The video camera confidential reference items are about physical parameters such as focal length of camera, are to demarcate before the three-dimensional registration; The video camera external parameter mainly is rotation and translation variable, has mainly represented the mapping relations of real world coordinates system and virtual world coordinate system.In the process that video camera moves, external parameter is a real-time change, if obtain correct three-dimensional registration relation, obtains real actual situation and merges the drafting effect, and homography matrix just needs real-time renewal.
Utilize the 4 pairs of points that identify in the step 1, can calculate initial homography matrix, obtain the external parameter of video camera under the stationary state.Along with moving of video camera, known unique point may shift out outside the visual angle, and new unique point occurs.The present invention adopts the KLT tracking, follows the tracks of the unique point of initial frame, the result of renewal homography matrix that just can be real-time.But traditional KLT tracking has its defective, 1) be too fast when moving, tracking will be failed; 2) be the unique point that KLT just follows the tracks of previous frame, and the homography matrix method requires to use the point on the same plane could guarantee to calculate correct.At problem 1, the present invention has carried out error handling processing, when counting of tracing into is less than 20, just detects angle point again, serves as the point of following the tracks of former frame with present point.Because the unique point of tracking origin is exactly an angle point, though mobile camera moving comparatively fast causes following the tracks of failure, change can not differ greatly between two frames, still have common angle point, be rational so detect the angle point tracking again.At problem 2, use the incorrect point that the RANSAC algorithm is rejected the point on some on-plane surfaces and the method in 1 of dealing with problems obtains.
Step 3: reorientation is respectively two kinds of methods.
Though the logarithm of the match point that homography matrix needs is minimum, but to satisfy strict constraint, can only use the feature on the same plane exactly.When many planes actual situation merges drafting, using homography matrix to calculate will lead to the failure.At this problem, the present invention has used method for relocating, (special shape or the color that define is arranged) on this plane when another plane is shifted at the visual angle, will carry out reorientation.
First method: only use color.
Suppose to draw dummy object respectively on two different planes, the present invention can use two kinds of different colors, and is for example red and green.When the change color that occurs in the visual angle, will enter the reorientation module, the initialization of carrying out in the step 1 is demarcated, and starts different drafting schemes, carries out the drafting of actual situation fusion.
Second method: add the identification of shape.
Because the visual angle turns to different planes, may cause when camera angle changes greatly under the varying environment, lamplight brightness changes also under the big situation, for the color codomain, may cause bigger variation, what cause detecting is inaccurate, so the present invention has added the label detection module of ARToolkit on the basis of color attribute information.When different signs occurring, will the trigger re-positioning module, carry out the initialization homography matrix according to the 4 pairs of special match points that identify again and calculate, starts different drafting schemes, carry out the drafting of actual situation fusion.Fig. 4, the drafting effect that Fig. 5 merges for actual situation, Fig. 5 background are the signs that increases shape, are red in the middle big square frame.
Claims (4)
1. based on the dynamic augmented reality process registration in many planes of homography matrix, it is characterized in that:
Carry out the detection and the identification of scene characteristic, utilize the colouring information on the specific identification plate to discern defined feature initialization calculating homography matrix;
According to the real-time calculating homography matrix of physical feature point, utilize the result who calculates to judge the mapping relations of real world and virtual world, thereby carry out the drafting of dummy model;
Come trigger re-positioning according to identification to the different shape information on the special sign plate;
The detection of described scene characteristic specifically comprises with identification:
(1) color space conversion: each two field picture of storage is become hsv color space form by RGB color space format conversion, solve the variation of the color codomain that RGB color image storage format causes the susceptibility of illumination;
(2) feature point detection: the method for utilizing the Harris Corner Detection detects angle point feature in the image according to the first order difference of gray scale and filtering;
(3) unique point identification: for the angle point feature that detects, calculate color value in its neighborhood, and utilize sort algorithm to find out wherein specifically defined 4 maximum points of color value in the neighborhood, utilize this 4 points to calculate initial homography matrixes.
2. the dynamic augmented reality process registration in the many planes based on homography matrix according to claim 1 is characterized in that: described tracking according to physical feature point is calculated homography matrix in real time and is specifically comprised: ask for initialized homography matrix result according to 4 unique points asking for
Utilize the conplane unique point of KLT tracking real-time follow-up, and conplane unique point substitution formula
In, upgrade the homography matrix result in real time
Wherein
Be the i frame homography matrix result who calculates according to the 0th frame result, thereby obtain the coordinate Mapping relation of real world and virtual world in real time.
3. the dynamic augmented reality process registration in the many planes based on homography matrix according to claim 1, it is characterized in that: described identification according to the shape information on the special sign plate comes trigger re-positioning specifically to comprise: every two field picture all can detect color or shape, when detecting good special shape of predefined or colouring information, can triggering system carry out reorientation.
4. the dynamic augmented reality process registration in the many planes based on homography matrix according to claim 2, it is characterized in that: the conplane unique point of the described KLT of utilization method real-time follow-up specifically comprises: in order to reduce calculated amount, adopt pyramidal compression method compressed picture, dwindle the hunting zone; Adopt the RANSAC method to reject frontier point and Characteristics of Fault point, i.e. rejecting is not the unique point on the same plane and is not the unique point of participating in calculating in the previous frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010535231 CN101976464B (en) | 2010-11-03 | 2010-11-03 | Multi-plane dynamic augmented reality registration method based on homography matrix |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010535231 CN101976464B (en) | 2010-11-03 | 2010-11-03 | Multi-plane dynamic augmented reality registration method based on homography matrix |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101976464A CN101976464A (en) | 2011-02-16 |
| CN101976464B true CN101976464B (en) | 2013-07-31 |
Family
ID=43576346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010535231 Expired - Fee Related CN101976464B (en) | 2010-11-03 | 2010-11-03 | Multi-plane dynamic augmented reality registration method based on homography matrix |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101976464B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109740587A (en) * | 2018-12-22 | 2019-05-10 | 武汉新运维光电科技股份有限公司 | Transmission line of electricity augmented reality intelligent inspection system and method |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101569600B1 (en) * | 2011-06-08 | 2015-11-16 | 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 | Two-dimensional image capture for an augmented reality representation |
| CN102521859B (en) * | 2011-10-19 | 2014-11-05 | 中兴通讯股份有限公司 | Reality augmenting method and device on basis of artificial targets |
| CN103632374B (en) * | 2013-12-10 | 2017-04-26 | 深圳先进技术研究院 | Unmarked augmented reality system and target tracking method thereof |
| CN107248169B (en) * | 2016-03-29 | 2021-01-22 | 中兴通讯股份有限公司 | Image positioning method and device |
| CN107038758B (en) * | 2016-10-14 | 2020-07-17 | 北京联合大学 | Augmented reality three-dimensional registration method based on ORB operator |
| CN107340889B (en) * | 2017-06-30 | 2020-05-12 | 华勤通讯技术有限公司 | Positioning initialization method and device |
| CN108830936B (en) * | 2018-05-24 | 2022-07-05 | 讯飞幻境(北京)科技有限公司 | 3D model jitter prevention method and device |
| CN109782905A (en) * | 2018-12-27 | 2019-05-21 | 佛山科学技术学院 | A kind of augmented reality assembly bootstrap technique and system suitable for AGV steering wheel |
| CN109979015B (en) * | 2019-03-21 | 2019-12-13 | 苏州市职业大学 | AR reference object characteristic point correction method based on rigid body kinematics |
| WO2021217420A1 (en) * | 2020-04-28 | 2021-11-04 | 华为技术有限公司 | Lane tracking method and apparatus |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101101505A (en) * | 2006-07-07 | 2008-01-09 | 华为技术有限公司 | Method and system for implementing three-dimensional enhanced reality |
| CN101339654A (en) * | 2007-07-04 | 2009-01-07 | 北京威亚视讯科技有限公司 | Reinforced real environment three-dimensional registering method and system based on mark point |
| CN101442619A (en) * | 2008-12-25 | 2009-05-27 | 武汉大学 | Method for splicing non-control point image |
| CN101520904A (en) * | 2009-03-24 | 2009-09-02 | 上海水晶石信息技术有限公司 | Reality augmenting method with real environment estimation and reality augmenting system |
| CN101866496A (en) * | 2010-06-04 | 2010-10-20 | 西安电子科技大学 | Augmented reality method based on concentric ring pattern group |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100119109A1 (en) * | 2008-11-11 | 2010-05-13 | Electronics And Telecommunications Research Institute Of Daejeon | Multi-core multi-thread based kanade-lucas-tomasi feature tracking method and apparatus |
-
2010
- 2010-11-03 CN CN 201010535231 patent/CN101976464B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101101505A (en) * | 2006-07-07 | 2008-01-09 | 华为技术有限公司 | Method and system for implementing three-dimensional enhanced reality |
| CN101339654A (en) * | 2007-07-04 | 2009-01-07 | 北京威亚视讯科技有限公司 | Reinforced real environment three-dimensional registering method and system based on mark point |
| CN101442619A (en) * | 2008-12-25 | 2009-05-27 | 武汉大学 | Method for splicing non-control point image |
| CN101520904A (en) * | 2009-03-24 | 2009-09-02 | 上海水晶石信息技术有限公司 | Reality augmenting method with real environment estimation and reality augmenting system |
| CN101866496A (en) * | 2010-06-04 | 2010-10-20 | 西安电子科技大学 | Augmented reality method based on concentric ring pattern group |
Non-Patent Citations (1)
| Title |
|---|
| 管涛等.基于标识与平面间单应性矩阵的虚实注册方法.《华中科技大学学报（自然科学版）》.2006,第34卷(第10期), * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109740587A (en) * | 2018-12-22 | 2019-05-10 | 武汉新运维光电科技股份有限公司 | Transmission line of electricity augmented reality intelligent inspection system and method |
| CN109740587B (en) * | 2018-12-22 | 2022-05-10 | 武汉新运维光电科技股份有限公司 | Augmented reality intelligent inspection system and method for power transmission line |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101976464A (en) | 2011-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101976464B (en) | Multi-plane dynamic augmented reality registration method based on homography matrix | |
| Hu et al. | Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries | |
| US10192145B2 (en) | Method of providing image feature descriptors | |
| CN110580723B (en) | Method for carrying out accurate positioning by utilizing deep learning and computer vision | |
| CN101697233B (en) | Structured light-based three-dimensional object surface reconstruction method | |
| US20140270362A1 (en) | Fast edge-based object relocalization and detection using contextual filtering | |
| CN103839277A (en) | Mobile augmented reality registration method of outdoor wide-range natural scene | |
| CN104167003A (en) | Method for fast registering remote-sensing image | |
| CN104050475A (en) | Reality augmenting system and method based on image feature matching | |
| CN103854283A (en) | Mobile augmented reality tracking registration method based on online study | |
| CN101520849A (en) | Reality augmenting method and reality augmenting system based on image characteristic point extraction and random tree classification | |
| Mainali et al. | Robust low complexity corner detector | |
| Uchiyama et al. | Toward augmenting everything: Detecting and tracking geometrical features on planar objects | |
| CN104794748A (en) | Three-dimensional space map construction method based on Kinect vision technology | |
| CN104123554A (en) | SIFT image characteristic extraction method based on MMTD | |
| CN104217459A (en) | Spherical feature extraction method | |
| CN112033408A (en) | Paper-pasted object space positioning system and positioning method | |
| Donoser et al. | Robust planar target tracking and pose estimation from a single concavity | |
| Wang et al. | Accurate detection and localization of curved checkerboard-like marker based on quadratic form | |
| KR20070014767A (en) | Id judgment method by using extension type visual marker including direrction information | |
| CN109064536B (en) | Page three-dimensional reconstruction method based on binocular structured light | |
| CN113240656A (en) | Visual positioning method and related device and equipment | |
| JP2014199559A (en) | Viewpoint estimation device and sorter learning method therefor | |
| Uma et al. | Marker based augmented reality food menu | |
| CN107330436B (en) | Scale criterion-based panoramic image SIFT optimization method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130731 Termination date: 20151103 |
|
| EXPY | Termination of patent right or utility model |