CN106097260B - Structure-preserving augmented reality identification hiding method based on hollow identification - Google Patents
Structure-preserving augmented reality identification hiding method based on hollow identification Download PDFInfo
- Publication number
- CN106097260B CN106097260B CN201610382424.8A CN201610382424A CN106097260B CN 106097260 B CN106097260 B CN 106097260B CN 201610382424 A CN201610382424 A CN 201610382424A CN 106097260 B CN106097260 B CN 106097260B
- Authority
- CN
- China
- Prior art keywords
- repaired
- mark
- image
- mask
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 24
- 230000000694 effects Effects 0.000 claims abstract description 8
- 239000011800 void material Substances 0.000 claims description 7
- 238000001514 detection method Methods 0.000 claims description 5
- 230000002146 bilateral effect Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 3
- 238000007634 remodeling Methods 0.000 claims description 3
- 239000003550 marker Substances 0.000 abstract 2
- 238000005516 engineering process Methods 0.000 description 7
- 230000000903 blocking effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Processing Or Creating Images (AREA)
Abstract
A method for hiding a structure-preserving augmented reality identifier based on a hollowed-out identifier comprises the following steps: acquiring a video frame of a mark to be hidden from camera equipment; calculating the three-dimensional position and orientation of the camera relative to the marker according to the two-dimensional position of the marker; projecting the mask of the hollow area with the hollow mark on a plane where the mark is located, so as to determine a mask to be repaired in the video picture; according to the mask to be repaired, automatically detecting background structure information adjacent to the area to be repaired to obtain background structure characteristics; repairing the detected background structure according to the background structure characteristics to obtain a structure repairing image and a structure mask to be repaired; and repairing and hiding the non-structure identification by using the existing image repairing algorithm according to the structure repairing image and the mask to be repaired of the structure to obtain a final repairing effect image. The hollow mark reduces the area needing to be repaired after the mark is removed, can keep stronger structural property when the mark is hidden, and improves the efficiency and the effect of hiding the mark.
Description
Technical field
The present invention relates to a kind of hidden method based on augmented reality mark, especially a kind of guarantor's knot based on hollow-out mark
Structure augmented reality identifies hidden method.
Background technology
Augmented reality is a kind of new technology by growing up based on virtual reality, the void provided using computer system
Intend visual perception of the information enhancement user to real world, while utilizing related computer technology by the mark of calibration virtual
In model real-time rendering to the video pictures of real scene.In order to ensure merging between the dummy model and real scene that render
Consistency needs the inside and outside parameter that video camera is calculated using the position of mark and Camera Calibration Methods.
In view of cost of manufacture, ease for use and stability, mark becomes phase mostly important and general in augmented reality
Machine calibrates auxiliary tool.User makes mark according to relevant calibration pattern in papery, and the mark is placed into video camera
In the video pictures of shooting, you can detect inside and outside parameter of the video camera with respect to the mark.Camera Calibration based on hollow-out mark
Characteristic information of the method independent of environment is not limited by scene texture and partial occlusion interferes, can in real time, steadily
Video camera is calibrated, the extensive approval and use obtained.
Currently, the hollow-out mark in augmented reality still requires the mark of printing being entirely positioned in true environment, mark
Part real scene has been blocked in the placement of knowledge, affects the sense of reality and aesthetics of augmented reality picture, significantly limits this
Technology applications in various fields.To avoid influence of the appearance of mark to the augmented reality sense of reality, typically now there are two types of methods
Handle the mark problem in augmented reality.One is being rendered into mark using dummy model, marked region is covered in, but this is only
The situation bigger in mark volume suitable for dummy model.Another method is to acquire true field in advance before placing mark
Scape background video frame, to repair the mark occlusion area placed and generated after mark.Such a process increases the interaction behaviour of user
Make, reduces the practicability of augmented reality system.
Currently, what is identified in augmented reality hides, mainly utilizes and lost in image restoration technology reconstruction image or video
Part, using the picture material for not being identified blocking in video, to repair the region for being identified blocking, to reach hidden identification
Purpose.Block-based image repair method sets area to be repaired sub-block, according to son according to area to be repaired along boundary
The confidence level of block searches the sub-block with son multiblock best match to be repaired in the information of known image, carries out covering reparation.So
And traditional template class and occlusion area of the coding class mark in video pictures are usually bigger, using relatively good recently
Image repair method be identified it is hiding when, can not generally ensure the consistency with structural context information, cause to identify hidden
It hides result and the distortion phenomenons such as repairing mark and gap occurs.
Invention content
Picture of large image scale reparation is needed to avoid existing augmented reality from identifying hidden method, and can not ensure background structure
Consistency problem, the present invention devise it is a kind of based on hollow-out mark structure-preserving augmented reality mark hidden method.This method
The advantage for making full use of hollow-out mark shielded area small is extracted background structure information from the de-occlusion region inside mark, is adopted
The method hidden identification for repairing other regions again with structure is first repaired.It is as follows：
1) mark video frame I to be concealed, is obtained from camera apparatus；
2), according to the two-dimensional position of mark, three-dimensional position and direction of the video camera relative to the mark are calculated, i.e. camera is joined
Number；
3), in the plane by the void region mask projection of hollow-out mark to where identifying, so that it is determined that in video pictures
Mask Ma to be repaired；
4), according to mask Ma to be repaired, the automatic detection background structure information adjacent with area to be repaired obtains background knot
Structure feature S；
5), the background structure detected is repaired according to background structure feature S, obtains structure repair image Is and knot
Structure mask Ms to be repaired；
6) it, according to structure repair image Is and structure mask Ms to be repaired, is carried out using existing image repair algorithm non-
The reparation of structural identification is hidden, and final repairing effect image Ie is obtained.
The structure-preserving mark based on hollow-out mark hides the hiding comparison with existing traditional augmented reality mark, changes
Real background environment is blocked into being that the mark of hollow out is reduced, it can be according to the information extraction around hollow out information and mark
It is identified the background structure information blocked, the consistency of background structure can be kept when mark is hidden, it is hidden so as to improve identifying
The effect of Tibetan.Further, detecting the background structure information adjacent with area to be repaired in the step (4) automatically includes mainly
Following steps：
(4.1) it to mask Ma to be repaired, utilizesThe expansion of small range pixel, a are carried out along boundary
3 pixels, the shadow interference generated in removal mark imaging process are taken to obtain expanding and repair mask Md；
(4.2) to video frame images I, carry out bilateral filtering processing, obtain image In after denoising, wherein color parameter and
Spatial parameter takes 10；
(4.3) gradient is calculated to denoising image application Sobel operators, obtains gradient image Ig；
(4.4) according to gradient image Ig, judge to obtain the boundary of area to be repaired using 4 neighborhoods, further according to formulaThe boundary pixel of gradient magnitude local maximum is selected as potential structure feature
Point { Fi}；
(4.5) according to color tensor field method, a smooth tangential direction field is built using Sobel gradients, is cut
To field picture If；
(4.6) from potential structure characteristic point, a length is tracked out along the field of direction using Runge Kutta integration method
For 30 streamline, the sum of the gradient magnitude of this streamline present position is calculatedWork as GiWhen >=25, Fi
It is the real characteristic point for disclosing background structure；
(4.7) it utilizes structure feature point match method and the structural remodeling method of Hermite curves to calculate background structure curve, obtains
To background structure feature S, solid line indicates that background structure curve, dotted line indicate the corresponding road to be repaired obtained by curve matching
Diameter.
Further, the reparation of the structure curve in the path to be repaired in the step (5) mainly includes the following steps that：
(5.1) using a as the length of side, it is not identified the texture blockage blocked along the acquisition of background structure curve, establishes line
Manage block data source { Sj}；
(5.2) from path both ends to be repaired, by { SjIn most matched texture block repaired in subregion T with current
Known pixels calculate argminj SSD(T,Sj) best matching blocks are obtained, then cover on the corresponding position in area to be repaired；
(5.3) above-mentioned area to be repaired block is returned into step (5.1), to what is not yet repaired labeled as restoring area
Structure lines are repaired, until all structure lines regions are repaired.
Further, the reparation in non-structural region is mainly included the following steps that in the step (6)：
(6.1) according to the image after structure repair, along a series of 9*9 rectangles picture of the boundary demarcation of area to be repaired
Plain block, while the serial number of calibrating block；
(6.2) it according to the numeric order of block, is matched with the block in known image, matching degree is highest to be filled into this
Region repeats this step labeled as restoring area, until all labeled serial number blocks are repaired；
(6.3) boundary for updating area to be repaired, returns to step (6.1), is repaiied to the boundary serial number re-flagged
It is multiple, until being repaired for all areas.
The present invention technical concept be：Based on the augmented reality of legacy identification when mark is hidden, it can not ensure background knot
The consistency of structure so that phenomena such as mark generates the distortion of some structural informations after hiding.Relative to legacy identification, hollow-out mark
Advantage be that identifying the internal visible void region of several backgrounds tends to provide certain background structure characteristic information.
The void region of hollow-out mark has preferable visibility to the detection of background structure, can be with the mark of structure-preserving when mark is hidden
Know and hide, obtains relatively good visual effect.
The advantage of the invention is that：The mark of hollow out, which reduces, blocks background environment, can reduce the face of image repair
Product reduces the hiding error of mark；The visibility of mark void region can provide more features for the detection of background structure
Information；When augmented reality based on hollow-out mark is identified hiding, the structural information of background can be preferably kept；Based on engraving
The augmented reality of sky mark is hidden relative to legacy identification, and faster, effect is more preferable for speed.
Description of the drawings
Fig. 1 is the flow chart of the present invention
Fig. 2 is background structure overhaul flow chart
Specific implementation mode
With reference to attached drawing, further illustrate the present invention：
A kind of structure-preserving augmented reality mark hidden method based on hollow-out mark, includes the following steps：
1) mark video frame I to be concealed, is obtained from camera apparatus；
2), according to the two-dimensional position of mark, three-dimensional position and direction of the video camera relative to the mark are calculated, i.e. camera is joined
Number；
3), in the plane by the void region mask projection of hollow-out mark to where identifying, so that it is determined that in video pictures
Mask Ma to be repaired；
4), according to mask Ma to be repaired, the automatic detection background structure information adjacent with area to be repaired obtains background knot
Structure feature S；
5), the background structure detected is repaired according to background structure feature S, obtains structure repair image Is and knot
Structure mask Ms to be repaired；
6) it, according to structure repair image Is and structure mask Ms to be repaired, is carried out using existing image repair algorithm non-
The reparation of structural identification is hidden, and final repairing effect image Ie is obtained.
The structure-preserving mark based on hollow-out mark hides the hiding comparison with existing traditional augmented reality mark, changes
Real background environment is blocked into being that the mark of hollow out is reduced, it can be according to the information extraction around hollow out information and mark
It is identified the background structure information blocked, the consistency of background structure can be kept when mark is hidden, it is hidden so as to improve identifying
The effect of Tibetan.Further, detecting the background structure information adjacent with area to be repaired in the step (4) automatically includes mainly
Following steps：
(4.1) it to mask Ma to be repaired, utilizesThe expansion of small range pixel, a are carried out along boundary
3 pixels, the shadow interference generated in removal mark imaging process are taken to obtain expanding and repair mask Md；
(4.2) to video frame images I, carry out bilateral filtering processing, obtain image In after denoising, wherein color parameter and
Spatial parameter takes 10；
(4.3) gradient is calculated to denoising image application Sobel operators, obtains gradient image Ig；
(4.4) according to gradient image Ig, judge to obtain the boundary of area to be repaired using 4 neighborhoods, further according to formulaThe boundary pixel of gradient magnitude local maximum is selected as potential structure feature
Point { Fi}；
(4.5) according to color tensor field method, a smooth tangential direction field is built using Sobel gradients, is cut
To field picture If；
(4.6) from potential structure characteristic point, a length is tracked out along the field of direction using Runge Kutta integration method
For 30 streamline, the sum of the gradient magnitude of this streamline present position is calculatedWork as GiWhen >=25, Fi
It is the real characteristic point for disclosing background structure；
(4.7) it utilizes structure feature point match method and the structural remodeling method of Hermite curves to calculate background structure curve, obtains
To background structure feature S, solid line indicates that background structure curve, dotted line indicate the corresponding road to be repaired obtained by curve matching
Diameter.
Further, the reparation of the structure curve in the path to be repaired in the step (5) mainly includes the following steps that：
(5.1) using a as the length of side, it is not identified the texture blockage blocked along the acquisition of background structure curve, establishes line
Manage block data source { Sj}；
(5.2) from path both ends to be repaired, by { SjIn most matched texture block repaired in subregion T with current
Known pixels calculate arg minj SSD(T,Sj) best matching blocks are obtained, then cover on the corresponding position in area to be repaired；
(5.3) above-mentioned area to be repaired block is returned into step (5.1), to what is not yet repaired labeled as restoring area
Structure lines are repaired, until all structure lines regions are repaired.
Further, the reparation in non-structural region is mainly included the following steps that in the step (6)：
(6.1) according to the image after structure repair, along a series of 9*9 rectangles picture of the boundary demarcation of area to be repaired
Plain block, while the serial number of calibrating block；
(6.2) it according to the numeric order of block, is matched with the block in known image, matching degree is highest to be filled into this
Region repeats this step labeled as restoring area, until all labeled serial number blocks are repaired；
(6.3) boundary for updating area to be repaired, returns to step (6.1), is repaiied to the boundary serial number re-flagged
It is multiple, until being repaired for all areas.
Currently, in augmented reality field, the camera Calibration method based on legacy identification is increasingly ripe, and that calibrates is steady
Qualitative and computational efficiency is obtained for being widely recognized as academia and industrial quarters.The present invention is directed to the placement identified in such methods
Deficiency to the sense of reality and aesthetics that enhance picture, it is proposed that a kind of structure-preserving augmented reality mark based on hollow-out mark is hidden
Tibetan method is real-time hidden identification in video, obtains preferable visual effect and provide technical foundation.
Content described in this specification embodiment is only enumerating to the way of realization of inventive concept, protection of the invention
Range is not construed as being only limitted to the concrete form that embodiment is stated, protection scope of the present invention is also and in art technology
Personnel according to present inventive concept it is conceivable that equivalent technologies mean.
Claims (1)
1. a kind of structure-preserving augmented reality based on hollow-out mark identifies hidden method, include the following steps：
1) mark video frame I to be concealed, is obtained from camera apparatus；
2), according to the two-dimensional position of mark, three-dimensional position and direction of the video camera relative to the mark, i.e. camera parameter are calculated；
3), in the plane by the void region mask projection of hollow-out mark to where identifying, so that it is determined that being waited in video pictures
Repair mask Ma；
4), according to mask Ma to be repaired, it is special to obtain background structure for the automatic detection background structure information adjacent with area to be repaired
Levy S；
(4.1) it to mask Ma to be repaired, utilizesThe expansion of small range pixel is carried out along boundary, a takes 3
Pixel, removal identify the shadow interference generated in imaging process, obtain expanding and repair mask Md；
(4.2) to video frame images I, bilateral filtering processing is carried out, obtains image In after denoising, wherein color parameter and space
Parameter takes 10；
(4.3) gradient is calculated to denoising image application Sobel operators, obtains gradient image Ig；
(4.4) according to gradient image Ig, judge to obtain the boundary of area to be repaired using 4 neighborhoods, further according to formulaThe boundary pixel of gradient magnitude local maximum is selected as potential structure feature
Point { Fi}；
(4.5) according to color tensor field method, a smooth tangential direction field is built using Sobel gradients, obtains tangential field
Image If；
(4.6) from potential structure characteristic point, it is 30 to track out a length along the field of direction using Runge Kutta integration method
Streamline, calculate the sum of the gradient magnitude of this streamline present positionWork as GiWhen >=25, FiIt is only true
The positive characteristic point for disclosing background structure；
(4.7) it utilizes structure feature point match method and the structural remodeling method of Hermite curves to calculate background structure curve, is carried on the back
Scape structure feature S, solid line indicate that background structure curve, dotted line indicate the corresponding path to be repaired obtained by curve matching；
5), the background structure detected is repaired according to background structure feature S, structure repair image Is is obtained and structure waits for
Repair mask Ms；
(5.1) using a as the length of side, it is not identified the texture blockage blocked along the acquisition of background structure curve, establishes texture block
Data source { Sj}；
(5.2) from path both ends to be repaired, by { SjIn most matched texture block with it is known in current area to be repaired T
Pixel calculates argminjSSD(T,Sj) best matching blocks are obtained, then cover on the corresponding position in area to be repaired；
(5.3) above-mentioned area to be repaired is returned into step (5.1) labeled as restoring area, it is to be repaired to what is not yet repaired
Region is repaired, until all areas to be repaired are repaired；
6) it, according to structure repair image Is and structure mask Ms to be repaired, is carried out using existing image repair algorithm non-structural
The reparation of mark is hidden, and final repairing effect image Ie is obtained；
(6.1) according to the image after structure repair, along a series of 9*9 rectangular pixels of the boundary demarcation of area to be repaired
Block, while the serial number of calibrating block；
(6.2) it according to the numeric order of block, is matched with the block in known image, matching degree is highest to be filled into the pixel
Block repeats this step labeled as restoring area, until all labeled boundary serial number blocks are repaired；
(6.3) boundary for updating area to be repaired, returns to step (6.1), is repaired to the boundary serial number block re-flagged,
Until the boundary of all areas to be repaired is repaired.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610382424.8A CN106097260B (en) | 2016-06-01 | 2016-06-01 | Structure-preserving augmented reality identification hiding method based on hollow identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610382424.8A CN106097260B (en) | 2016-06-01 | 2016-06-01 | Structure-preserving augmented reality identification hiding method based on hollow identification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106097260A CN106097260A (en) | 2016-11-09 |
| CN106097260B true CN106097260B (en) | 2018-10-23 |
Family
ID=57447862
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610382424.8A Active CN106097260B (en) | 2016-06-01 | 2016-06-01 | Structure-preserving augmented reality identification hiding method based on hollow identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106097260B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108491843B (en) * | 2018-04-12 | 2022-11-18 | 腾讯科技（深圳）有限公司 | Image processing method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102122384A (en) * | 2011-01-25 | 2011-07-13 | 机密科技公司 | Method for manufacturing hidden pattern during plate making for presswork |
| CN103295047A (en) * | 2013-06-25 | 2013-09-11 | 谢婧 | Image identifier capable of obtaining hidden information and manufacturing and reading method thereof |
| CN103473746A (en) * | 2013-09-16 | 2013-12-25 | 浙江工业大学 | Real-time removing method for augmented reality calibration plate |
| CN105321190A (en) * | 2015-10-28 | 2016-02-10 | 上海大学 | Moving object detection method based on structurally similar background modeling |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100542370B1 (en) * | 2004-07-30 | 2006-01-11 | 한양대학교 산학협력단 | Vision-based augmented reality system using invisible marker |
-
2016
- 2016-06-01 CN CN201610382424.8A patent/CN106097260B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102122384A (en) * | 2011-01-25 | 2011-07-13 | 机密科技公司 | Method for manufacturing hidden pattern during plate making for presswork |
| CN103295047A (en) * | 2013-06-25 | 2013-09-11 | 谢婧 | Image identifier capable of obtaining hidden information and manufacturing and reading method thereof |
| CN103473746A (en) * | 2013-09-16 | 2013-12-25 | 浙江工业大学 | Real-time removing method for augmented reality calibration plate |
| CN105321190A (en) * | 2015-10-28 | 2016-02-10 | 上海大学 | Moving object detection method based on structurally similar background modeling |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106097260A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107292965B (en) | Virtual and real shielding processing method based on depth image data stream | |
| CN109544456B (en) | Panoramic environment sensing method based on two-dimensional image and three-dimensional point cloud data fusion | |
| CN103679749B (en) | A kind of image processing method and device based on motion target tracking | |
| KR101758058B1 (en) | Apparatus and method for estimating camera motion using depth information, augmented reality system | |
| CN109462747B (en) | DIBR system cavity filling method based on generation countermeasure network | |
| WO2021018894A1 (en) | Method and computer program product for producing 3-dimensional model data of a garment | |
| CN107240073B (en) | Three-dimensional video image restoration method based on gradient fusion and clustering | |
| US20170309075A1 (en) | Image to item mapping | |
| CN106778730A (en) | A kind of adaptive approach and system for quickly generating OCR training samples | |
| JP2005293141A (en) | Method and device for detecting marker, and method for estimating position and attitude | |
| CN107622480B (en) | Kinect depth image enhancement method | |
| CN112258658A (en) | Augmented reality visualization method based on depth camera and application | |
| WO2017004882A1 (en) | Video 3d rendering method for flat display apparatuses | |
| JP2009081853A (en) | Imaging system and method | |
| CN101996407A (en) | Colour calibration method for multiple cameras | |
| CN110941996A (en) | Target and track augmented reality method and system based on generation of countermeasure network | |
| CN110390657B (en) | Image fusion method | |
| JP2015171143A (en) | Camera calibration method and apparatus using color-coded structure, and computer readable storage medium | |
| CN106682652A (en) | Structure surface disease inspection and analysis method based on augmented reality | |
| CN108629742A (en) | True orthophoto shadow Detection and compensation method, device and storage medium | |
| CN106097260B (en) | Structure-preserving augmented reality identification hiding method based on hollow identification | |
| Inzerillo et al. | High quality texture mapping process aimed at the optimization of 3d structured light models | |
| CN103473746B (en) | The real-time removing method of calibration plate in augmented reality | |
| JP5295044B2 (en) | Method and program for extracting mask image and method and program for constructing voxel data | |
| CN109840951A (en) | The method and device of augmented reality is carried out for plane map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |