KR20130019485A - Augmented reality system based on marker and object augment method thereof - Google Patents
Augmented reality system based on marker and object augment method thereof Download PDFInfo
- Publication number
- KR20130019485A KR20130019485A KR1020110081456A KR20110081456A KR20130019485A KR 20130019485 A KR20130019485 A KR 20130019485A KR 1020110081456 A KR1020110081456 A KR 1020110081456A KR 20110081456 A KR20110081456 A KR 20110081456A KR 20130019485 A KR20130019485 A KR 20130019485A
- Authority
- KR
- South Korea
- Prior art keywords
- marker
- information
- detected
- augmented reality
- reality system
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Processing Or Creating Images (AREA)
Abstract
The present invention relates to an augmented reality system and an object augmentation method based on the marker. An augmented reality system based on a marker according to the present invention includes an image pickup unit for photographing an external object; An image processing unit which performs image processing for converting the captured image into an image displayable on a display unit; A marker information detector for detecting information of a marker included in each frame constituting the converted image; A marker information storage unit for storing information of the marker; When the information of the marker is detected, the information of the detected marker is stored in the marker information storage unit, and the object is augmented. If the marker is not detected, the feature point of the marker is traced according to the information of the stored marker. It characterized in that it comprises an object reinforcing unit for augmenting the object. As a result, the present invention can continuously augment an object even when the marker is temporarily hidden in the acquired image, and the augmented reality can be interacted with by using a masking phenomenon of the marker.
Description
The present invention relates to an augmented reality system based on a marker, and more particularly, to an augmented reality system based on a marker capable of augmenting an object even when the marker is temporarily hidden and an object augmentation method thereof. .
In general, Augmented Reality (AR) is a modified form of Virtual Reality (VR), and refers to a user interface technology that matches a virtual image generated by a computer with an image of a real user viewed by the user.
The methods for implementing such augmented reality are largely divided into a marker-based implementation method and a markerless method implemented using feature points collected from an actual object, and augmented based on a marker to facilitate recognition of an image. There are many ways to implement reality.
1 is a flowchart illustrating a method of augmenting an object in a conventional augmented reality system based on a marker.
As shown in FIG. 1, the conventional augmented reality system acquires an image captured by an imaging unit (S1), and determines whether a marker is detected every frame of the acquired image (S2). Thus, when the marker is detected, information of the marker is detected (S3), and then a virtual object is represented (S4). If the marker is not detected, the process of acquiring the image again is repeated.
However, in the conventional augmented reality system, since the virtual image can be matched to the object only when all the markers are completely recognized, there is a problem that the object to be augmented disappears even if a part of the marker is hidden.
Accordingly, an object of the present invention is to provide an augmented reality system and an object augmentation method based on a marker capable of continuously augmenting an object even when the marker is temporarily hidden in the acquired image.
In addition, an object of the present invention is to provide an augmented reality system and an object augmentation method based on a marker capable of interacting with augmented reality using the occlusion of the marker.
However, the objects of the present invention are not limited to those mentioned above, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.
In order to achieve the above objects, an augmented reality system based on a marker according to an aspect of the present invention includes an image pickup unit for imaging an external object; An image processing unit which performs image processing for converting the captured image into an image displayable on a display unit; A marker information detector for detecting information of a marker included in each frame constituting the converted image; A marker information storage unit for storing information of the marker; When the information of the marker is detected, the information of the detected marker is stored in the marker information storage unit, and the object is augmented. If the marker is not detected, the feature point of the marker is traced according to the information of the stored marker. It may include an object enhancer for augmenting the object.
Preferably, the object reinforcement unit may perform initialization to track a feature point of the marker when information of the marker is not detected.
Preferably, the object reinforcement unit may terminate the tracking of the feature point of the marker when the initialization for tracking the feature point of the marker is not completed.
Preferably, the information of the marker may include geometric feature points of the marker and vertex information of the marker.
Preferably, when the marker is not detected, the object enhancer may augment the object by tracking a feature point of the marker according to the information of the marker detected in the previous frame and stored in the marker information storage.
Preferably, when the information of the marker is not detected, when the object enhancer acquires all coordinates of the vertices of the markers determined using the stored information of the markers, the coordinates of the vertices of the markers and the previously stored The feature point of the marker may be tracked by comparing the information about the vertices of the marker to determine the marker.
Preferably, when the information of the marker is not detected, when the object enhancer acquires a part of the coordinates of the vertex of the marker determined by using the stored information of the marker, the object reinforcement unit does not acquire using the known size of the marker. The feature point of the marker may be tracked by calculating coordinates of the failed vertex and determining the marker by comparing the calculated coordinates of the vertex of the marker with information about the stored vertices of the marker.
Preferably, the object enhancer predicts a vertex of the moved marker by applying information about a vertex of the marker previously stored in the optical flow algorithm when the marker is not detected and the information of the marker is not detected. The feature point of the marker may be tracked by comparing the predicted coordinate of the vertex of the marker with information about previously stored vertices of the marker to determine the marker.
Object augmentation method of the augmented reality system based on a marker according to another aspect of the present invention comprises the steps of imaging the external object; Converting the captured image into an image displayable on a display unit; Detecting information of a marker included in each frame constituting the image; And reinforcing the object after storing the information of the detected marker when the information of the marker is detected, and reinforcing the object by tracking a feature point of the marker according to the information of the stored marker when the marker is not detected. It may include a step.
Preferably, the step of augmenting the object may further include performing initialization to track a feature point of the marker when information of the marker is not detected.
Preferably, the performing of the initialization may terminate the tracking of the feature point of the marker when the initialization for tracking the feature point of the marker is not completed.
Preferably, the information of the marker may include geometric feature points of the marker and vertex information of the marker.
Preferably, in the step of augmenting the object, when the marker is not detected, the object may be augmented by tracking feature points of the marker according to the information of the marker detected and stored in the previous frame.
Preferably, in the step of augmenting the object, when the information of the marker is not detected, when all the coordinates of the vertex of the marker determined using the previously stored information of the marker are acquired, A feature point of the marker may be tracked by comparing the stored information about the vertices of the marker to determine the marker.
Preferably, in the step of augmenting the object, when the information of the marker is not detected, when a part of the coordinates of the vertex of the marker determined by using the previously stored information of the marker is obtained, the already known size of the marker is used. The feature point of the marker may be tracked by calculating a coordinate of a vertex that has not been acquired, and comparing the calculated coordinate of the vertex of the marker with information about the stored vertex of the marker to determine the marker.
Preferably, the step of augmenting the object predicts a vertex of the moved marker by applying information about a vertex of the marker previously stored in the optical flow algorithm when the marker is not moved to detect the information of the marker. The feature point of the marker may be tracked by comparing the predicted coordinates of the marker with information on the previously stored vertices of the marker to determine the marker.
According to the augmented reality system and the object augmentation method based on the marker according to the present invention, the object can be continuously augmented even when the marker is temporarily hidden in the acquired image.
In addition, according to the augmented reality system and the object augmentation method based on the marker according to the present invention, it is possible to interact with augmented reality using the occlusion of the marker.
1 is a flowchart illustrating a method of augmenting an object in a conventional augmented reality system based on a marker,
Figure 2 is a block diagram showing the configuration of augmented reality system based on a marker according to the present invention,
3 is a view showing the feature points of the marker used in the augmented reality system based on the marker according to the present invention,
4 is an exemplary view illustrating various states in which information of a marker is hidden;
5 is a first exemplary view for explaining a principle of tracking feature points of a marker according to the present invention;
6 is a second exemplary view for explaining the principle of tracking feature points of a marker according to the present invention;
7 is a flowchart illustrating an object augmentation method of an augmented reality system based on a marker according to the present invention;
8 is a flowchart illustrating a method of storing a marker in an augmented reality system based on a marker according to the present invention;
9 is a flowchart illustrating a method for tracking feature points of a marker in an AR system based on the marker according to the present invention.
Hereinafter, with reference to Figures 2 to 9 attached to the augmented reality system and the object augmentation method based on the marker according to an embodiment of the present invention. The present invention will be described in detail with reference to the portions necessary for understanding the operation and operation according to the present invention. Like reference numerals in the drawings denote like elements throughout the specification.
Figure 2 is a block diagram showing the configuration of augmented reality system based on a marker according to the present invention.
As shown in FIG. 2, the augmented reality system 100 according to the present invention includes an
The
The
The
For example, when the marker is formed in a rectangle and has an ID (ID) such as text, numbers, and symbols inside the rectangle, the
The marker
Here, when the information on the marker is detected by the
3 is a view showing the feature points of the marker used in the augmented reality system 100 based on the marker according to the present invention.
As shown in FIG. 3, for example, when the marker is implemented in a square shape, the marker's feature points are the outermost four vertices A, B, C, and D, and of course, as in the midpoint of each corner of the outer rectangle. Geometric feature points and the like may also be stored in the
When the information of the marker is detected, the
Of course, when the information of the marker is not detected to track the feature point of the marker, the principle of tracking the feature point of the marker will be described with reference to FIGS.
4 is an exemplary diagram illustrating various states in which information of a marker is hidden.
As shown in FIG. 4, when a marker that is continuously augmented is covered, it is not possible to determine whether it is a marker or not, and the marker is hidden. In other words, the marker cannot be recognized because the rectangle cannot be found.
5 is a first exemplary view for explaining the principle of tracking feature points of a marker according to the present invention.
As shown in FIG. 5, the marker is hidden while the marker is not moving. If a part of the marker is hidden as shown in Fig. (A), but the coordinates a1, b1, c1, and d1 of all four vertices of the rectangle are known, the markers can be determined by comparing them with information about the vertices of the previously stored markers. It becomes possible.
As shown in (b), a part of the marker is hidden and only the coordinates a2 and c2 of two vertices are known, and the size of the marker can be known from the information about the vertices of the previously stored markers. The coordinates b2 and d2 of two vertices can be calculated. The markers can be determined when the coordinates of the four vertices thus obtained are compared with the information on the vertices of the previously stored markers.
The coordinates of the four vertices thus obtained may be updated in the marker
6 is a second exemplary view for explaining the principle of tracking feature points of a marker according to the present invention.
As shown in FIG. 6, when a part of the marker is hidden but the marker moves, information about a vertex of the previously stored marker may be applied to an optical flow algorithm to predict a vertex of the marker in a moved state.
Here, the optical flow algorithm is known as the most useful algorithm for tracking the motion of an object by deriving a change in velocity field from successive image frames.
On the other hand, when the
The
Hereinafter, an object augmentation method of an augmented reality system based on a marker according to the present invention will be described with reference to FIG. 7.
As shown in FIG. 7, the augmented reality system may first photograph an external object by the imaging unit 10 (S710). Next, the augmented reality system may convert the captured image into an image that can be displayed on the display unit 60 (S720).
The augmented reality system may detect information of a marker included in each frame constituting the image (S730). That is, the augmented reality system finds a rectangular candidate group within each frame and recognizes it as a marker candidate group, compares the ID inside the marker candidate group with a previously stored ID, and determines whether the marker is correct according to the comparison result.
Next, when the information of the marker is detected in step S730, the augmented reality system may store or update the information of the detected marker (S741).
In contrast, when the marker is not detected in step S430, the augmented reality system may track a feature point of the marker according to the information of the marker detected and stored in the previous frame stored in the marker information storage 40 (S742).
On the other hand, when the process is first performed, if the marker information is not stored in step S741, the step S710 of capturing the external object until the marker information is detected and stored and the step S720 of converting the captured image should be performed. When the marker information is stored through this step, step S742 of tracking the feature point according to the stored marker information is performed.
Finally, the augmented reality system may augment the object according to the information of the marker (S750). As a result, the present invention can augment an object according to the movement of the marker by tracking a feature point of the marker even when a part of information of the marker is not detected.
Hereinafter, a method of storing a marker in an augmented reality system based on a marker according to the present invention will be described with reference to FIG. 8.
As shown in FIG. 8, first, when the marker is detected by the
Next, the AR system may store or update the feature point information of the detected marker in the marker information storage 40 (S820). The augmented reality system may perform initialization for tracking the feature points of the marker, that is, confirm whether or not the feature point information of the marker is stored or updated (S830). In other words, it is checked whether the feature point information of the marker extracted in the previous frame is stored.
Finally, the augmented reality system may augment the object by matching a virtual image generated by a computer with the object using the stored feature point information of the marker (S840).
9 is a flowchart illustrating a method for tracking feature points of a marker when a marker is not detected in the augmented reality system based on the marker according to the present invention.
As shown in FIG. 9, first, the AR system performs initialization to track a feature point of the marker when the marker is not detected in each frame of the image captured by the
Next, when the initialization for tracking the feature point of the marker is successfully performed, the augmented reality system may load the feature point information of the marker from the marker
In detail, the augmented reality system loads feature point information of a pre-stored marker when the marker does not move. 1) If the coordinates of the vertices of the marker determined by using the feature point information of the marker can be known, the pre-stored The marker's feature points are tracked by comparing the marker's vertices with information on the marker's vertices.2) If only a part of the marker's vertices can be known, the coordinates of the hidden vertices are calculated using a known marker size. The marker is determined by comparing the calculated coordinates of the vertices with information on the vertices of the previously stored markers to track the feature points of the markers.
In addition, the AR system predicts the vertices of the markers in the moving state by applying the information about the vertices of the previously stored markers to the optical flow algorithm when the markers are moved, and the coordinates of the predicted vertices and the vertices of the stored markers. The marker is determined by comparing with and the feature points of the marker are tracked.
Next, the AR system may predict the information of the temporarily hidden marker using the tracked marker feature points (S930).
Finally, the augmented reality system augments the object using the information of the predicted marker (S940). As a result, the present invention enables interaction with augmented reality.
Those skilled in the art to which the augmented reality system and the object augmentation method based on the marker according to the present invention belong will be able to various modifications and modifications without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.
10: imaging unit
20: image processing unit
30: marker information detection unit
40: marker information storage unit
50: object enhancement unit
60 display unit
Claims (16)
An image processing unit 20 for performing image processing for converting the captured image into an image displayable on the display unit 60;
A marker information detector (30) for detecting information of a marker included in each frame constituting the converted image;
A marker information storage unit 40 for storing or updating information of the marker;
When the information of the marker is detected, after storing the information of the detected marker in the marker information storage unit 40, the object is augmented. Object reinforcement unit 50 for augmenting the object by tracking the;
Augmented reality system based on the marker, characterized in that it comprises a.
The object enhancing unit 50,
Augmented reality system based on the marker, characterized in that for performing the initialization to track the feature point of the marker when the information of the marker is not detected.
The object enhancing unit 50,
If the initialization for tracking the feature point of the marker is not completed, the augmented reality system based on the marker, characterized in that to terminate the feature point tracking of the marker.
The marker information is augmented reality system based on the marker, characterized in that it comprises the geometric feature of the marker, and the vertex information of the marker.
The object enhancing unit 50,
If the marker is not detected, the augmented reality system based on the marker, characterized in that the augmented object by tracking the feature points of the marker according to the information of the marker is detected in the previous frame and stored in the marker information storage unit 40 .
The object enhancing unit 50,
When the information of the marker is not detected, when all coordinates of the vertices of the markers determined using the previously stored information of the markers are acquired, information about the coordinates of the vertices of the markers and the stored vertices of the markers Augmented reality system based on the marker, characterized in that to track the feature points of the marker by comparing the process of determining the marker.
The object enhancing unit 50,
When the information of the marker is not detected, when a part of the coordinates of the vertex of the marker determined by using the stored information of the marker is obtained, the coordinates of the vertex that are not obtained using the known size of the marker are calculated. ,
And a feature point of the marker is tracked by comparing the calculated coordinates of the vertex of the marker with information on the stored vertices of the marker to determine the marker.
The object enhancing unit 50,
When the marker is not moved and the information of the marker is not detected, the stored information about the vertex of the marker is applied to an optical flow algorithm to predict the vertex of the moved marker.
And a feature point of the marker is tracked by comparing the predicted coordinates of the marker with information on the stored vertices of the marker to determine the marker.
Converting the captured image into an image displayable on the display unit 60;
Detecting information of a marker included in each frame constituting the image;
Reinforcing the object after storing the information of the detected marker when the information of the marker is detected, and reinforcing the object by tracking a feature point of the marker according to the information of the stored marker if the marker is not detected. ;
Object augmentation method of augmented reality system based on a marker comprising a.
Augmenting the object,
If the information of the marker is not detected, the method for augmenting an object of the augmented reality system based on the marker further comprising the step of performing an initialization for tracking the feature point of the marker.
Performing the initialization,
When the initialization for tracking the feature point of the marker is not completed, the object augmentation method of the augmented reality system based on the marker, characterized in that the end of the tracking of the feature point of the marker.
The information of the marker is an object augmented method of augmented reality system based on the marker, characterized in that it comprises the geometric feature of the marker, and the vertex information of the marker.
Augmenting the object,
If the marker is not detected, augmented reality system based on the marker, characterized in that to augment the object by tracking the feature points of the marker according to the information of the marker detected and stored in the previous frame.
Augmenting the object,
When the information of the marker is not detected, when all coordinates of the vertices of the markers determined using the previously stored information of the markers are obtained, the information on the coordinates of the vertices of the markers and the stored vertices of the markers are obtained. And a method of reinforcing an object of an augmented reality system based on a marker, characterized in that the feature track of the marker is tracked by comparing the marker.
Augmenting the object,
When the information of the marker is not detected, when a part of the coordinates of the vertex of the marker determined by using the previously stored information of the marker is obtained, the coordinates of the vertex that are not obtained by using the known marker size are calculated. ,
An object of the marker-based augmented reality system which tracks the feature points of the marker by comparing the calculated coordinates of the vertices of the markers with information on the stored vertices of the markers to determine the markers. Augmentation method.
Augmenting the object,
When the marker is not moved and the information of the marker is not detected, the stored information about the vertex of the marker is applied to an optical flow algorithm to predict the vertex of the moved marker.
Comparing the coordinates of the predicted vertex of the marker and the information on the previously stored vertex of the marker to determine the marker to track the feature point of the marker based on a marker-based augmented reality system Object augmentation method.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020110081456A KR20130019485A (en) | 2011-08-17 | 2011-08-17 | Augmented reality system based on marker and object augment method thereof |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020110081456A KR20130019485A (en) | 2011-08-17 | 2011-08-17 | Augmented reality system based on marker and object augment method thereof |
Publications (1)
Publication Number | Publication Date |
---|---|
KR20130019485A true KR20130019485A (en) | 2013-02-27 |
Family
ID=47897568
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
KR1020110081456A KR20130019485A (en) | 2011-08-17 | 2011-08-17 | Augmented reality system based on marker and object augment method thereof |
Country Status (1)
Country | Link |
---|---|
KR (1) | KR20130019485A (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101281314B1 (en) * | 2013-02-18 | 2013-07-03 | 구덕회 | System and method for augmented reality |
KR101973653B1 (en) * | 2018-04-30 | 2019-04-29 | 주식회사 유니크유엑스 | System And Method For Providing Contents On Augmented Reality |
CN111325984A (en) * | 2020-03-18 | 2020-06-23 | 北京百度网讯科技有限公司 | Sample data acquisition method and device and electronic equipment |
-
2011
- 2011-08-17 KR KR1020110081456A patent/KR20130019485A/en not_active Application Discontinuation
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101281314B1 (en) * | 2013-02-18 | 2013-07-03 | 구덕회 | System and method for augmented reality |
KR101973653B1 (en) * | 2018-04-30 | 2019-04-29 | 주식회사 유니크유엑스 | System And Method For Providing Contents On Augmented Reality |
CN111325984A (en) * | 2020-03-18 | 2020-06-23 | 北京百度网讯科技有限公司 | Sample data acquisition method and device and electronic equipment |
CN111325984B (en) * | 2020-03-18 | 2023-05-05 | 阿波罗智能技术(北京)有限公司 | Sample data acquisition method and device and electronic equipment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108805917B (en) | Method, medium, apparatus and computing device for spatial localization | |
CN105814611B (en) | Information processing apparatus and method, and non-volatile computer-readable storage medium | |
US10410089B2 (en) | Training assistance using synthetic images | |
US11037325B2 (en) | Information processing apparatus and method of controlling the same | |
US20180197321A1 (en) | Image stitching | |
JP6491517B2 (en) | Image recognition AR device, posture estimation device, and posture tracking device | |
US10255504B2 (en) | Object position tracking using motion estimation | |
US20130051626A1 (en) | Method And Apparatus For Object Pose Estimation | |
EP1966648A1 (en) | Method and device for controlling auto focusing of a video camera by tracking a region-of-interest | |
JP2007293722A (en) | Image processor, image processing method, image processing program, and recording medium with image processing program recorded thereon, and movile object detection system | |
US9269018B2 (en) | Stereo image processing using contours | |
CN111833403B (en) | Method and apparatus for spatial localization | |
KR20150067680A (en) | System and method for gesture recognition of vehicle | |
US10970807B2 (en) | Information processing apparatus and storage medium | |
US20180197308A1 (en) | Information processing apparatus and method of controlling the same | |
JP2018113021A (en) | Information processing apparatus and method for controlling the same, and program | |
JP6420605B2 (en) | Image processing device | |
CN110490131B (en) | Positioning method and device of shooting equipment, electronic equipment and storage medium | |
KR20130019485A (en) | Augmented reality system based on marker and object augment method thereof | |
JP2017207818A (en) | Image processing apparatus, image processing method and program | |
CN110009683B (en) | Real-time on-plane object detection method based on MaskRCNN | |
JP2014102805A (en) | Information processing device, information processing method and program | |
CN118648019A (en) | Advanced temporal low-light filtering with global and local motion compensation | |
US20230127539A1 (en) | Information processing apparatus, information processing method, and information processing program | |
JP2009077022A (en) | Driving support system and vehicle |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
N231 | Notification of change of applicant | ||
WITN | Withdrawal due to no request for examination |