KR20140001152A - Areal-time coi registration system and the method based on augmented reality - Google Patents

Abstract

A disclosed technique relates to an interface capable of registering a context of interest (COI) in an augmented reality environment and, more specifically, to an augmented reality interface technique capable of specifying various types of objects of interest in the real space varied dynamically and adding and intuitively presenting virtual guidance/information associated therewith without physical constraints. The interface according to an embodiment of the disclosed technique can overcome the constraints of an augmented reality marker learned and defined previously to register a three-dimensional object of interest in an augmented reality space according to an immediate situation and can also provide an information representing technique in which augmented guide information provided when registering a three-dimensional object of interest varied dynamically according to work steps. [Reference numerals] (AA) COI capturing; (BB) Grouping(I); (C) Integrated context(Env. & User); (CC) COI area; (DD) Non-COI area; (EE) Type(Si); (FF) Content(Si); (GG) User intent; (HH) User data; (I) Key frame image; (i,j) Index; (II) Scenes(FOV); (JJ) Environmental data; (KK) Contextual information manager; (LL) Space management; (MM) Visualization management; (NN) Contextual visualization manager; (OO) Step(S_i-1, C_j-1); (PP) Step(S_i-, C_j); (QQ) Workflow; (RR) Exception(Breakdowns) overcome; (S) Step in workflow; (SS) Intent-User observation; (TT) Exception-Unintented act recognition; (UU) Act analysis-dissatisfaction element extraction; (VV) Combine-Context(Cj) update; (WW) Step(S_i+1, C_j+1)

Description

Augmented Reality-based Real-Time Interest Context Registration System and Its Method {AREAL-TIME COI REGISTRATION SYSTEM AND THE METHOD BASED ON AUGMENTED REALITY}

The disclosed technology relates to an interface that can register a context of interest (COI) in an augmented reality environment, and more specifically, a virtual guide that designates various types of objects of interest in a dynamically changing real space and associates them with them. / Augmented reality interface technology that can add and intuitively present information without physical constraints.

Since the advent of augmented reality technology, a wide range of user systems and age groups have broadened and various application systems have been introduced to provide an augmented reality experience. In particular, in the ubiquitous virtual reality application system that utilizes augmented reality interface as a system-user interaction tool, u connects reality and virtual seamlessly, provides realistic five senses, and enables real-time two-way interaction based on these interfaces. -Content is expected to be actively produced and processed. Therefore, as time passes, the augmented reality content will be accumulated and provided in a variety of forms based on the object of interest to the user.

Various computing technologies (web, sensor, mobile, etc.) that support the interaction between the physical interests and the real users in the real space that operates in real time have emerged to the level applicable to real life. However, the research on organic coupling with augmented reality interface for designing various types of interests and adding and intuitively presenting virtual guides / information without physical limitations is still in an early stage.

(1) Chang-gu Kang, Tae-jin Ha, Yoo-soo Oh, Un-taek Woo, “Augmented Reality Content Technology and Application for Realizing Ubiquitous Virtual Reality,” Journal of the Institute of Electronics Engineers, Vol. 38, No. 6, pp.449-455, 2011 (2) Choi, A., Park, Y., Jang, Y., and Woo, W., “mARGraphy: Mobile AR-based Dynamic Information Visualization,” Int. Symposium on Ubiquitous Virtual Reality, pp. 37-39, 2011 (3) Kim, K., Lepetit, V., and Woo, W., “Keyframe-based Modeling and Tracking of Multiple 3D Objects,” Int. Symposium on Mixed and Augmented Reality (ISMAR’10), pp.193-198, 2010 (4) Vertegaal, R., and Poupyrev, I., “Organic User Interfaces,” ACM Communications, Vol. 51, No. 6, 2008.

The disclosed technology provides an augmented reality-based organic interface that can intelligently process and present information in a dynamically changing real space. The disclosed technique provides a process for real-time registration of a context of interest (COI) that redefines an object of interest to present information as an initial step.

Context of Interest (COI) is a physical entity or context information that pays attention from the user's point of view, and actually represents data including an interactive area between the user and the system. This includes a point of interest (PoI), an object of interest (OoI), a location of interest (LoI), and the like. COI registration refers to generating and registering a corresponding COI object (including 2D / 3D area) coordinate system in a dual space corresponding to real space.

An augmented reality-based real-time interest context registration system according to an embodiment of the disclosed technology may receive an object feature point from a user input device and adjust a focus of a target interest context to provide feature point coordinates. And an object modeling unit configured to check whether the context is registered, and an object modeling unit which receives the feature point coordinates from the interested context focusing unit and models the object that is the target for registering the interested context.

In addition, in the augmented reality-based real-time interest context registration system according to the present invention, the interest context recognizer recognizes the interest context using the RFID ID tag information of the object, or recognizes the target object by detecting the ROI. .

In addition, in the augmented reality-based real-time interest context registration system according to the present invention, the object modeling unit, an object selection unit for selecting the object to register as the interest context, a feature point collection unit for collecting the three-dimensional feature points of the selected object, selected And an object shape selector for selecting shape information of the object and an object extender for three-dimensionally extending the object with the selected shape information.

In the real-time interest context registration method based on the reality of the disclosed technology, (a) checking whether the interest context is registered using the interest context recognizer, and (b) recognizing the interest context of the object in the step (c). If not, using the object modeling unit, modeling the object in three-dimensional shape and (c) using the context tag unit, by setting the tag information identifying the context with the coordinate information of the three-dimensional shape modeled object database It characterized in that it comprises a step of registering in.

The interface according to the disclosed technology allows a user to register an object of interest in the augmented reality space even though there is no prior learning about the application system. That is, the user can immediately register the object of interest according to the organic guide provided.

1 is a diagram illustrating a process of registering a COI in real time through an organic augmented reality interface according to an embodiment of the disclosed technology.
2 is a block diagram of an augmented reality-based real-time interest context registration system according to the present invention.
3 is a diagram illustrating a concept of recognizing a target object by a method for detecting a region of interest in the augmented reality-based real-time interest context registration system according to the present invention.
Figure 4 is a block diagram showing the detailed configuration of the object modeling in the augmented reality-based real-time interest context registration system according to the present invention.
5 is an embodiment screen showing the operation of the guide information in the augmented reality-based real-time interest context registration system according to the present invention.
Figure 6 is a flow chart showing the overall flow of the augmented reality-based real-time interest context registration method according to the present invention.
7 is a flowchart illustrating a detailed flow of step S30 of the augmented reality-based real-time interest context registration method according to the present invention.
8 is an embodiment screen showing the operation of step S30 in the augmented reality-based real-time interest context registration method according to the present invention.
9 is a diagram illustrating a process of modeling an object through a mobile terminal in detail.

The description of embodiments of the disclosed technique is merely illustrative for the structural and functional explanations of the disclosed technology, and thus the scope of the disclosed technology should not be construed as limited by the embodiments described herein. In other words, it should be understood that the embodiments of the disclosed technology are capable of various changes and various forms, and therefore, are capable of implementing the technical idea of the disclosed technology.

It is to be understood that the singular forms "a", "an", ",", "an", "an" , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof .

Each step described in the disclosed technique may take place differently from the stated order unless explicitly stated in the context of the specific order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

In order to augment the virtual object in the real space in real time, a reference object including a feature for stably maintaining the posture of the virtual object is required even when the camera viewpoint changes. Generating stable coordinates for learning and augmenting a region of interest or an object may be referred to as COI registration.

In particular, the predefined two-dimensional rectangular marker is difficult to be widely used in real life because of the limitation of the place and aesthetically unnatural. To compensate for this, there are registration techniques that use natural feature points. Intuitive registration is possible by calculating a posture by learning a part of an image in a patch unit, but there is a limitation in applying it to a 3D object.

In the case of an interface technology that registers a simple three-dimensional polygon object in real time on the premise that most objects present in and interacting with the real space can be modeled as an uncomplicated polygon, the user registers a reality object of interest that exists in an unfamiliar space. This allows for a more diverse and intuitive interaction environment.

1 is a view illustrating a process of registering a COI in real time through an organic augmented reality interface according to an embodiment of the disclosed technology.

OrganicAR is an interface that redefines the form of OUI (Organic User Interface), which is a next-generation interface that closely combines existing hardware and software, with an Augmented Form, and is improved to be suitable for an augmented reality environment. . Augmented form refers to an additional virtual information transfer entity that allows a user to experience various modalities through multiple detectable devices in real space. The three main elements to satisfy organic augmented reality are: 1) user-friendly mobile input is delivered to the wearable output device, 2) system / application functions have a close connection with augmented form of visualization, and 3) augmented form. Is that the content contained in should be dynamically changed to describe the flow of the system / application.

The first factor is that the results of input response from the user's personal mobile device, hand, wand, etc. are immediately presented to output devices such as head mounted display (HMD), eyeglass display, and mobile device. Say the environment. The second element is deeply related to the technology of visualizing in a form (Where, What, How) suitable for a given environment, user, and image by using the actual contextual information that can be acquired at the user's situation or work stage. The final element is information presentation and guidance techniques that lead to effective progression between stages, given that most of the work is done in a series of successive stages.

1 is a flowchart illustrating an application of organic augmented reality to a real-time COI registration process, and FIG. 2 is a diagram illustrating a real-time COI registration process. FIG. 1 shows a process of presenting an augmented form for inducing meaningful keyframe addition of a three-dimensional map expansion step in a series of registration processes (FIG. 2).

First, a COI capturing process is performed to separate a COI region and a non-COI region based on previously added keyframe information and user intention in the current image. After that, through the space management using the context information continuously collected in the field to extract the appropriate location / area (Where) of the various augmented forms. After that, visualization management is responsible for augmenting guide contents by selecting appropriate contents (What, How) from the current work list in the area recommended through space management.

In addition, when the user acts greatly out of a given guide and is detected through analysis of the context in which the behavior is collected, additional guide information may be generated to lead in the original correct direction.

2 is a block diagram showing the overall configuration of the augmented reality-based real-time interest context registration system according to the present invention, the interest context focusing unit 21, the interest context recognition unit 22, the object modeling unit 23 and the context tag unit (24).

The interest context focusing unit 21 receives the object feature points extracted from the user input device 10 in real time from a shape from motion (SFM), adjusts the focus of the interest context as a target, and serves to provide feature point coordinates. The coordinate of the feature point is transmitted to the interest context recognizer 22.

The interest context recognizer 22 checks whether the interest context is registered, and the interest context recognizer 22 according to the present invention recognizes an interest context with RF ID tag information of an object, or FIG. 3. As shown in FIG. 6, the target object is recognized by the ROI detection.

The object modeling unit 23 receives the feature point coordinates from the interested context focusing unit and performs a 3D shape modeling of an object that is a target for registering the interested context. The object modeling unit 23 according to the present invention As shown in FIG. 4, an object selecting unit 231, a feature point collecting unit 232, an object shape selecting unit 235, and a guide information unit 236 are included.

The object selector 231 serves to select a target object to be registered as a context of interest, and the feature point collector 232 collects 3D feature points of the selected object.

The object shape selection unit 235 serves to select shape information of the selected object, and the object shape selection unit according to the present invention forms a reference plane selection unit 233 for selecting a reference plane of the object and a shape of the selected object reference plane. It is preferable to include a shape information selection unit 234 for inputting a reference line or reference point for inputting information.

When the shape of the object reference plane is a polygon, the shape information selecting unit 234 according to the present invention connects two vertices, inputs a diagonal located on the plane of the polygon, and the shape of the object reference plane is circular. In this case, three points located on a circle are input. When two points are first selected, a circular guide shape is generated, and another point on the circle is selected to input accurate circle shape information.

The object extension unit 236 expands an object by the selected shape information three-dimensionally, thereby generating a three-dimensional shape of the object.

The guide information unit 237 is connected to the object selecting unit 231, the feature point collecting unit 232, the reference plane selecting unit 233, and the shape information selecting unit 234, and inputs object selection and object shape information. The city serves as a guide.

That is, the existing COI registration is insufficient to provide a guide for completing the registration to users without knowledge / experience, using the guide information unit 236, augmented the user guide suitable for each step of the context registration operation. When creating the initial environment map, it is implemented to provide key guides to help extract keyframes and to correct user's unnecessary behavior, and to provide a character guide that changes dynamically according to the steps.

5A illustrates an example of a user's movement direction guide for generating an environment map, FIG. 5B illustrates an example of information presentation for diagnosing / overcoming an exception by a user when creating an environment map, and FIG. An example text guide, (d) shows an example text guide drawing in extension to a three-dimensional model.

The context tag unit 24 serves to set tag information for identifying a context and register it in a database using coordinate information of a 3D shape modeled object, and to track an interest context according to the set tag information. And sharing is possible.

FIG. 6 is a diagram illustrating an entire flow of a method for registering a real-time interest context based on augmented reality according to the present invention. The interest context recognizer 22 is used to recognize an interest context (S10), and the recognized context is a database. The method checks whether or not it is registered at 30 (S20).

If it is not possible to recognize the context of interest of the object in step S20, the object modeling unit 23 may be used to model the object in three dimensions (S30).

7 is a diagram illustrating a detailed flow of the step S30 according to the present invention, by using the object selection unit 231, to select the object to register in the context of interest (S31).

In the present invention, it is preferable to provide the guide information according to the selection of the object using the guide information unit 237 before the step S31, in the embodiment of the present invention as shown in (a) of FIG. Similarly, the guide information is displayed on one side of the display screen which can be checked by the user, and the region of interest for object selection is displayed.

Next, by using the feature point collection unit 232, performing the step (S33) of collecting the three-dimensional feature point of the selected object, the step S33, using the user input device, to move around the target object It is desirable to collect the feature points sufficiently.

8 (a) is a view showing an embodiment of the step S31, Figure 8 (b) is a view showing an embodiment of the step S33.

Next, the shape information of the object is selected using the object shape selection unit 235 (S35), and in step S35 according to the present invention, the reference plane selection unit 233 is illustrated in FIG. 9. Selecting a reference plane of the object (S351), and using the shape information selection unit 234, inputting a reference line or reference point for inputting the shape information of the selected object reference plane (S353) It is desirable to.

8 (c) is a view showing an embodiment of the step S351, Figure 8 (d) is a view showing an embodiment of the step S353, in the embodiment of the present invention the step S351 and the S353 Before the step, the guide information unit was used to provide guide information according to the shape information of the object.

Next, using the object extension unit 236, performing the step (S37) of expanding the object by the shape information in three dimensions, in the embodiment of the present invention as shown in (e) of FIG. The three-dimensional shape of was created.

Next, using the context tag unit 24, tag information for identifying a context is determined using coordinate information of a 3D shape modeled object and registered in a database (S40).

As described above, when applying the augmented reality-based real-time interest context registration method according to the present invention, the user can easily grasp the state or intention of the system through the two-dimensional guide that dynamically changes according to the situation, and consequently, the knowledge about the modeling. You can enjoy the technical effects of performing every step without

FIG. 9 is a diagram illustrating a process of modeling an object through a mobile terminal in detail. A plurality of 3D objects may be registered by performing Cuboid modeling including a complex 3D object appearance. Real-time recognition / tracking of 3D objects is possible.

Touch can accurately model the edges of an object (bezel-touch), and push-pull allows for more accurate modeling of three-dimensional objects by pushing or pulling objects in the depth direction (intrusion & extrusion). Can be.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention as defined in the following claims It can be understood that it is possible.

10: user input device 20: context register
21: Interest context focus 22: Interest context recognizer
23: object modeling unit 24: context tag unit
30: database 40: context interpreter
231: object selection unit 232: feature point collection unit
233: reference plane selection unit 234: shape information selection unit
235: object shape selection unit 236: object extension unit
237 Guide Information Department

Claims (12)

An interest context focus unit configured to receive an object feature point from a user input device, adjust a focus of a target context of interest, and provide feature point coordinates;
An interest context recognizer for checking whether the interest context is registered;
Augmented reality-based real-time interest context registration system, characterized in that it comprises an object modeling unit for receiving the feature point coordinates from the interest context focusing unit to model the object that is a target for registration of interest context.
The method of claim 1,
The interest context recognizer
An augmented reality-based real-time interest context registration system, characterized in that it recognizes a context of interest by RFID ID tag information of an object or recognizes a target object by detecting a region of interest.
The method of claim 1,
The object modeling unit,
An object selecting unit for selecting an object to be registered as a context of interest;
A feature point collector for collecting a 3D feature point of the selected object;
An object shape selection unit for selecting shape information of the selected object, and
Augmented reality-based real-time interest context registration system comprising an object expansion unit for three-dimensionally extending the object with the selected shape information.
The method of claim 3,
The object shape selection unit,
A reference plane selection unit for selecting a reference plane of the object;
And a shape information selection unit which inputs a reference line or a reference point for inputting shape information of the selected object reference plane.
5. The method of claim 4,
The shape information selection unit
When the shape of the object reference plane is a polygon, connecting two vertices, input a diagonal line located on the plane of the polygon,
When the shape of the object reference plane is a circular shape, a real-time interest context registration system based on augmented reality, characterized in that to input three shape points on the circle, the circular shape information.
The method of claim 1,
In connection with the object modeling unit,
Augmented reality-based real-time interest context registration system further comprises a context tag unit for setting the tag information for identifying the context, using the coordinate information of the three-dimensional shape modeled object.
The method of claim 3,
The object modeling unit,
Augmented reality-based real-time interest context registration system, characterized in that it further comprises a guide information for providing guide information when selecting the object and the shape information of the object.
(a) checking whether an interest context is registered using an interest context recognizer;
(b) if the context of interest of the object is not recognized in step (c), using the object modeling unit, modeling the object in three-dimensional shape; and
(c) using the context tag unit, setting tag information identifying a context using coordinate information of a 3D shape modeled object and registering the tag information in a database. .
9. The method of claim 8,
The step (a)
Recognizing a context of interest by the RFID ID tag information of the object, or a method of registering a real-time interest context based on augmented reality, characterized in that the step of recognizing the target object by the detection of the region of interest.
9. The method of claim 8,
The step (b)
(b-1) selecting an object to be registered in the context of interest using the object selector;
(b-2) collecting a 3D feature point of the selected object by using the feature point collector;
(b-3) selecting shape information of the object by using an object shape selection unit;
and (b-4) using the object extension unit to three-dimensionally expand the object with the shape information.
10. The method of claim 9,
Step (b-3),
(b-3-1) selecting a reference plane of the object using a reference plane selection unit; and
and (b-3-2) inputting a reference line or a reference point for inputting the shape information of the selected object reference plane by using the shape information selection unit.
10. The method of claim 9,
Before step (b-1)
(b-5) using the guide information unit, further comprising providing guide information according to the selection of the object,
Before step (b-3)
(b-6) The method of registering real-time interest context based on augmented reality, characterized by providing guide information based on input of shape information of an object using a guide information unit.

Images