KR20170042039A - A method for guiding based on augmented reality using mobile device - Google Patents

Abstract

Provided is an augmented reality-based guide method using a mobile device. The method includes: a step of determining whether a mobile device is located in a region of interest (RoI) which indicates an area within a predetermined boundary from a point of interest (PoI); a step of executing a two-dimensional map view display mode to display the current position of the mobile device and at least one among the RoI and the PoI on a two-dimensional map through a display part of the mobile device in response to the determination that the mobile device is not located in the RoI; and a step of executing an augmented reality view display mode to display a PoI menu and a camera view of the mobile device through the display part of the mobile device in response to the determination that the mobile device is located in the RoI. As such, the present invention is capable of improving a reality sense of a guide system.

Description

TECHNICAL FIELD [0001] The present invention relates to a method of guiding based on augmented reality using a mobile device,

The present invention relates to a method of providing a guide service to a user using a mobile device, and more particularly, to a method of providing a guide service to a user based on an augmented reality.

Augmented Reality (AR) technology enables a user experience with enhanced realism. Three-dimensional (3D) virtual information uses AR technology to enhance the user's environment. In this case, the user sees a mixed environment consisting of a combination of real information and virtual information. Typically, the user interprets the mixed information as an integrated entity rather than as a separate entity, which highlights the essential characteristics of the AR. Here, the AR allows the user to emphasize the absence of mediation (see Non-Patent Document 1).

AR technology maintains a sense of realism when used as a guidance system in a physically large area such as an outdoor environment. The guide essentially provides the ability to know where the tourists are, know what he / she is looking at, predict and answer questions that she may raise, and interact with others and the environment (See Non-Patent Document 2). However, although it is efficient to use the AR visualization technology integrated with the essence of the guide system, it is still difficult for the user to recognize the absence of the mediation because it is still a computer-mediated environment (see Non-Patent Document 3). Therefore, the use of suitable AR technology in the guidance system is important.

As described above, the augmented reality can be used for the guide service, but the development of the outdoor guide system using the AR generates much more challenges and problems than the indoor guide system (see Non-Patent Document 4). A continuous stream of augmented guide information requires constant user attention to the surroundings, provides unhelpful information that prevents a user from concentrating on real information, and ultimately provides an intuitive characteristic of enhancing real and virtual information .

Korean Patent Application No. 10-2012-0146230 Korean Patent Application No. 10-2013-0055869 Korean Patent Application No. 10-2013-0041011 Korean Patent Application No. 10-2011-0122176

 R. Azuma, "The challenge of making augmented reality work outdoors." In Mixed Reality: Merging Real and Virtual, 1999, pp. 379-390.  G. D. Abowd, et al. "Cyberguide: A mobile context-aware tour guide," Wireless networks, vol. 3, 1997, pp. 421-433.  M. Lombard, et al. "At the heart of it all: The concept of presence." Journal of Computer Mediated Communication, vol. 3, 1997, pp.  R. Azuma, "A survey of augmented reality". Presence, vol. 6 (4), 1997, pp. 355-385.  Sukan, M., Elvezio, C., Oda, O., Feiner, S., & Tversky, B. (2014, October). ParaFrustum: visualization techniques for guiding a user to a set of viewing positions and orientations. InProceedings of the 27th annual ACM symposium on User interface software and technology (pp. 331-340). ACM.  Feiner, S., MacIntyre, B., Hollerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1 (4), 208-217.

It is an exemplary object of the present invention to solve the above-mentioned problems, and it is an object of the present invention that the outdoor guidance system may have to selectively visualize the AR by the enhancement in relation to the user preference and can be augmented only when intensification is intuitive, The present invention provides a guide method based on Augmented Reality using a mobile device capable of adapting to a reaction.

It should be understood, however, that the present invention is not limited to the above-described embodiments, but may be variously modified without departing from the spirit and scope of the invention.

An augmented reality (AR) based guided method using a mobile device according to an embodiment of the present invention includes the steps of determining whether the mobile device is located within a region of interest (RoI) The region representing an area within a predetermined radius from a point of interest (PoI); In response to a determination that the mobile device is not located within the area of interest, displaying on the display of the mobile device a two-dimensional map view display that displays at least one of the area of interest and interest objects and the current location of the mobile device on a two- step; And displaying an augmented reality view display in response to a determination that the mobile device is located within the area of interest, wherein the display of the mobile device displays a camera view of the mobile device and a menu of objects of interest.

According to one embodiment, the step of displaying the augmented reality view further comprises displaying augmented reality content associated with the object of interest in the display of the mobile device in response to determining that the camera view of the mobile device includes the object of interest Step < / RTI >

According to one embodiment, the step of displaying the augmented reality view further comprises displaying augmented reality content associated with the object of interest in the display of the mobile device in response to a rotation or translation movement of the mobile device . ≪ / RTI >

According to one embodiment, the step of displaying the augmented reality view further comprises displaying the direction of the object of interest in the camera view of the mobile device in response to a determination that the camera view of the mobile device does not include the object of interest . ≪ / RTI >

According to one embodiment, in response to determining that the first ROI and the second ROI are located within a predetermined distance, the first ROI may be configured to include all of the first ROI and the second ROI.

According to one embodiment, the predetermined radius from the object of interest may be 35 m.

According to one embodiment, the first region of interest may be within a predetermined radius from the midpoints of the first ROI and the second ROI.

According to one embodiment, the mobile device may be a head-mounted display, a smart phone, or a tablet PC.

According to one embodiment, the mobile device can determine the location of the mobile device based on the GPS sensor.

According to one embodiment, the mobile device may receive location information of the object of interest from the object location information server of interest.

According to an augmented reality-based guide method using a mobile device according to an embodiment of the present invention, a two-dimensional view display mode in which a two-dimensional map is displayed based on a distance between a mobile device and an object of interest, The augmented reality view display mode in which the augmented contents are displayed can be automatically switched to each other.

Thus, the outdoor guidance system may have to selectively visualize the AR by augmentation with respect to user preferences, and can only be augmented if intensification is intuitive, and it may be possible to adapt to user response. In addition, by applying the augmented reality-based guide method using a mobile device according to an embodiment of the present invention to a guide system in an outdoor environment, the reality of the guide system can be improved.

1 is a system flow diagram of a guide system according to an embodiment of the present invention.
Figure 2 is an illustration of a two-dimensional map view display mode for a region of interest and an object of interest.
3 is a conceptual diagram showing the switching between the two-dimensional map view display mode and the augmented reality view display mode.
4 is a flowchart illustrating a method of guiding augmented reality based on a mobile device according to an exemplary embodiment of the present invention.
5 is a configuration diagram of a system in which an augmented reality-based guide method using the mobile device can be implemented.

For the embodiments of the invention disclosed herein, specific structural and functional descriptions are set forth for the purpose of describing an embodiment of the invention only, and it is to be understood that the embodiments of the invention may be practiced in various forms, The present invention should not be construed as limited to the embodiments described in Figs.

The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprise", "having", and the like are intended to specify the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, , Steps, operations, components, parts, or combinations thereof, as a matter of principle.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be construed as meaning consistent with meaning in the context of the relevant art and are not to be construed as ideal or overly formal in meaning unless expressly defined in the present application .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

Hereinafter, the term 'outdoor space' refers to a space opposite to the indoor space, which is not piled up on a building or an outer wall. In addition, it may mean a wide space, for example, a radius of 100 m or more, rather than a limited space such as a building.

On the other hand, '2D map' may mean a map in which various position information exists in the bird's-eye view as in the existing 2D map service.

Meanwhile, 'Augmented Reality (AR) technology' is a technology in which a user inputs virtual three-dimensional information to an output device (eg, a camera) through an input device (for example, For example, a display device).

On the other hand, 'Guide' means telling where a user is touring, showing what they are looking at, giving answers to questions they have and allowing them to interact with the environment or with others .

Related Technology

Recently, a variety of map services have been provided in various portal sites. For example, as shown in FIG. 1, various 2D map services based on a web or a smart phone may provide a function of searching for and searching for a place desired by a user. In addition to providing virtual maps, you can also provide guides through various visualization methods such as satellite maps and road views.

[Figure 1] Next map

On the other hand, there are cases where not only two-dimensional maps such as Google ™ from Google ™, but also actual three-dimensional position information are provided on a flat display. In the case where the user searches for specific location information (address, zone name) and the like, real information can be realistically visualized by providing corresponding real information.

[Figure 2] Google Earth

However, in the case of the conventional map service or Google Earth (R) as described above, there is a weak point that the user can not provide the correct route. That is, although it is possible to roughly display the location of a specific address or building on a map, it is not possible to precisely guide the position of an object that a user in a particular place may be interested in.

On the other hand, referring to Non-Patent Document 5, based on an image-based tracking technique and a sensor for estimating a head posture as shown in FIG. 3, it is possible to determine a position where a user should stand in a limited space , And to provide information. It also mentions providing feedback to ensure that you are in the right place, and that you are looking at the right place, depending on your location.

[Figure 3]

However, in the above non-patent document 5, since the above-mentioned guidance is performed on the assumption that the user is located in an indoor space, that is, a very limited space, there is a problem that it can not be applied to a guide system in an outdoor environment.

In the non-patent document 6, as shown in FIG. 4 below, a system capable of navigating in an urban environment provides augmented virtual information when the user looks at the front, And the like. Also, a configuration is disclosed in which virtual information is provided based on approximate location information of a user using a GPS sensor.

However, in the non-patent document 6, there is a limit to provide only the approximate positional information, not the accurate positional information.

[Figure 4]

Guide system

The augmented reality-based guide method using a mobile device according to an exemplary embodiment of the present invention can be implemented in a 2D space in an outdoor space using a mobile device such as a head-mounted display, a smart phone and a tablet PC, Mapping and Augmented Reality (AR) techniques can be used to provide functional elements that guide tourists (users). Therefore, it is possible to provide a guide system for automatically switching the 2D map view and the AR view according to the user's position, and a system for guiding a more specific position that the 2D map view can not be provided through the AR view .

1 is a system flow diagram of a guide system according to an embodiment of the present invention. As shown in FIG. 1, the guide system according to the present invention can be designed based on an actual user location and a point of interest (PoI). Such a system can automatically and interchangeably switch between a 2D map view mode and an Augmented Reality (AR) view mode. The user of the system is presumed to want to receive information related to the object of interest, such as where the object of interest is located and what the object of interest is. A user located away from a particular object of interest may receive the location of the object of interest from the guide system via a GPS-based 2D map in a two-dimensional map view mode, To access the object of interest. When a user approaches a region of interest (RoI) in which a plurality of objects of interest are located, the system automatically selects an augmented reality view mode that displays a camera input video and a PoI Menu You can switch.

When a user enters a region of interest, the guiding system may augment the content of the site-of-interest-object of interest sequentially. Augmented reality visualization based on image-based tracking represents a limited amount of information. Thus, the system may progressively visualize more information based on the user's location and the rotation and translation of the mobile device. The user approaching the area of interest can be first guided to the observation site, and upon arrival, the system can visualize the viewpoint of the object of interest. When the user aligns the observation point and the camera position in a straight line, the augmented reality contents of the object of interest may be displayed for the user's operation.

mobile Device  Augmented reality based guide

4 is a flowchart illustrating a method of guiding augmented reality based on a mobile device according to an exemplary embodiment of the present invention. Referring to FIG. 4, an augmented reality-based guide method using a mobile device according to an embodiment of the present invention will be described in detail below.

As described above, according to the augmented reality-based guide method using the mobile device according to the embodiment of the present invention, according to the user's actual position and the point of interest (PoI) in the outdoor space, Dimensional map view display mode and the augmented reality view display mode can be automatically switched. By automatically switching the two-dimensional map view display mode and the augmented reality view display mode, it is possible to automatically select an accurate location of interest objects and objects of interest It is possible to easily provide all of the information stored in the storage medium.

The augmented reality-based guide method using a mobile device according to an embodiment of the present invention determines whether a mobile device is located within a region of interest (RoI) as shown in FIG. 4 (S401) . Here, the region of interest may represent an area within a predetermined radius from a point of interest (PoI).

Figure 2 is an illustration of a two-dimensional map view display mode for a region of interest and an object of interest. As shown in FIG. 2, a plurality of interested objects (PoI1 to PoI8) may be present in the outdoor area, and a region located at a predetermined distance from the object of interest may be a region of interest (RoI1 to RoI7) Lt; / RTI > Whether or not to set an area which is far from the object of interest as the area of interest may be determined in consideration of the size of the object of interest or may vary according to the setting of the user. For example, the predetermined radius from the object of interest may be set to 35 m.

On the other hand, in response to determining that the first ROI and the second ROI are located within a predetermined distance, the first ROI may be configured to include all of the first ROI and the second ROI. In other words, basically, the ROI can exist as many as the number of PoIs of interest, but when the distance between the PoIs of interest is less than a predetermined distance, a plurality of ROIs are integrated into one ROI, May include objects of interest. For example, referring to FIG. 2, one ROI may include two ROIs, such as RoI3 surrounding PoI4 and PoI5.

When one ROI includes a plurality of ROIs, an ROI having a distance from any one of a plurality of ROIs within a predetermined distance may be designated as an ROI, As a region of interest. That is, the first ROI may represent a first ROI included in the first ROI and a region located within a predetermined radius from the middle point of the second ROI.

Referring again to FIG. 4, the augmented reality-based guide method using a mobile device according to an exemplary embodiment of the present invention includes a two-dimensional map view display mode and an augmented reality view display Mode can be freely switched.

If the user of the mobile device is not yet located in a region of interest that is an area of interest, the user may be more useful to reach the location of the object of interest than the side information of the object of interest. Thus, in response to the determination that the mobile device is not located within the area of interest, a two-dimensional map view display mode is provided that displays on the two-dimensional map at least one of the area of interest and the object of interest, (S402). Referring back to FIG. 2 cited above, in the two-dimensional map view display mode, on the map of the two-dimensional form, first, the objects of interest (PoI 1 to PoI 8), the regions of interest (RoI 1 to RoI 7) can do. Further, the position of the user measured based on the GPS sensor of the mobile device can be further displayed.

Thus, the user can confirm his or her approximate position in the 2D map view and easily access the region of interest, which is the region where the PoI of interest is present. That is, when the user's position is not included in any area of interest, the position of the object of interest and the user's position can be roughly displayed in the 2D map view on the display unit of the mobile device.

On the other hand, when the user is approaching the region of interest, which is the region in which the object of interest exists, the user may want additional information about the object of interest itself rather than geographic information about the object of interest. Accordingly, in response to the determination that the mobile device is located within the area of interest, the augmented reality view display mode may be performed to display the camera view of the mobile device and the object of interest menu on the display of the mobile device (S403). That is, when the user is included in the area of interest, the AR may be automatically switched to the augmented reality (AR) view so that an input device such as a video stream input by the mobile device may be displayed on the display unit, A PoI menu associated with the implementation of the augmented reality content, which may include additional information associated with the object of interest.

Then, when the user looks at a particular object of interest with the mobile device switched to the camera view, the user can confirm the virtual information matched thereto. For example, the image represented by the camera view of the mobile device may be further displayed on the display of the mobile device in response to a determination that the object of interest is included in the augmented reality content associated with the object of interest. On the other hand, augmented reality visualization based on image-based tracking may represent a limited amount of information. Thus, the system may progressively visualize more information based on the user's location and the rotation and translation of the mobile device. For example, the augmented reality content associated with the object of interest may be further displayed on the display of the mobile device in response to a rotation or translation movement of the mobile device.

On the other hand, even if the user is located in the area of interest, the user may not immediately recognize the location of the object of interest. For example, the size of the object of interest may be very small, and structures, buildings, and the like may be enclosed in the vicinity of the object of interest. In the conventional map-based location guidance service, only the geographical location information is guided, so that there is a problem in that it is impossible to guide such specific location.

Accordingly, the augmented reality-based guide method using a mobile device according to an embodiment of the present invention includes: in augmented reality view display mode, in response to a determination that the camera view of the mobile device does not include the interest object, To further display the direction for the object of interest. Based on the measurement results of sensors such as the gyroscope that can be included in the mobile device and the location of the user, the object orientation information calculated based on the positional relationship between the user's location and the object of interest, The calculated camera view azimuth information may be used.

3 is a conceptual diagram showing the switching between the two-dimensional map view display mode and the augmented reality view display mode. 3, when the user moves out of the area of interest during the operation of the augmented reality view display mode, the mode is again switched to the two-dimensional map view display mode, and the user inputs the location of the user and the number of PoIs to the mobile device Can be confirmed through Further, during operation of the two-dimensional map view display mode, when the user moves into the area of interest, the augmented reality view mode can be automatically activated.

System configuration diagram

5 is a configuration diagram of a system in which an augmented reality-based guide method using the mobile device can be implemented. 5, an augmented reality-based guide method using a mobile device according to an embodiment of the present invention includes an input device, a region of interest and a user location recognition component, a screen visualization component, and an output device ≪ / RTI >

The input device may include, as system inputs, an input device such as a camera, as well as an internal sensor such as a GPS sensor and a gyroscope sensor. The ROI and the user location recognition component can acquire the location information based on the GPS sensor and acquire the location information of the ROI acquired from the ROI server. In addition, it is possible to acquire information about an image-based object of interest and a user's viewpoint. The screen visualization component can visualize the object of interest and the user location based on the 2D map for use in the 2D map view display mode and can visualize the object of interest and observation point based on the augmented reality for use in the augmented reality view display mode have. The output device may operate as a display of the mobile device.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the following claims. It will be understood.

Claims (3)

An augmented reality (AR) based guiding method using a mobile device, the method comprising:
Determining whether the mobile device is located within a region of interest (RoI), the region of interest representing an area within a predetermined radius from a point of interest (PoI);
In response to a determination that the mobile device is not located within the area of interest, displaying on the display of the mobile device a two-dimensional map view display that displays at least one of the area of interest and interest objects and the current location of the mobile device on a two- step; And
Displaying a camera view and an object of interest menu of the mobile device on a display of the mobile device in response to a determination that the mobile device is located within the area of interest, Augmented reality, AR) based guided method. The method according to claim 1,
In the augmented reality view display step,
Further comprising displaying in a display of the mobile device an augmented reality content associated with the object of interest in response to a determination that the camera view of the mobile device includes the object of interest. . The method according to claim 1,
In the augmented reality view display step,
Further comprising displaying in the camera view of the mobile device a direction for the object of interest in response to a determination that the camera view of the mobile device does not include the object of interest.

Images