KR20240030345A - Method and apparatus for display virtual realiry contents on user terminal based on the determination that the user terminal is located in the pre-determined customized region - Google Patents

Abstract

Based on the location of the user terminal, it is determined whether the user terminal is located in a preset custom area in the space, and whether the user terminal is located in a recognition area that allows the user terminal to recognize objects in the custom area. , A method for controlling a user terminal is provided, which recognizes an object when the user terminal is located in a recognition area and displays preset virtual reality (VR) content in association with the recognized object through an AR view.

Description

Method and device for displaying virtual reality content on a user terminal based on the location of the user terminal in a preset custom area DETERMINED CUSTOMIZED REGION}

The description below relates to a method and device for controlling a user terminal based on the location of the user terminal. As the user terminal located in a preset custom area in the space recognizes the object in the custom area, the object recognized by the user terminal It relates to a method and device for displaying virtual reality (VR) content related to.

Augmented Reality (AR) refers to a technology that fuses and complements virtual objects and information created with computer technology in the real world. In other words, augmented reality is a technology that augments and displays virtual content in a real environment, and users can view augmented content corresponding to the real environment through an electronic device.

Through the user terminal, which is the user's electronic device, content related to objects existing at a certain distance or within a certain radius based on the location of the user terminal is provided by augmenting the image of the user terminal. In this way, when augmented content is provided only based on the physical distance from the user terminal, there are cases where an unnecessarily large amount of content is augmented, increasing the user's cognitive load and the load on the user terminal. Additionally, content that does not match the service intended by the user may be inappropriately augmented on the user terminal.

Therefore, when a user is located in a specific area in a space, there is a need for a method of providing content so as to output only content that is highly useful to the user to the user terminal and provide the user with an immersive experience.

As a technology related to various services utilizing augmented reality technology, for example, in Korean Patent Publication No. 10-2014-0065963 (published on May 30, 2014), an image of the driving path captured by a camera installed in a vehicle An augmented reality navigator is disclosed that displays on a display and maps virtual display information guiding a driving route to the image displayed on the display.

The information described above is for illustrative purposes only and may include content that does not form part of the prior art and may not include what the prior art would suggest to a person skilled in the art.

As an object within the custom area is recognized through the AR view by a user terminal located in a preset custom area within the space, a method of displaying preset VR content in association with the recognized object may be provided.

In order to provide more detailed and rich information about the object to the user terminal, VR content related to the object in the custom area can be displayed through an AR view.

When the user terminal is located in a preset custom area, settings of the user terminal, including execution of a module for object recognition, can be controlled according to scenario conditions defined for the custom area.

Various scenario conditions can be set for the custom area, and depending on whether the scenario conditions are satisfied, VR content provided to the user terminal may be different or additional content may be provided to the user terminal.

According to one aspect, in a method of controlling a user terminal performed by a computer system that is a server or a user terminal, based on the location of the user terminal, whether the user terminal is located in a preset custom area in space is determined. Determining whether the user terminal is located in a recognition area that allows the user terminal to recognize an object in the custom area; If the user terminal is located in the recognition area, the camera of the user terminal Recognizing the object through an AR view including an image captured by the user and displaying preset virtual reality (VR) content in association with the recognized object through the AR view. A method for controlling a terminal is provided.

At least one service element is associated with the custom area, the service element includes data for displaying the VR content related to a service associated with the space, and the user terminal is in the custom area Depending on the location in the area, scenario conditions for controlling the user terminal may be defined based on the service element.

The service element includes data for controlling the settings of the user terminal to provide the user terminal with a service related to the space, and the method of controlling the user terminal includes when the user terminal enters the custom area. The method may further include controlling settings of the user terminal, including execution of a module for recognizing the object, according to the scenario conditions.

The step of determining whether the user terminal is located in the recognition area may include determining that the user terminal is located in the recognition area when the object is displayed in the AR view.

The step of determining whether the user terminal is located in the recognition area includes a condition that the distance between the object and the user terminal is less than a predetermined distance, a condition that the object is displayed in a specific area in the AR view, and the AR If at least one condition among the conditions for displaying the object more than a predetermined percentage of the view is satisfied, it may be determined that the user terminal is located in the recognition area.

The recognition area is a predefined area within the custom area, and the step of determining whether the user terminal is located in the recognition area includes, if the location of the user terminal is located in the predefined recognition area, the recognition area It may be determined that the user terminal is located in .

The method of controlling the user terminal may further include controlling settings of the user terminal including execution of a module for recognizing the object when the user terminal enters the recognition area.

The step of displaying the VR content includes changing the viewpoint according to a change in the viewpoint of the camera to display the VR content, and controlling the user terminal when the user terminal is not located in the recognition area. If not, the step of terminating the display of the VR content may be further included.

In the step of displaying the VR content, the VR content may be displayed on the entire screen of the AR view, and the VR content may depict a virtual space around the object.

The virtual space may be expressed as a virtual sphere space centered on the object.

The virtual space may be expressed as a virtual three-dimensional space determined by reflecting the geometry of the surrounding space where the object is located.

The scenario condition is that when the user terminal is located in the custom area, the VR content, which is determined based on another custom area that the user terminal entered before entering the custom area, is the AR. It is set to be displayed through a view, and the step of displaying the VR content may display VR content determined based on the different custom area through the AR view.

The scenario condition is set so that when the user terminal is located in the custom area, the VR content, which is determined based on the point through which the user terminal passed, is displayed through the AR view, and the VR content is displayed. In the step, VR content determined based on the point passed by the user terminal may be displayed through the AR view.

A plurality of points are defined inside or outside the custom area, and the step of displaying the VR content includes when the user terminal passes through a first point among the plurality of points and the user terminal When passing through a second point among the plurality of points, different VR contents can be displayed through the AR view.

A plurality of custom areas are set in the space, and the scenario conditions include i) a first scenario condition indicating whether the user terminal has passed through a custom area other than the custom area, and ii) a and at least one of a second scenario condition indicating whether the number of user terminals located falls within a predetermined range, wherein the scenario condition determines whether the scenario condition is satisfied or not, and the space is determined based on the service element for the user terminal. is set to provide different services or information related to or to differently control settings of the user terminal, and the step of displaying the VR content includes displaying the VR content through the AR view only when the scenario condition is satisfied. Alternatively, different VR content may be displayed when the scenario conditions are satisfied and when the scenario conditions are not satisfied.

The method for controlling the user terminal further includes receiving a route guidance request to a destination from the user terminal and providing route guidance to the user terminal according to the route guidance request, and providing route guidance to the user terminal in the custom area. Determining whether the user terminal is located, determining whether the user terminal is located in the recognition area, recognizing the object, and displaying the VR content include: It may be performed when the user terminal moves to the destination.

The scenario condition is set so that VR content determined based on the number of times the user terminal has entered the custom area is displayed through the AR view, and the step of displaying the VR content includes displaying the VR content through the AR view. VR content determined based on the number of times the user terminal has entered the area may be displayed.

In another aspect, a computer system implementing a user terminal or a server, comprising at least one processor configured to execute computer-readable instructions included in a memory, wherein the at least one processor is configured to determine the location of the user terminal. Based on, determine whether the user terminal is located in a preset custom area in the space, determine whether the user terminal is located in a recognition area that allows the user terminal to recognize objects in the custom area, and , when the user terminal is located in the recognition area, the object is recognized through an AR view including an image captured by the camera of the user terminal, and a preset virtual object is associated with the recognized object through the AR view. A computer system is provided that displays Virtual Reality (VR) content.

Define a custom area as a two-dimensional or three-dimensional space on a map of the space, associate service elements with the custom area, and create scenarios to provide services or information related to the space or control settings of the user terminal based on the service elements. By defining it, you can create a custom area to control the user terminal according to the scenario.

When the user terminal is located in a preset custom area, a module for object recognition is executed according to the scenario conditions defined for the custom area, and objects in the custom area are recognized, so that the user terminal and the server with which the user terminal communicates The use of unnecessary resources can be prevented.

By displaying VR content related to an object in the custom area through an AR view, more detailed and rich information about the object can be provided to the user terminal, and an immersive experience can be provided to the user of the user terminal.

The effectiveness of content provision can be increased by setting various scenario conditions for the custom area and varying VR content provided to the user terminal or providing additional content to the user terminal depending on whether the scenario conditions are satisfied.

FIG. 1A illustrates a method of controlling a user terminal based on the user terminal being located in a preset custom area, according to an embodiment.
FIG. 1B illustrates a method of controlling the user terminal to display VR content when the user terminal is located in a preset custom area and an object in the custom area is recognized, according to one embodiment.
Figure 2 shows a computer system that performs a method for controlling a user terminal, according to one embodiment.
Figure 3 is a flowchart showing a method of controlling the user terminal to display VR content when the user terminal is located in a preset custom area and an object in the custom area is recognized, according to an embodiment.
FIG. 4 is a flowchart illustrating a method of controlling settings of a user terminal as the user terminal enters a recognition area that enables recognition of objects in a custom area, according to an example.
FIG. 5A illustrates a method of controlling settings of a user terminal based on the user terminal being located in a preset custom area, according to an example.
FIG. 5B illustrates a method of providing VR content to a user terminal based on the user terminal being located in a preset custom area, according to an example.
Figure 6 shows a method of defining AR content associated with an object in a custom area, according to one example.
7 illustrates a method of defining VR content associated with an object in a custom area, according to one example.
Figure 8 shows a method of defining AR content and VR content associated with an object in a custom area, according to an example.
Figure 9 shows a custom area defined in three-dimensional space, according to an example.
Figure 10 shows the DoF (Degree Of Freedom) of a user terminal, according to an example.
Figure 11 shows a custom area including a plurality of points, according to an example.
Figure 12 shows a method in which a plurality of custom areas are set in a space and scenario conditions for controlling a user terminal are set for each custom area, according to an example.

Hereinafter, embodiments will be described in detail with reference to the attached drawings.

Figure 1 shows a method of controlling a user terminal based on the user terminal being located in a preset custom area, according to an embodiment.

“Control” of the user terminal 100 in embodiments to be described later, for example, as shown in FIG. 1, uses the AR view 10 of the user terminal 100 to control the image of the AR view 10. Augmenting and displaying content representing services or information related to the space 5 (eg, AR content 30, etc.); Displaying a user interface (UI) related to a service associated with the space 5 on the user terminal 100; displaying map element content 35 related to a service associated with space 5 on the map (or map view 20) of the user terminal 100; It may include at least one of controlling settings of the user terminal 100 in order to provide the user terminal 100 with a service related to the space 5. Additionally, “controlling” the user terminal in embodiments of the present disclosure may include controlling functions and/or operations of the user terminal, but is not limited to the above examples.

'Space 5' related to the provision of augmented content in embodiments may represent an indoor space and/or an outdoor space. In other words, the space 5, which will be described later, may be defined as a space including an indoor location or an outdoor location, and may also be defined as a complex space in which indoor space and outdoor space are combined. The space 5 may be at least a part of the service target area for providing services including provision of content.

In the detailed description to be described later, the location of the user holding the user terminal 100 may be described as the location of the user terminal 100 for convenience of explanation. Additionally, for convenience of explanation, 'user' and 'user terminal 100' owned by the user may be used interchangeably.

In the detailed description to be described later, “augmented and displaying” content and/or information (e.g., indicators, etc.) on an image refers to overlapping and displaying the corresponding content and/or information on the video/screen of the AR view, depending on the embodiment. It can be interpreted in an inclusive sense.

As shown, a user terminal 100, such as a smart phone, can photograph the surroundings using a camera and display AR content 30 through the AR view 10 including the image captured by the camera. . AR content 30 may be related to an object existing in space 5. This AR content 30 may be mapped to the corresponding object (or node representing the object).

At least one custom area may be defined on the map of the space 5 shown. In the illustrated example, a total of two custom areas, a first custom area 60 and a second custom area 70, are defined. At least one service element may be associated with each custom area 60, 70. In addition, scenarios for providing services or information related to the space 5 or controlling settings of the user terminal 100 may be defined in each custom area 60 and 70 based on the corresponding service elements. The services or information may include information for advertising or marketing. These scenarios may include conditions for controlling the user terminal 100 (i.e., scenario conditions). The 'custom areas 60 and 70' may be preset on the server 200. For example, on the server 200, at least a partial area of the space 5 or a distinct partial area of the space 5 may be set as a 'custom area 60, 70'.

A method of defining each custom area (60, 70) and establishing (creating) the custom areas (60, 70) by associating service elements and scenarios with the custom areas (60, 70) will be described later.

In an embodiment, it may be determined whether the user terminal 100 is located in the first custom area 60 in the space 5 based on the location of the user terminal 100, and the user terminal 100 may be located in the first custom area 60. If it is determined to be located in the custom area 60, AR content 30 may be displayed in the AR view 10 of the user terminal 100 based on service elements associated with the custom area 60. For example, when the user terminal 100 enters the first custom area 60, AR content 30 associated with the object 'metal type' may be displayed in the AR view 10. At this time, the current location 50 of the user terminal 100 may be displayed on the map view 20. The current location 50 may correspond to a location 50' in space 5. In the illustrated embodiment, when the user terminal 100 enters the first custom area 60, AR content 30 associated with “metal type”, which is an object existing outside the first custom area 60, is displayed. An example of the display is shown, but depending on the service element associated with the first custom area 60, when the user terminal 100 enters the first custom area 60, inside the first custom area 60 Other AR content related to an existing object, for example, “Eumbo of the Queen Mother,” etc. may be displayed.

Meanwhile, in another embodiment, according to scenario conditions defined in the first custom area 60, such as when the user terminal 100 moves away from the first custom area 60 based on the location of the user terminal 100, AR In the view 10, different types of content may be displayed, an appropriate UI may be provided, appropriate map element content may be displayed, or appropriate settings of the user terminal 100 may be changed.

The map view 20 displayed together with the AR view 10 on the screen of the user terminal 100 may include a map matching the image of the AR view 10. The map view 20 may include the current location 50 of the user terminal 100. The map view 20 may include a two-dimensional map or a three-dimensional map. The map view 20 may include objects within the space 5 through which the user terminal 100 moves. Map element content 35 may be displayed in map view 20 corresponding to AR content 30. Map element content 35 may be displayed when AR content 30 is displayed in AR view 10, or when an object associated with AR content 30 is included in map view 20.

The user terminal 100 may communicate with the server 200 to control entry and exit into the custom areas 60 and 70. For example, the server 200 may control the user terminal 100 based on service elements associated with the custom areas 60 and 70 where the user terminal 100 is located.

The server 200 is a computer system that manages the custom areas 60 and 70 defined for the space 5, and is a server that manages the association of service elements and definition of scenarios for the custom areas 60 and 70. You can. Additionally, the server 200 may be a map server that provides digital maps (3D maps and/or 2D).

1B to 14, which will be described later, as a more specific method of controlling the user terminal 100 as the user terminal 100 is located in the custom area 60 and 70, virtual reality for the user terminal 100 How to provide (Virtual Reality; VR) content is explained in more detail.

FIG. 1B illustrates a method of controlling the user terminal to display VR content when the user terminal is located in a preset custom area and an object in the custom area is recognized, according to one embodiment.

In FIG. 1B, when the user terminal 100 enters the custom area 60 and 70 while moving according to the route guidance and recognizes the object 105, the object 105 recognized in the AR view 10 and A method of displaying VR content 107 is shown in association.

The maps 101 and 102 shown may correspond to the map view 20 described above. The map view 101 may display a route 132 for route guidance to a predetermined destination. The user terminal 100 can move along the path 132 and recognize the object 105 after entering the custom area 60 and 70. When the object 105 is recognized, VR content 107 may be displayed in association with the recognized object 105 in the AR view 10.

In addition, in FIG. 1B, regardless of route guidance, when the user terminal 100 enters the custom area 60 and 70 while moving and recognizes the object 105, the object 105 recognized in the AR view 10 A method of displaying VR content 107 in association with an object 105 is shown.

The object 105 is disposed in the space 5 and, for example, when the space 5 is a museum, it may be an exhibit displayed in the museum. In the illustrated example, the object 105 may be “King Jinheung’s Pure Monument.”

Unlike in FIG. 1A, the user terminal 100 does not simply display the AR content 30 associated with the object 105, but also displays the VR content 107 as an AR view 10 to provide a more immersive experience to the user. It can be displayed through . VR content 107 may provide richer information about the related object 105 compared to AR content 30. For example, VR content 107 may include AR content and environmental content depicting a background related to the object 105, such as a background surrounding the object.

VR content 107 may be displayed in the AR view 10 based on service elements associated with the custom areas 60 and 70 and scenario conditions defined for the custom areas 60 and 70 .

For more detailed information about the VR content 107 and a more specific method of controlling the user terminal 100 to display the VR content 107 in the AR view 10, refer to FIGS. 2 to 12 to be described later. It is explained in detail.

Figure 2 shows a computer system and a server that perform a method for controlling a user terminal, according to one embodiment.

The user terminal 100 described above with reference to FIG. 1 according to embodiments may be implemented through the illustrated computer system 2. For example, a computer program for implementing the methods of the embodiments may be installed and driven in the computer system 2, and the computer system 2 may provide augmented content according to the embodiments under the control of the driven computer program. method can be performed.

The method of controlling the user terminal 100 according to embodiments may be performed through a PC-based program or a mobile terminal-specific application. For example, the method of controlling the user terminal 100 according to embodiments is implemented in the form of a program that operates independently, or is configured in the form of an in-app of a specific application and can be operated on the specific application. It can be implemented to do so. This specific application, for example, is installed on the computer system 2 and provides VR content 107 through the AR view 10, thereby performing a method of controlling the user terminal 100 of the embodiment.

The computer system 2 is an electronic device corresponding to the user terminal 100 described above, and may be a smartphone or similar device capable of installing and executing an application or program as shown in FIG. 1. In addition, the computer system 2 may be a personal computer (PC), a laptop computer, a laptop computer, a tablet, an Internet of Things device, or a wearable computer. You can.

As shown, the computer system 2 may include a memory 110, a processor 120, a communication interface 130, and an input/output interface 140 as components for executing a method for controlling the user terminal 100. You can.

The memory 110 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer system 2 as a separate permanent storage device that is distinct from the memory 110. Additionally, an operating system and at least one program code may be stored in the memory 110. These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 110 through the communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer system 2 based on computer programs being installed by files received over network 160.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 120 by the memory 110 or the communication interface 130. For example, the processor 120 may be configured to execute received instructions according to program codes stored in a recording device such as memory 110.

That is, the processor 120 can manage the components of the computer system 2 and execute programs or applications used by the computer system 2. For example, the processor 120 executes an application for performing a method of controlling the user terminal 100 of the embodiment, and data received from the server 200 to control the user terminal 100 (e.g., AR view ( It may be configured to process data received from the server 200 to provide AR content 30 through 10). Additionally, the processor 120 may process operations necessary for executing programs or applications and processing data, and may be at least one processor of the computer system 2 or at least one core within the processor.

The communication interface 130 may provide a function for the computer system 2 to communicate with another computer system (not shown) through the network 160. For example, requests, commands, data, files, etc. generated by the processor 120 of the computer system 2 according to the program code stored in a recording device such as memory 110 are transmitted to the network ( 160) and can be transmitted to other computer systems. Conversely, signals, commands, data, files, etc. from other computer systems may be received by the computer system 2 through the communication interface 130 of the computer system 2 via the network 160. Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or memory 110, and files, etc. may be stored in a storage medium (as described above) that the computer system 2 may further include. It can be stored as a permanent storage device). For example, the communication interface 130 may be a hardware module such as a network interface card, network interface chip, and networking interface port of the computer system 2, or a software module such as a network device driver or networking program.

The input/output interface 140 may be a means for interfacing with the input/output device 150. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 140 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. The input/output device 150 may be composed of a single device with the computer system 2.

Additionally, in other embodiments computer system 2 may include more components than those of FIG. 2 . However, it is not necessary to clearly show most prior art components. For example, the computer system 2 is implemented to include at least some of the input/output devices connected to the input/output interface 140 described above, or a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, It may further include other components such as a database, etc. As a more specific example, when the computer system 2 is implemented in the form of a mobile device such as a smartphone, the mobile device generally includes a camera, an acceleration sensor or a gyro sensor, various physical buttons, buttons using a touch panel, Various components such as input/output ports and vibrators for vibration may be implemented to be further included in the computer system 2.

For example, the computer system 2 corresponding to the user terminal 100 may include a camera that photographs the surroundings to execute the AR view 10. The computer system 2 can display an image captured through a camera as an AR view 10, and display AR content 30 and VR content 107 by augmenting the image of the AR view 10. . For example, the computer system 2 determines whether the user terminal 100 is located in the preset custom area 60, 70 according to the positioning, and determines whether the user terminal 100 is located in the custom area 60, 70. When 105 is recognized, VR content 107 may be displayed in association with the recognized object 105 in the AR view 10. A method using visual localization or VPS (Visual Positioning System) may be used to determine the position of the user terminal 100. Alternatively, the positioning of the user terminal 100 may include a method by recognizing a marker, a method using GPS, a method using WPS (Wi-Fi Positioning System), a method using a Bluetooth module, and odometry. Methods such as using can be used.

Meanwhile, the actual operation of determining whether the computer system 2 is located in the custom areas 60 and 70 and whether the object 105 is recognized may be performed by the server 200 rather than the computer system 2. there is. The server 200 communicates with the computer system 2 and provides data for controlling the user terminal 100 to the computer system 2, thereby enabling the user terminal 100 to be controlled. It could be a device. The server 200 is a device that manages (registers, changes, deletes, stores, maintains, etc.) the data and content to be provided to the computer system 2 and the custom areas 60 and 70, and includes a database or such a database. You can communicate with. Additionally, server 200 may be or include a map server that provides digital maps (3D maps and/or 2D). Server 200 may include at least one computer system. The computer system included in the server 200 may include similar configurations to the computer system 2 described above, and redundant description thereof will be omitted.

In the detailed description to be described later, for convenience of explanation, embodiments will be described focusing on the computer system 2 corresponding to the user terminal 100, and communication with the server 200 and operations on the server 200 side will be briefly or omitted. There are cases where it happens. In addition, for convenience of explanation, the operations (steps) performed by the components (e.g., processor, etc.) of the computer system 2 (or the server 200) are referred to as the computer system 2 (or the server 200). It can be described as being carried out by .

The description of the technical features described above with reference to FIG. 1 can also be applied to FIG. 2 , so overlapping descriptions will be omitted.

Figure 3 is a flowchart showing a method of controlling the user terminal to display VR content when the user terminal is located in a preset custom area and an object in the custom area is recognized, according to an embodiment.

Referring to FIG. 3, a method for controlling the user terminal 100 performed by the computer system 2 will be described. The computer system 2 may be the user terminal 100 described above with reference to FIG. 1, and in the detailed description to be described later, the terms user terminal 100 and server 200 will be used instead of the computer system 2. Examples will be described below.

In addition, at least some of the steps 302 to 350 to be described later and the steps described with reference to FIG. 4 may be configured to be performed by the server 200 rather than the user terminal 100. Below, the execution of steps will be described focusing on the user terminal 100, and redundant explanation regarding the server 200 may be omitted.

In step 310, the user terminal 100 or the server 200, based on the location of the user terminal 100, allows the user terminal 100 to enter the preset custom areas 60 and 70 within the space 5. It can be determined whether it is located in . The custom areas 60 and 70 may be areas defined based on an absolute coordinate system within the space 5. The custom areas 60 and 70 may be associated with at least one service element. For example, the user terminal 100 may determine the location of the user terminal 100 through a method based on at least one of GPS, VPS (Visual Positioning System), and WPS (Wi-Fi Positioning System), and based on the determined location Thus, it is possible to determine whether the user terminal 100 is located in the custom area 60 or 70, or whether it enters the custom area 60 or 70 or leaves the custom area 60 or 70. Alternatively, the server 200 may receive the determined location of the user terminal 100, and depending on the received location, determine whether the user terminal 100 is located in the custom area 60 or 70, or in the custom area 60. , 70), it is possible to determine whether to enter or exit.

In step 320, the user terminal 100 or the server 200 places the user terminal 100 in a recognition area that allows the user terminal 100 to recognize the object 105 in the custom areas 60 and 70. It can be determined whether this is located or not.

In step 330, the user terminal 100 may recognize the object 105 within the custom areas 60 and 70. In other words, the user terminal 100 can recognize the object 105 in the recognition area. The object 105 may be recognized by a module for recognizing the object 105. The module for recognizing the object 105 may be, for example, a module for VOT (Vision Object Tracking). Alternatively, the module for recognizing the object 105 may be a configuration used to recognize the object 105 using the smart lens of the user terminal 100. This module may be part of the processor 120 or may be separately implemented within the user terminal 100. Additionally, these modules may be software modules that the processor 120 executes.

The object 105 may be recognized through the AR view 10. For example, when the object 105 is included in an image captured by a camera included in the AR view 10, the object 105 may be recognized.

In step 340, the user terminal 100 may display preset VR content 107 in association with the object 105 recognized in step 330 through the AR view 10. VR content 107 may be 360-degree content. For example, VR content 107 may include a panoramic image (imgae). VR content 107 may include AR content depicting the object 105 and environmental content depicting a background related to the object 105, for example, a background surrounding the object. VR content 107 may be associated with a node defined at a location corresponding to the object 105 in space 5.

To increase the user's sense of immersion, the user terminal 100 may display VR content 107 on the entire screen of the AR view 10. This VR content 107 may depict a virtual space around the object 105. For example, when the object 105 is the “Monument to King Jinheung,” environmental content depicting a virtual space around the object 105 may depict the geographical environment in which the Monument to King Jinheung is located. The environmental content may further include text describing the object 105, as shown in FIG. 1B. Alternatively, the environmental content may depict a virtual space for reproduction of a past historical space, transformation into a space with a theme, simulation of a future space, etc., depending on the type of object 105.

The user terminal 100 may display the VR content 107 by changing the viewpoint according to a change in the viewpoint of the camera in the AR view 10. Accordingly, the user can feel a sense of immersion as if he or she is actually exploring the virtual space where the object 105 is located.

Additionally, VR content 107 may include animation effects. For example, when the object 105 is recognized and the VR content 107 is displayed in the AR view 10, the VR content 107 is expressed as an animation that gradually encroaches on the AR view 10 and reaches the entire screen. You can. Alternatively, after the AR content corresponding to the object 105 is displayed, the VR content 107 may be displayed by overlaying the above-described environmental content in the space surrounding the AR content.

This VR content 107 may be displayed while the object 105 is recognized by the user terminal 100.

In step 350, when the user terminal 100 is no longer located in the recognition area, display of the VR content 107 may be terminated. For example, when the viewpoint of the camera in the AR view 10 changes and the user terminal 100 leaves the recognition area, or when the user terminal 100 moves and leaves the recognition area, the VR view Display of the content 107 can be ended.

Animation effects may also be included at the end of display of VR content 107. For example, when the user terminal 100 leaves the recognition area of the object 105, the VR content 107 may gradually disappear from the AR view 10 and the original AR view 10 may be displayed.

In an embodiment, custom areas 60, 70 have at least one service element associated with them. Additionally, scenario conditions for controlling the user terminal 100 based on service elements may be set in the custom area as the user terminal 100 is located in the custom areas 60 and 70. The scenario conditions may include conditions for providing services or information related to the space 5 or controlling settings of the user terminal 100, based on service elements associated with the custom areas 60 and 70.

The display of VR content 107 in the above-described AR view 10 may also be based on service elements associated with the custom areas 60 and 70 and scenario conditions set for the custom areas 60 and 70.

As the user terminal 100 or the server 200 determines that the user terminal 100 is located in the custom area, the user terminal 100 may be controlled based on service elements associated with the custom area 60 and 70. You can. In other words, the user terminal 100 may be controlled to execute a specific operation based on data included in service elements associated with the custom areas 60 and 70.

Here, “control” of the user terminal 100 refers to content representing services or information related to the space 5 on the image of the AR view 10 using the AR view 10 of the user terminal 100 (e.g., AR augmented display of content 30 and/or VR content 107, etc.); Displaying a user interface (UI) related to a service associated with the space 5 on the user terminal 100; displaying map element content 35 related to a service associated with space 5 on the map (or map view 20) of the user terminal 100; It may include at least one of controlling settings of the user terminal 100 in order to provide the user terminal 100 with a service related to the space 5.

Service elements associated with the custom areas 60 and 70 may include data for displaying AR content 30 related to the service associated with the space 5 .

Additionally, service elements associated with the custom areas 60 and 70 may include data for displaying VR content 107 related to the space 5 and services associated with the space. Accordingly, in the embodiment, VR content 107 may be displayed in the AR view 10 based on these service elements and scenario conditions defined for the custom areas 60 and 70.

Refer to FIGS. 4 to 12 to be described later for a method of recognizing the object 105 to display VR content 107 and a method of displaying VR content 107 in the AR view 10 according to specific scenario conditions. This is explained in more detail.

Meanwhile, the method of controlling the user terminal 100 in the embodiment may be performed in combination with a route guidance service. For example, the method of controlling the user terminal 100 in the embodiment may be performed in combination with AR navigation that provides route guidance indoors or outdoors.

In step 302, the user terminal 100 may receive a request for route guidance to the destination. For example, the user of the user terminal 100 may request route guidance from the user terminal 100 by running an application or program that provides an AR navigation service or other route guidance service on the user terminal 100 and inputting a destination. . The route guidance request may be transmitted to the server 200 that provides the route guidance service.

In step 304, the user terminal 100 may provide route guidance according to the route guidance request received in step 302. For example, the user terminal 100 may receive data for route guidance to the destination from the server 200 that provides a route guidance service, and based on the received data, information for route guidance to the destination (e.g., Guide information) can be displayed.

The user of the user terminal 100 can move to the destination according to route guidance. At this time, the above-described steps 310 to 340 may be performed when the user terminal 100 moves to the destination according to the route guidance.

For example, when the user terminal 100 moves to a destination according to route guidance, it may be determined whether the user terminal 100 is located in the custom area 60 or 70 and whether the object 105 has been recognized, and the determination According to the result, the user terminal 100 may display VR content 107 in the AR view 10. Since related examples were described above in FIG. 1B, redundant descriptions will be omitted.

The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIG. 3 , so overlapping descriptions will be omitted.

FIG. 4 is a flowchart illustrating a method of controlling settings of a user terminal as the user terminal enters a recognition area that enables recognition of objects in a custom area, according to an example.

Referring to FIG. 4 , a method for recognizing the object 105 by determining whether the user terminal 100 is located in a recognition area for recognizing the object 105 will be described in more detail.

As described above, the service elements associated with the custom areas 60 and 70 may include data for controlling the settings of the user terminal 100 to provide the user terminal 100 with services associated with the space 5. .

In step 410, the user terminal 100 or the server 200 operates when the user terminal 100 enters the custom area 60 or 70 or when the user terminal 100 enters the preset recognition area. , the settings of the user terminal 100 can be controlled according to scenario conditions set for the custom areas 60 and 70.

Controlling the settings of the user terminal 100 may include controlling functions and/or operations of the user terminal 100 related to services that can be provided to the user terminal 100 within the space 5. . Controlling a function and/or operation may include turning the function and/or operation on/off or activating/deactivating the function and/or operation.

For example, when the user terminal 100 or the server 200 enters the custom area 60 or 70, the object 105 according to the scenario conditions set in the custom area 60 or 70. You can control the settings of the user terminal, including the execution of the module for recognizing. In other words, when the user terminal 100 enters the custom area 60 or 70, a module for recognizing the object 105 may be executed. The module for recognizing the object 105 may be, for example, the VOT module described above. When the user terminal 100 enters the custom area 60 or 70, a module for recognizing the object 105 of the user terminal 100 may be turned on or initiated, and accordingly, the user terminal ( 100 can recognize the object 105 within the custom areas 60 and 70. Meanwhile, when the user terminal 100 leaves the custom areas 60 and 70, the module for recognizing the object 105 of the user terminal 100 may be turned off or deactivated again. However, depending on the embodiment, the controlled settings may be maintained even if the user terminal 100 leaves the custom areas 60 and 70.

According to this embodiment, the user terminal 100 does not indiscriminately recognize and detect objects in all areas within the space 5, but recognizes and detects objects only in a specific custom area or only after entering a specific custom area. It becomes possible. Therefore, taking into account the user's movement line within the space 5 and the field of view or perspective (given that the custom area may be a three-dimensional space with a height range), the service associated with the space 5 is provided to the user terminal 100. can be provided efficiently.

While the module for recognizing the object 105 is executed, the user terminal 100 may recognize the object 105 through the AR view 10. For example, when the object 105 is displayed in the AR view 10, the user terminal 100 may determine that the user terminal 100 is located in a recognition area for recognition of the object 105, and accordingly, the object 100 (105) can be recognized. Alternatively, the user terminal 100 may operate under the condition that the distance between the object 105 and the user terminal 100 is less than a predetermined distance, and the object 105 is located in a specific area (e.g., the center area or its vicinity) within the AR view 10. ) and a condition in which the object 105 is displayed more than a certain percentage of the AR view 10 (i.e., a condition in which the object 105 occupies the AR view 10 more than a certain percentage). If this is satisfied, it can be determined that the user terminal 100 is located in the recognition area for recognition of the object 105, and accordingly, the object 105 can be recognized. Conditions for recognition of the above-described object 105 may be included in scenario conditions set for the custom areas 60 and 70.

In another embodiment, the recognition area for recognition of the above-described object 105 may be a predefined area within the custom areas 60 and 70. This predefined area may be a certain range of area around the object 105. When the user terminal 100 or the server 200 is located in the predefined recognition area, the user terminal 100 is located in the recognition area for recognition of the object 105. It can be determined that In this case, when it is determined that the user terminal 100 has entered the recognition area, the user terminal 100 or the server 200 executes a module for recognizing the object 105. ) settings can be controlled. In other words, if a recognition area, which is a predefined area, exists within the custom areas 60 and 70, the module for recognizing the object 105 can be executed only when the user terminal 100 is located in the recognition area. , Accordingly, the object 105 can be recognized. The previously defined recognition areas described above may be included in the scenario conditions set in the custom areas 60 and 70.

When the object 105 is recognized by the user terminal 100, VR content 107 may be displayed through the AR view 10 in association with the recognized object 105.

The description of the technical features described above with reference to FIGS. 1 to 3 can also be applied to FIG. 4 , so overlapping descriptions will be omitted.

FIG. 5A illustrates a method of controlling settings of a user terminal based on the user terminal being located in a preset custom area, according to an example.

Service elements associated with the custom areas 60 and 70 may include data for controlling settings of the user terminal 100 to provide services related to the space 5 .

The user terminal 100 or the server 200, when the user terminal 100 enters the custom area 60 or 70, based on the service elements associated with the custom area 60 or 70, the user terminal 100 You can run the module for object recognition.

As shown, a custom area 510-1 is set in the space 500-1, and when the user terminal 100 enters the custom area 510-1 from the location 505, the user terminal ( By controlling the settings of the user terminal 100, the module for object recognition of the user terminal 100 can be turned on or initiated.

After the module for object recognition is executed, when the user terminal 100 recognizes King Jinheung Sunbi as the object 105, VR content 107 will be displayed in association with the recognized object 105 in the AR view 10. You can.

Through this embodiment, the module for object recognition is executed only when the user terminal 100 enters the custom area 510-1, thereby preventing the user from being exposed to too much content augmented through the AR view 10. The complexity of the interface, the increase in the user's cognitive load, and the increase in the load on the user terminal 100 can be resolved.

Meanwhile, Figure 5b shows a method of providing VR content to a user terminal based on the user terminal being located in a preset custom area, according to an example.

As shown, a custom area 510-2 is set in the space 500-2. When the user terminal 100 enters the custom area 510-2, the settings of the user terminal 100 are controlled, and a module for object recognition of the user terminal 100 may be turned on or initiated. .

Meanwhile, a plurality of points A and B may be defined in the custom area 510-2. Each of the plurality of points A and B may represent an entrance for entering the space corresponding to the custom area 510-2.

The scenario conditions for the custom area 510-2 are VR content 107 when the user terminal 100 is located in the custom area 510-2, based on the point through which the user terminal 100 passed. The determined VR content 107 may be set to be displayed through the AR view 10. Accordingly, the user terminal 100 may display the VR content 107 determined based on the point through which the user terminal 100 passed through the AR view 10. For example, the user terminal 100 enters the custom area 510-2 through point A and when the user terminal 100 enters the custom area 510-2 through point B. At times, different VR contents 107 can be displayed through the AR view 10.

Meanwhile, the “point through which the user terminal 100 passed” may be a predetermined point outside the custom area 510-2 through which the user terminal 100 passed, or a point defined within the custom area 510-2. It may be a certain point.

The above-described “determined VR content” refers to displaying existing VR content in a different form, applying different animations, or applying different effects, displaying VR content with different content, or displaying additional content to existing VR content ( Additional information (voice, text, image, video, etc.) may be added.

Other examples of displaying VR content in the AR view 10 according to scenario conditions set for the custom area will be described in more detail with reference to the drawings to be described later.

The description of the technical features described above with reference to FIGS. 1 to 4 can also be applied to FIG. 5 , so overlapping descriptions will be omitted.

Figure 6 shows a method of defining AR content associated with an object in a custom area, according to one example. 7 illustrates a method of defining VR content associated with an object in a custom area, according to one example.

The illustrated AR content 610 may be defined to be output within the AR view 10 in association with the object 105, “Jinheungwang Sunsunbi.”

The AR content 610 may be placed in the UI 600 where data representing the space 5 is loaded or rendered, and may be placed in the custom areas 60 and 70 set within the space 5. AR content 610 may be included in service elements associated with custom areas 60 and 70.

The VR content 710 shown in FIG. 7 may be defined to be output within the AR view 10 in association with the object 105, “Jinheungwang Sunbi.” VR content 710 may include AR content 610 and environmental content depicting a virtual space surrounding the object 105. As shown, the virtual space around the object 105 depicted by the VR content 710 is expressed as a virtual sphere space or hemisphere space centered on the object 105 (environmental content). You can. A virtual space expressed as a virtual sphere centered on the object 105 is included in the VR content 710, allowing the user to feel immersed through the AR view 10 that displays the VR content 710. do.

The UIs 600 and 700 may include an indoor map creation UI for creating an indoor map for the space 5 when the space 5 is indoor, or may be part of the indoor map creation UI. This indoor map creation UI includes detailed spaces (exhibition rooms, etc.) within space (5); Means of transportation such as stairs, escalators, elevators, etc.; Facilities such as restrooms, signs, signs, information centers, etc.; Poles and other obstacles; path information within space (5); A function for setting nodes and links connecting nodes may be provided. For example, the user of the administrator terminal to create an indoor map and create custom areas (60, 70) on the indoor map sets the area corresponding to the custom area (60, 70) through the UI (600, 700) , Custom areas 60, 70 by directly placing AR content 610 and/or VR content 710 in a set area or selecting AR content 610 and/or VR content 710 to be placed in a set area. and AR content 610 and/or VR content 710 (i.e., service elements including them) may be associated. The UIs 600 and 700 may provide separate UIs for selection and placement of service elements to be associated with areas corresponding to the custom areas 60 and 70. Users can select and place service elements through methods such as clicking, dragging and dropping.

In the illustrated example of UI (600, 700), a map corresponding to space (5) is displayed, and it is assumed that custom areas (60, 70) are created on the map.

Meanwhile, unlike what is shown, the UIs 600 and 700 include data representing the space 5, such as a map for the space 5, point cloud data (PCD) collected for the space 5, and at least one of the images taken of the space 5 may be rendered, and custom areas 60 and 70, which are two-dimensional and/or three-dimensional spaces, may be created on the rendered data.

When the indoor map creation UI includes the UI (600, 700), the custom areas (60, 70) can be created along with the construction of the indoor map for the space (5), and therefore, the custom areas (60, 70) ) can also be performed together with the content 610 and/or the VR content 710.

The description of the technical features described above with reference to FIGS. 1 to 5 can also be applied to FIGS. 6 and 7, so overlapping descriptions will be omitted.

Figure 8 shows a method of defining AR content and VR content associated with an object in a custom area, according to an example.

As described above, in order to increase the user's sense of immersion, the user terminal 100 may display the VR content 107 on the entire screen of the AR view 10, and the VR content 107 may be displayed around the object 105. A virtual space can be described.

At this time, the virtual space depicted by the VR content 107 may be expressed as a virtual three-dimensional space determined by reflecting the geometry of the surrounding space where the object 105 is located. In other words, the virtual space depicted by the VR content 107 may reflect the shape of the actual space around where the object 105 is located.

For example, in the case where the object 810 is placed in the exhibition room space 820 as shown, and the exhibition room space 820 has a hexahedral geometry, the object displayed in the AR view 10 in association with the object 810 VR content may include AR content 830 representing an object 810 and environmental content 840 representing an exhibition room space 820. In this way, the environmental content 840 reflects the geometry of the exhibition room space 820 and may have a hexahedral shape similar to the exhibition room space 820.

As the VR content displayed in the AR view 10 reflects the geometry of the actual exhibition room space 820 where the object 810 is placed, the immersion of the user watching the VR content can be further improved.

Meanwhile, the geometry of the exhibition room space 820 is a base obtained when creating an indoor map of the space 5 including the exhibition room space 820 or collecting data (PCD, captured image, etc.) representing the space 5. It may be data from . The geometry of the exhibition room space 820 may be stored and managed in the server 200.

Accordingly, as described above with reference to FIG. 7, when placing VR content 710 in the custom area 60, 70 through UI 600, 700, the object 105 depicted by VR content 710 The virtual space around may not be a virtual spherical space or hemispherical space centered on the object 105, but may be a three-dimensional space that reflects the geometry of the actual space where the object 105 is placed. This three-dimensional space can be automatically displayed on the UI (600, 700) based on the geometry of the known real space.

The description of the technical features described above with reference to FIGS. 1 to 7 can also be applied to FIG. 8 , so overlapping descriptions will be omitted.

Below, the custom areas 60 and 70 created by defining the space 5 and the scenario conditions set for the custom areas 60 and 70 will be described in more detail.

Figure 9 shows a custom area defined in three-dimensional space, according to an example.

In Figure 9, the custom area 900 defined in three-dimensional space and the scenario conditions defined there are explained in more detail.

The above-described custom areas 60 and 70 may be defined to include at least one of a two-dimensional space and a three-dimensional space on the map of the space 5.

The user terminal 100 or the server 200, when the location of the user terminal 100 is located in the custom area 60 or 70 on the map (or when it is determined to be outside the custom area 60 or 70) , the user terminal 100 can be controlled based on service elements associated with the custom areas 60 and 70.

The custom area 900 may be defined to include a two-dimensional area and a three-dimensional space with a set height range. Alternatively, the custom area 900 may be defined as a complex dimensional space including a 2-dimensional space and a 3-dimensional space.

The two-dimensional area constituting the custom area 900 may include a two-dimensional coordinate space (x, z). The height range constituting the custom area 900 may include a y-axis coordinate space. The height range may be a range with upper and lower limits as shown (H), or may be a range without upper or lower limits as shown (H).

The user terminal 100 or the server 200 is located in a two-dimensional area constituting the custom area 900, and the height of the user terminal 100 is within a set height range constituting the custom area 900. If it belongs to, it is determined that the user terminal 100 is located in the custom area 900, and the user terminal 100 can be controlled based on service elements associated with the custom area 900. Therefore, when it is determined that the user terminal 100 is located in the custom area 900, the module for recognizing the object 105 is executed, the object 105 is recognized, and the VR content 107 is displayed in the AR view 10. ) may be displayed.

The height of the user terminal 100 may be obtained by a sensor included in the user terminal 100. The height of the user terminal 100 may represent, for example, the height of the camera of the user terminal 100.

Although one height range (H) is illustrated in FIG. 9, a plurality of height ranges may be defined in the custom area 900, and a service element may be associated with and a scenario may be defined for each height range.

In this way, by setting the custom area 900 in a three-dimensional space, even if the user terminal 100 is located in the same two-dimensional space, the user terminal 100 can be controlled according to different scenarios depending on the height of the user terminal 100. It can be.

Below, the scenarios defined for the custom areas 60 and 70 will be described in more detail with reference to FIG. 10 .

Figure 10 shows the DoF (Degree Of Freedom) of a user terminal, according to an example.

The custom areas 60 and 70 may have 3DoF or 6DoF posture conditions of the user terminal 100 defined. 6DoF can represent movement in the x-, y-, and z-axis directions, and rotation (roll, yaw, and pitch) in the axial, y-, and z-axis directions.

These attitude conditions may be included in scenario conditions defined for the custom areas 60 and 70, and as described above, these scenario conditions may be included in service elements associated with the custom areas 60 and 70.

When the user terminal 100 or the server 200 determines that the user terminal 100 is located in the custom area 60 or 70, the posture of the user terminal 100 is defined in the custom area 60 or 70. If the posture condition is satisfied, the user terminal 100 can be controlled based on service elements associated with the custom areas 60 and 70. The posture of the user terminal 100 may be acquired by a sensor included in the user terminal 100, for example, a gyro sensor.

As such posture conditions are defined in the custom areas 60 and 70, for example, the user terminal 100 enters the custom areas 60 and 70 and the AR view 10 is maintained at a specific angle or a specific angle range. Only then can the user terminal 100 be controlled based on service elements. For example, VR content that must be displayed in a specific area within the custom area 60, 70, for example, at a position above or below a certain height, is displayed when the user terminal 100 enters the custom area 60, 70 and displays an AR view. (10) can be displayed in the AR view 10 of the user terminal 100 only when it is maintained at an angle or angle range that illuminates a specific area above or below a certain height.

Accordingly, in the embodiment, VR content can be displayed at a more appropriate location and viewpoint of the AR view 10.

Below, other examples of scenario conditions defined for custom areas 60 and 70 are further described.

The custom areas 60 and 70 include at least one condition of a first condition indicating a time condition, a second condition indicating movement of the user terminal 100, and a third condition indicating an operation on the user terminal 100. Scenario conditions may be set.

The first condition, for example, requires that the user terminal 100 stay in the custom area 60, 70 for a certain time or more (e.g., 3 seconds or more), or to stay in the custom area 60, 70 for a certain time or less. It may be a condition. Alternatively, the first condition is a condition requiring the user terminal 100 to enter the custom area 60, 70 at a predetermined time period, or to enter the custom area 60, 70 at a time other than the predetermined time period. It can be. The second condition, for example, represents physical manipulation of the user terminal 100 and may be a condition requiring the user terminal 100 to be shaken a certain number of times or with a certain intensity or more. Alternatively, the second condition may be a condition requiring an action to move the user terminal 100 in a specific shape (eg, the letter '8'). The third condition may be an act of electronically manipulating the user terminal 100, for example, inputting a specific button or key, inputting a specific gesture, inputting a specific voice or audio, or photographing a specific object. It may be a condition that requires something to be done, etc.

When the user terminal 100 or the server 200 determines that the user terminal 100 is located in the custom area 60 or 70, the user terminal 100 satisfies at least one of the first to third conditions. If the in-scenario condition is satisfied, the user terminal 100 can be controlled based on service elements associated with the custom areas 60 and 70. For example, the user terminal 100 displays AR content or VR content differently, displays UI differently, displays map element content differently, or controls settings of the user terminal 100 depending on whether the scenario conditions are satisfied. can be done differently.

The description of the technical features described above with reference to FIGS. 1 to 8 can also be applied to FIGS. 9 and 10, so overlapping descriptions will be omitted.

Figure 11 shows a custom area including a plurality of points, according to an example.

The illustrated custom area 1100 may be an area defined on the map of space 5. The custom area 1100 may include a point indicating entrance A and a point indicating entrance B. Additionally, the custom area 1100 may further include a point representing object A and a point representing object B. The user terminal 100 or the server 200 can determine whether the user terminal 100 has entered or exited the custom area 1100 by tracking the location of the user terminal 100. there is.

Additionally, a plurality of points may be defined not only inside the custom area 1100 but also outside the custom area 1100.

A plurality of service elements may be associated with the custom area 1100. At this time, each service element of the plurality of service elements may be associated with a different point within the custom area 1100. Scenarios or scenario conditions defined in the custom area may be defined to provide services or information related to the space 5 differently depending on the order in which the user terminal 100 passes through points associated with each service element. Services or information provided differently may be AR content or VR content, or other types of content that can be output from the user terminal 100, such as voice, text, image, video, etc.

For example, a scenario condition that causes the user terminal 100 to output different types of content is defined in the custom area 1100 by distinguishing between cases where the user terminal 100 enters and exits the custom area 1100. It can be.

Alternatively, when the user terminal 100 enters the custom area 1100 through a specific point (e.g., at least one of the points of entrance A, entrance B, object A, and object B shown), the custom area 1100 Scenario conditions that allow the user terminal 100 to output different types of content, such as when entering the custom area 1100 after passing through a specific external point, may be defined in the custom area 1100. For example, when the user terminal 100 enters the custom area 1100 through a point indicating entrance A, general welcome content may be output to the user terminal 100, but the user terminal 100 enters the custom area 1100 through entrance B. When the custom area 1100 is entered through the indicated point, other content, for example, coupon content, may be output on the user terminal 100. Coupon content may include discount coupons or benefits available within the space (5).

In this way, in the case where a plurality of points are defined inside or outside the custom area 1100, the user terminal 100 passes through the first point among the plurality of points and the user terminal ( When 100) passes through the second point among the plurality of points, different VR contents may be displayed through the AR view 10. That is, even when the same object is recognized, the AR view is displayed depending on whether the user terminal 100 entered the custom area 1100 through a certain point or a point within the custom area 1100. Through (10), different VR contents can be displayed.

In addition, when the user terminal 100 is located in a specific custom area 60 or 70, it can be determined whether it has entered the custom area 60 or 70 or has left the custom area 60 or 70, and the decision is made. Depending on the results and set scenario conditions, the user terminal 100 may be controlled to output different VR content when entering and exiting the custom areas 60 and 70. At this time, based on the current location of the user terminal 100 and the path along which the user terminal 100 has already moved or will move, the user terminal 100 is determined to have entered the custom area 60 or 70 ([1] ), it can be determined that it will advance from the custom area 60, 70 ([2]). The user terminal 100 or the server 200 can determine which node the user terminal 100 has already passed through or which node it will pass through based on the path, and accordingly, the user terminal 100 is currently It can be determined whether the user has entered the custom area or whether to advance from the custom area. In cases [1] and [2] above, the user terminal 100 may be controlled to display different VR content. That is, even when the same object is recognized, different VR contents may be displayed through the AR view 10 depending on whether the user terminal 100 enters or exits the custom areas 60 and 70.

Alternatively, when the user terminal 100 enters the custom area 1100, an additional action is requested from the user terminal 100, such as requesting to move along the edge of the custom area 1100. As an action is performed, scenario conditions that allow the user terminal 100 to output different types of content may be defined in the custom area 1100. The request for the above additional action may also be provided to the user terminal 100 in the form of content or UI.

Alternatively, if the user terminal 100 counts the number of times it has entered the custom area 1100 and enters the custom area 1100 more than n times (n is a natural number of 2 or more) (the user terminal 100 enters the custom area ( 1300) may be counted only within a certain period of time (eg, 1 hour or 1 day), a scenario condition for outputting specific content to the user terminal 100; When the user terminal 100 stays in the custom area 1100 for a predetermined time or more, a scenario condition for outputting specific content to the user terminal 100 may be defined in the custom area 1100. For example, the scenario condition may be set so that VR content determined based on the number of times the user terminal 100 enters the custom area 1100 is displayed through the AR view. At this time, VR content determined based on the number of times the user terminal 100 has entered the custom area 1100 may be displayed through the AR view 10. That is, even when the same object is recognized, different VR contents may be displayed through the AR view 10 depending on the number of times the user terminal 100 enters the custom area 1100.

Meanwhile, when the user terminal 100 leaves the custom area 1100 (i.e., advances), all content displayed on the user terminal 100 may be non-displayed, that is, de-displayed. When the user terminal 100 leaves the custom area 1100, a predetermined notification sound may be output to the user terminal 100. Alternatively, when the user terminal 100 leaves the custom area 1100, AR content or VR content indicating that the user terminal 100 has left the custom area 1100 may be displayed in the AR view 10.

As such, in the embodiment, it may be determined whether the user terminal 100 is entering or leaving the custom area 1100, and depending on the determination result, the AR content or VR content may be displayed differently (e.g., upon entry) Augmentation of entry guide AR content or VR content, augmentation of entry guide AR content or VR content when advancing, etc.), different UI display, different display of map element content, or different setting control of the user terminal 100. there is.

The description of the technical features described above with reference to FIGS. 1 to 10 can also be applied to FIG. 11 , so overlapping descriptions will be omitted.

Figure 12 shows a method in which a plurality of custom areas are set in a space and scenario conditions for controlling a user terminal are set for each custom area, according to an example.

Below, other examples of scenario conditions defined for each of the plurality of custom areas 1210 to 1240 set in the space 1200 will be further described.

The above-described custom areas 60 and 70 may be any one of the custom areas 1210 to 1240.

In any one of the custom areas 60 and 70 of the custom areas 1210 to 1640, i) a second message indicating whether the user terminal 100 passed through a custom area other than the custom area 60 or 70; At least one scenario condition may be defined among 1 scenario condition and ii) a second scenario condition indicating whether the number of user terminals located in the custom areas 60 and 70 falls within a predetermined range. Depending on whether the corresponding scenario conditions are satisfied, the user terminal 100 is provided with services or information associated with the space 5, 1200 differently, based on service elements associated with the custom area 60, 70, or the user terminal The setting of 100 can be controlled differently.

The first scenario condition may be a condition that requires the user terminal 100, which has entered the custom area 60 or 70, to enter the custom area 60 or 70 after entering another specific custom area. For example, when the user terminal 100 enters a specific custom area among the custom areas 1220 to 1640 and then enters the custom area 1210 (within a predetermined time), the first scenario condition may be satisfied. Alternatively, when the user terminal 100 enters the custom area 1210 (within a predetermined time) after entering at least some of the custom areas 1210 to 1240 in a specific order, the first scenario condition may be satisfied. there is. The user terminal 100 displays AR content or VR content differently, displays UI differently, displays map element content differently, or controls settings of the user terminal 100 depending on whether the first scenario condition is satisfied. can be done differently.

For example, the scenario condition is, when the user terminal 100 is located in the custom area 60, 70, based on another custom area that the user terminal 100 entered before the user terminal 100 entered the custom area. The determined VR content may be set to be displayed through the AR view 10. At this time, VR content determined based on the other custom area may be displayed through the AR view 10 of the user terminal 100. That is, even when the same object is recognized, depending on whether the user terminal 100 entered another custom area before entering the custom area or whether it entered a specific other custom area, through the AR view 10 , different VR contents may be displayed.

The second scenario condition is that the number of user terminals located in the custom areas 60 and 70 or the cumulative number of user terminals entering the custom areas 60 and 70 (over a predetermined period of time) is greater than or equal to a certain number or less than a certain number. It may be a condition requiring something. Alternatively, the second scenario condition is the number of user terminals located in a custom area other than the custom area 60, 70 into which the user terminal 100 enters, or the number of user terminals that have entered the other custom area (for a predetermined period of time) ui) It may be a condition that requires the cumulative number to be more than a certain number or less than a certain number. Alternatively, the second scenario condition is the number of user terminals located in both the custom areas 60 and 70 and other custom areas, or the accumulation (over a predetermined time) of the user terminals that have entered the two custom areas. It may be a condition that requires the number to be greater than a certain number or less than a certain number. The user terminal 100 displays AR content or VR content differently, displays UI differently, displays map element content differently, or controls settings of the user terminal 100 depending on whether the second scenario condition is satisfied. can be done differently.

The above-described first scenario condition and second scenario condition may be used in combination with each other. For example, the user terminal 100 or the server 200 may determine whether the first scenario condition and the second scenario condition are satisfied when the user terminal 100 enters the custom area 60 or 70. and only when both scenario conditions are satisfied, the user terminal 100 can be controlled based on the service element. Alternatively, control of the user terminal 100 based on service elements may be varied depending on which condition of the first scenario condition and the second scenario condition is satisfied.

For example, the user terminal 100 displays VR content through the AR view 10 only when the scenario conditions are satisfied, or distinguishes between cases where the scenario conditions are satisfied and cases where the scenario conditions are not satisfied, and displays the AR view ( 10), different VR contents can be displayed.

The above-described “determined VR content” refers to displaying existing VR content in a different form, applying different animations, or applying different effects, displaying VR content with different content, or displaying additional content to existing VR content ( Additional information (voice, text, image, video, etc.) may be added.

The description of the technical features described above with reference to FIGS. 1 to 11 can also be applied to FIG. 12 , so overlapping descriptions will be omitted.

As such, in embodiments, a “rendering range”-based method of displaying AR/VR content or controlling the user terminal 100 based on the distance or radius of the position of an object located in space Or, a "Semantic Range"-based method that divides space into semantic unit areas and displays AR/VR content or controls the user terminal 100 when the user terminal 100 enters the unit space. In comparison, a “Custom Range” method that can display AR/VR content on the user terminal 100 and control the user terminal 100 in a more user-friendly and natural manner can be used. The custom range may use the “custom area” described with reference to the embodiments. The “custom range” method can utilize a custom area in which service elements are associated and scenario conditions are defined for display of AR/VR content and control of the user terminal 100.

When using a “rendering range”-based method or a “semantic range”-based method, AR/VR content is displayed inappropriately at the position of the object within the AR view 10 depending on the location of the user terminal 100. This may occur, and in some cases, a problem may occur where AR/VR content is displayed in an occlusion area. In the “custom range” method described with reference to the embodiments, this problem can be solved by utilizing a custom area.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for the purpose of being interpreted by or providing instructions or data to the processing device. there is. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (18)

In a method of controlling a user terminal performed by a computer system that is a server or a user terminal,
Based on the location of the user terminal, determining whether the user terminal is located in a preset custom area within the space;
determining whether the user terminal is located in a recognition area that allows the user terminal to recognize objects in the custom area;
When the user terminal is located in the recognition area, recognizing the object through an AR view including an image captured by a camera of the user terminal; and
Displaying preset virtual reality (VR) content in association with the recognized object through the AR view.
A method of controlling a user terminal, including. According to paragraph 1,
At least one service element is associated with the custom area,
The service element includes data for display of the VR content related to a service associated with the space,
A method for controlling a user terminal, wherein scenario conditions for controlling the user terminal are defined in the custom area based on the service element as the user terminal is located in the custom area. According to paragraph 2,
The service element includes data for controlling settings of the user terminal to provide the user terminal with a service associated with the space,
When the user terminal enters the custom area, controlling settings of the user terminal, including execution of a module for recognizing the object according to the scenario conditions.
A method of controlling a user terminal, further comprising: According to paragraph 1,
The step of determining whether the user terminal is located in the recognition area includes:
A method of controlling a user terminal, wherein when the object is displayed in the AR view, it is determined that the user terminal is located in the recognition area. According to paragraph 1,
The step of determining whether the user terminal is located in the recognition area includes:
At least one condition of a condition in which the distance between the object and the user terminal is less than or equal to a predetermined distance, a condition in which the object is displayed in a specific area in the AR view, and a condition in which the object is displayed in a predetermined ratio or more in the AR view. If satisfied, a method for controlling a user terminal determines that the user terminal is located in the recognition area. According to paragraph 1,
The recognition area is a predefined area within the custom area,
The step of determining whether the user terminal is located in the recognition area includes:
When the location of the user terminal is located in the predefined recognition area, it is determined that the user terminal is located in the recognition area. According to clause 6,
When the user terminal enters the recognition area, controlling settings of the user terminal, including execution of a module for recognizing the object.
A method of controlling a user terminal, further comprising: According to paragraph 1,
The step of displaying the VR content is,
Displaying the VR content by changing the viewpoint according to a change in the viewpoint of the camera,
If the user terminal is no longer located in the recognition area, terminating display of the VR content
A method of controlling a user terminal, further comprising: According to paragraph 1,
The step of displaying the VR content is,
Displaying the VR content on the entire screen of the AR view,
A method of controlling a user terminal, wherein the VR content depicts a virtual space around the object. According to clause 9,
A method of controlling a user terminal, wherein the virtual space is expressed as a virtual sphere space centered on the object. According to clause 9,
The virtual space is expressed as a virtual three-dimensional space determined by reflecting the geometry of the surrounding space where the object is located. According to paragraph 2,
The scenario condition is that when the user terminal is located in the custom area, the VR content, which is determined based on another custom area that the user terminal entered before entering the custom area, is the AR. is set to be displayed through the view,
The step of displaying the VR content is,
A method of controlling a user terminal, displaying VR content determined based on the different custom area through the AR view. According to paragraph 2,
The scenario condition is set so that when the user terminal is located in the custom area, the VR content, which is determined based on the point through which the user terminal passed, is displayed through the AR view,
The step of displaying the VR content is,
A method of controlling a user terminal, displaying VR content determined based on a point passed by the user terminal through the AR view. According to clause 13,
A plurality of points are defined inside or outside the custom area,
The step of displaying the VR content is,
When the user terminal passes through a first point among the plurality of points and when the user terminal passes through a second point among the plurality of points, different VR contents are displayed through the AR view. , A method of controlling a user terminal. According to paragraph 2,
A plurality of custom areas are set in the space,
The scenario conditions include: i) a first scenario condition indicating whether the user terminal has passed through a custom area other than the custom area, and ii) whether the number of user terminals located in the custom area falls within a predetermined range. Includes at least one of the second scenario conditions representing,
The scenario conditions are set to provide different services or information related to the space or control settings of the user terminal differently based on the service elements to the user terminal, depending on whether the scenario conditions are satisfied,
The step of displaying the VR content is,
A method for controlling a user terminal, displaying the VR content through the AR view only when the scenario condition is satisfied, or displaying different VR content when the scenario condition is satisfied and when the scenario condition is not satisfied. . According to paragraph 1,
Receiving, at the user terminal, a request for route guidance to a destination; and
Providing route guidance to the user terminal according to the route guidance request.
It further includes,
Determining whether the user terminal is located in the custom area, determining whether the user terminal is located in the recognition area, recognizing the object, and displaying the VR content include: A method of controlling a user terminal, which is performed when the user terminal moves to the destination according to guidance. According to paragraph 2,
The scenario condition is set so that VR content determined based on the number of times the user terminal enters the custom area is displayed through the AR view,
The step of displaying the VR content is,
A method of controlling a user terminal, displaying VR content determined based on the number of times the user terminal has entered the custom area through the AR view. In a computer system implementing a user terminal or server,
At least one processor configured to execute computer-readable instructions contained in memory
Including,
The at least one processor,
Based on the location of the user terminal, it is determined whether the user terminal is located in a preset custom area in space, and the user terminal is located in a recognition area that allows the user terminal to recognize objects in the custom area. When the user terminal is located in the recognition area, the object is recognized through an AR view including an image captured by the camera of the user terminal, and the recognized object and A computer system that displays associated, preset virtual reality (VR) content.

Images