KR102549072B1 - Method and apparatus for user interaction based on digital twin - Google Patents

Abstract

A digital twin-based user interaction method and device are provided. A digital twin-based user interaction method according to an embodiment of the present invention is performed by a computing device and includes storing digital twin information corresponding to a real space, and providing a virtual reality environment to a first user terminal based on the digital twin information. providing a first user's interaction with the virtual reality environment, and providing the received first user's interaction to a second user terminal, wherein the first user's interaction includes a location movement of the first user, and the second user terminal displays a first object corresponding to the first user on a display of the second user terminal in response to an interaction of the first user. Displays movement to the corresponding position.

Description

Method and apparatus for user interaction based on digital twin {METHOD AND APPARATUS FOR USER INTERACTION BASED ON DIGITAL TWIN}

The present invention relates to a digital twin-based user interaction method and apparatus. More specifically, it relates to a method and apparatus for performing interaction between users using a virtual space constructed based on a digital twin.

Virtual Reality (VR) and Augmented Reality (AR) are attracting attention as tools to share and expand human experiences.

Virtual reality creates a virtual space with virtual images composed of backgrounds and objects, but augmented reality is a technology that overlays a 3D virtual image on a real image to show it as a single image.

The virtual reality application interacts in a virtual space according to the user's actions delivered through separate equipment such as virtual reality glasses and motion recognition devices, and provides an experience environment to the user. Augmented reality applications are highly realistic because objects in the real world and virtual objects interact with each other.

Recently, various applications using virtual reality and augmented reality technologies have been developed due to the development of computer graphics technologies such as 3D rendering and software and hardware technologies such as high-speed/parallel processing technologies. Various industries such as games and industries are also interested in developing applications using virtual reality and augmented reality technologies.

Republic of Korea Patent Registration No. 10-2163378 (Announced on October 8, 2020)

A technical problem to be solved through some embodiments of the present invention is to provide a digital twin-based user interaction method and device that allows users to exchange interactions with each other based on a virtual reality space built with digital twin technology. .

Another technical problem to be solved through some embodiments of the present invention is to provide a digital twin-based user interaction method and apparatus capable of exchanging natural interactions between virtual reality terminal users and augmented reality terminal users.

Another technical problem to be solved through some embodiments of the present invention is to provide a digital twin-based user interaction method and device that can track the location and movement of the other user and accurately express them on a virtual reality terminal or augmented reality terminal. is to do

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the above technical problem, a digital twin-based user interaction method performed by a computer device according to embodiments of the present invention includes storing digital twin information corresponding to a real space, based on the digital twin information Providing a virtual reality environment to a first user terminal, receiving a first user's interaction with respect to the virtual reality environment, and providing the received first user's interaction to a second user terminal. and the interaction of the first user includes movement of the location of the first user, and the second user terminal corresponds to the first user on the display of the second user terminal in response to the interaction of the first user. The first object may be moved and displayed to a location corresponding to the location movement.

As an embodiment, the first user terminal displays a virtual reality background corresponding to the real space based on the virtual reality environment, and displays a second object representing a second user on the virtual reality background, A location on the virtual reality background where the second object is displayed may correspond to a location of the second user in the real space.

As an embodiment, the second user terminal displays an augmented reality background using an image captured through a camera of the second user terminal, and displays the first object on the augmented reality background, A location on the augmented reality background where an object is displayed may correspond to a location of the first user in the virtual reality environment.

As an example, the interaction of the first user may include chatting, conversation, gesture, or status display of the first user.

As an embodiment, the receiving of the interaction of the first user may include receiving Location Based Service (LBS) information or Global Positioning System (GPS) information indicating the location of the first user, and the LBS information and Calculating an error between the GPS information may be included.

As an embodiment, the providing to the second user terminal may provide the location information of the first user to the second user terminal based on the LBS information if the LBS information is receivable.

As an embodiment, in the providing of the LBS information to the second user terminal, if the LBS information is not receivable, the location information of the first user may be provided to the second user terminal based on the GPS information.

As an embodiment, the location information of the first user may be a value obtained by correcting the GPS information based on an error between pre-calculated LBS information and GPS information.

As an embodiment, the method may further include receiving an interaction of a second user from the second user terminal and providing the received interaction to the first user terminal.

As an embodiment, the first user terminal or the second user terminal may display a map guiding a movement path of the second user terminal from a first point to a second point.

In order to solve the above technical problem, a digital twin-based user interaction method performed by a computing device according to embodiments of the present invention includes receiving a virtual reality environment based on digital twin information from a server, the virtual reality environment Displaying a virtual reality background corresponding to the real space based on the above, displaying an object representing the user on the virtual reality background, and receiving the user's interaction from the server and displaying it on the virtual reality background. The virtual reality background may simulate the real space where the user is located, and the position of the object on the virtual reality background may correspond to the position of the user on the real space.

As an embodiment, in the displaying of the object representing the user, if the position of the object on the virtual reality background is within a predetermined viewing angle range, the object may be displayed as a character model.

As an embodiment, in the displaying of the object representing the user, if the position of the object on the virtual reality background is outside the predetermined viewing angle range, the object may be displayed as a name or symbol at the edge of the viewing angle range.

In order to solve the above technical problem, a digital twin-based user interaction method performed by a computing device according to embodiments of the present invention displays an augmented reality background on a screen using an image taken in a real space through a camera. and displaying an object representing the user on the augmented reality background, and receiving an interaction of the user from the server and displaying the user interaction on the augmented reality background, wherein the augmented reality object is displayed. A location on the background may correspond to a location of the user on a virtual reality space corresponding to the real space.

As an embodiment, in the displaying of the object representing the user, if the location of the object on the augmented reality background is within the screen, the object may be displayed as a character model.

As an embodiment, in the displaying of the object representing the user, a display angle of the character model may be adjusted using a gyro sensor.

As an embodiment, in the displaying of the object representing the user, if the location of the object on the augmented reality background is outside the screen, the object may be displayed as a name or symbol at an edge of the screen.

In order to solve the above technical problem, a digital twin-based user interaction device according to embodiments of the present invention includes a processor, a memory for loading a computer program executed by the processor, and a storage for storing the computer program. The computer program includes an operation of storing digital twin information corresponding to a real space, an operation of providing a virtual reality environment to a first user terminal based on the digital twin information, and a first user for the virtual reality environment. and instructions for performing an operation of receiving an interaction of the first user and providing an operation of providing the received interaction of the first user to a second user terminal, wherein the interaction of the first user causes a location movement of the first user. The second user terminal may move and display a first object corresponding to the first user to a position corresponding to the position movement on a display of the second user terminal in response to an interaction of the first user. there is.

In order to solve the above technical problem, a digital twin-based user interaction device according to embodiments of the present invention includes a processor, a memory for loading a computer program executed by the processor, and a storage for storing the computer program. The computer program includes an operation of receiving a virtual reality environment based on digital twin information from a server, an operation of displaying a virtual reality background corresponding to a real space based on the virtual reality environment, and a user on the virtual reality background. and instructions for performing an operation of displaying an object representing a , and an operation of receiving an interaction of the user from the server and displaying the received interaction on the virtual reality background, wherein the virtual reality background represents the real space where the user is located. The location of the object on the virtual reality background may correspond to the location of the user on the real space.

In order to solve the above technical problem, a digital twin-based user interaction device according to embodiments of the present invention includes a processor, a memory for loading a computer program executed by the processor, and a storage for storing the computer program. Including, but the computer program displays an augmented reality background on a screen using an image taken of a real space through a camera, displays an object representing a user on the augmented reality background, and the server A location on the augmented reality background where the object is displayed is a location of the user on a virtual reality space corresponding to the real space. can correspond with

According to the embodiments of the present invention described above, interactions between users can be exchanged based on a virtual reality space built with digital twin technology.

In addition, natural interactions can be exchanged between the virtual reality terminal user and the augmented reality terminal user, and the position and movement of the other user can be tracked and accurately expressed on the virtual reality terminal or augmented reality terminal.

1 is a diagram illustrating an entire system in which a digital twin-based user interaction method is performed according to an embodiment of the present invention.
2 is a diagram for explaining the concept of a digital twin.
FIG. 3 is a diagram illustrating an exemplary form of a first user terminal of FIG. 1 .
FIG. 4 is a diagram illustrating an exemplary form of a second user terminal of FIG. 1 .
5 is a flowchart illustrating a digital twin-based user interaction method according to an embodiment of the present invention.
6 is a diagram illustrating a method of adjusting a substitution ratio in a virtual reality environment according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating step S130 of FIG. 5 in more detail.
FIG. 8 is a diagram for amplifying step S132 of FIG. 7 .
9 is a flowchart illustrating a digital twin-based user interaction method according to another embodiment of the present invention.
FIG. 10 is a diagram for explaining a method of displaying an object on a first user terminal in the embodiment of FIG. 9 .
11 is a flowchart illustrating a digital twin-based user interaction method according to another embodiment of the present invention.
12 and 13 are diagrams for explaining a method of displaying an object on a second user terminal in the embodiment of FIG. 11 .
14 is a diagram illustrating an embodiment in which a map is also displayed on a second user terminal.
15 is a block diagram illustrating an exemplary hardware configuration of a computing device in which various embodiments of the invention may be implemented.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the technical idea of the present invention is not limited to the following embodiments and can be implemented in various different forms, only the following embodiments complete the technical idea of the present invention, and in the technical field to which the present invention belongs It is provided to fully inform those skilled in the art of the scope of the present invention, and the technical spirit of the present invention is only defined by the scope of the claims.

In adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined. Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being “connected,” “coupled to,” or “connected” to another element, that element is directly connected or connectable to the other element, but there is another element between the elements. It will be understood that elements may be “connected”, “coupled” or “connected”.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an entire system in which a digital twin-based user interaction method is performed according to an embodiment of the present invention. Referring to FIG. 1 , a server 100 , a first user terminal 200 , and a second user terminal 300 may be interconnected through a network 10 . In this embodiment, it is assumed that the first user terminal 200 is a virtual reality terminal and the second user terminal 300 is an augmented reality terminal.

The first user terminal 200 or the second user terminal 300 may be implemented as a portable terminal or a portable computer. Here, the portable terminal is a wireless communication device that guarantees portability and mobility, and includes Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), and Personal Digital Communication (PDA). Assistant), International Mobile Telecommunication (IMT)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, etc. )-based wireless communication device. Portable computers may include notebooks, laptops, and the like.

In addition, the first user terminal 200 or the second user terminal 300 may be various smart devices such as a smart phone, a smart note, a tablet PC, and a wearable computer.

The network 10 may be implemented as a wired network such as a Local Area Network (LAN), a Wide Area Network (WAN), or a Value Added Network (VAN). Alternatively, the network 10 may be any type such as a mobile radio communication network, a satellite communication network, Bluetooth, Wibro (Wireless Broadband Internet), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), and the like. can be implemented as a wireless network of If necessary, the network 10 may be configured as a mixed wired and wireless network.

The server 100 stores digital twin information corresponding to real space and provides a virtual reality environment to the first user terminal 200 based on the digital twin information.

The digital twin refers to a virtual reality environment that simulates a real space, and means that the background and components of the real space are implemented in virtual reality. For example, as shown in FIG. 2 , a digital twin is implemented on a virtual reality space 30 by replicating a building, tree, structure, etc. of the real space 20 similarly to the real thing. Digital twin information is defined to generally refer to the information used to implement the digital twin.

The virtual reality environment collectively refers to digital elements used to implement virtual reality, and includes virtual reality backgrounds, virtual objects, and other user experiences (UX).

The server 100 transfers user interactions between the first user terminal 200 and the second user terminal 300 .

For example, the server 100 receives location information of the first user from the first user terminal 200, calculates a location corresponding thereto in real space, and provides it to the second user terminal 300. The second user terminal 300 displays a first object representing the first user on the augmented reality background with reference to the location provided by the server.

And, the server 100 receives the interaction of the first user from the first user terminal 200 and provides it to the second user terminal 300 . The second user terminal 300 displays the provided interaction of the first user to the second user through its own screen. In this case, the interaction of the first user may include movement of the first user's location, conversation, chatting, or gestures.

Alternatively, the server 100 receives location information of the second user from the second user terminal 300 , calculates a location corresponding thereto in the virtual reality space, and provides the calculated location to the first user terminal 200 . The first user terminal 200 displays a second object representing the second user on the virtual reality background with reference to the location provided by the server.

And, the server 100 receives the interaction of the second user from the second user terminal 300 and provides it to the first user terminal 200 . The first user terminal 200 displays the provided interaction of the second user to the first user through a virtual reality environment provided by the first user terminal 200 . In this case, the second user's interaction may include location movement, conversation, chatting, or gestures of the second user.

FIG. 3 is a diagram illustrating an exemplary form of a first user terminal of FIG. 1 .

In the embodiment of FIG. 3 , the first user terminal 200 is illustrated as a Head Mounted Display (HMD). When the user wears the first user terminal 200 on the head, the first user terminal 200 displays the virtual reality environment 40 to the user based on the information provided by the server.

Meanwhile, in this embodiment, the first user terminal 200 is an HMD, but the scope of the present invention is not limited thereto. For example, in addition to the HMD, the first user terminal 200 may be virtual reality glasses, a hologram display, or a virtual reality experience device directly acting on human brain waves.

FIG. 4 is a diagram illustrating an exemplary form of a second user terminal of FIG. 1 .

In the embodiment of FIG. 4 , the second user terminal 300 is illustrated as a smart phone. The second user terminal 300 photographs the real space 51 through a built-in camera (not shown), and displays the photographed image as a background on its screen. In addition, an augmented reality environment is provided to the user by displaying the virtual object overlaid on the real space image on the screen.

Meanwhile, in this embodiment, the second user terminal 300 is a smart phone, but the scope of the present invention is not limited thereto. For example, the second user terminal 300 may be a portable terminal equipped with an HMD, augmented reality glasses, or other display means in addition to a smart phone.

5 is a flowchart illustrating a digital twin-based user interaction method according to an embodiment of the present invention. The user interaction method of FIG. 5 is performed by the server 100 of FIG. 1 . Therefore, in the following steps, if the performing subject is not specified, it is assumed that the performing subject is the server 100 .

In step S110, digital twin information corresponding to the real space is stored.

In step S120, a virtual reality environment is provided to the first user terminal based on the digital twin information.

Based on the provided virtual reality environment, the first user terminal may display a virtual reality background to the first user and display a plurality of virtual objects thereon.

As an example, the virtual object may be an object representing a real object existing in a real space. For example, when a second user is located in a specific real space, the first user terminal displays an image simulating the corresponding real space as a virtual reality background and displays a virtual object representing the second user on the displayed virtual reality background. can do.

In this case, the position of the virtual object on the virtual reality background may be determined based on the position of the real object in the real space. In this embodiment, since the virtual reality background simulates the real space, each location within the virtual reality background may correspond to a specific location on the real space. In this case, if the real object is located at the first position on the real space, the virtual object representing the real object is displayed at the second position corresponding to the first position on the virtual reality background.

As an embodiment, the real object may be a second user using an augmented reality terminal, for example, the second user terminal 300 of FIG. 1 , and the virtual object may be a 3D character or a representative image representing the second user. can

In step S130, a first user's interaction with the virtual reality environment is received from the first user terminal.

Interactions of the first user refer to actions, gestures, expressions, or other actions of the first user generated in response to a virtual reality environment provided by the first user terminal. It is received by the server through the terminal.

As an example, the interaction of the first user may include moving the location of the first user. For example, when the first user moves a location in a real space or a virtual reality environment in which the first user is located, information representing the location movement of the first user, for example, the changed location coordinates of the first user, is used as the first user's interaction. is received by the server as

In step S140, the received interaction of the first user is provided to the second user terminal. The second user terminal responds to the provided interaction of the first user and displays it to the second user.

In this regard, a case in which the second user terminal is an augmented reality terminal will be further described as an example.

First, the second user terminal photographs a real space through a camera, and displays an augmented reality background on a screen using the photographed image. And, the second user terminal displays an augmented reality object corresponding to the first user on the augmented reality background displayed on the screen. In this case, a location on an augmented reality background where the augmented reality object is displayed may correspond to a location of the first user in a virtual reality environment.

For example, if a first user is standing in front of a convenience store A in a virtual reality environment, the augmented reality object is displayed in front of the convenience store A on the augmented reality background of the second user terminal.

Next, when the second user terminal receives the first user interaction, the first user interaction is displayed through the current augmented reality screen in response thereto. For example, when the interaction of the first user is the movement of the location of the first user, the second user terminal moves the augmented reality object of the first user displayed on the screen of the second user terminal in response to the interaction of the first user. Move to the position corresponding to .

For example, if a first user in front of convenience store A moves to an alley next to the convenience store A in a virtual reality environment, the second user terminal responsively sets the location of the augmented reality object on the augmented reality background to the alley next to the convenience store A. display by moving to

Meanwhile, although the first user's interaction is location movement of the first user as an example, the scope of the present invention is not limited thereto. For example, the interaction of the first user may include chatting, conversation, gestures, or status display of the first user in addition to location movement.

In step S150, a second user's interaction is received from the second user terminal, and the received second user's interaction is provided to the first user terminal.

When the second user confirms the first user interaction, the second user may generate an interaction toward the first user in response thereto. Step S150 is a step related to this, and the server receives the second user interaction from the second user terminal and provides it to the first user terminal.

Similar to the first user interaction, the second user interaction refers to a second user's action, gesture, expression, or other action generated in response to an augmented reality environment provided by the second user terminal. When the second user interaction is received by the first user terminal, the first user terminal displays it on the virtual reality environment displayed by itself.

6 is a diagram illustrating a method of adjusting a substitution ratio in a virtual reality environment according to an embodiment of the present invention. 6 illustratively illustrates a method of determining a virtual object substitution ratio when a server or a first user terminal builds and implements a virtual reality environment. Here, the substitution ratio may be defined as a ratio of the position and size of the virtual object in the virtual reality environment to the position and size of the real object in the real space when the real object in the real space is expressed as a virtual object in the virtual reality environment.

First, when a specific virtual reality background 60 is set, a plurality of random points within the virtual reality background 60 are specified. Here, it is assumed that two points A and B are specified, and the coordinates of each point on the virtual reality background 60 are A(100, 100) and B(200, 200).

Then, the coordinates of the real space corresponding to the specified point are confirmed. For example, it is assumed that the real space coordinates are latitude and longitude, that the real space coordinates of point A are latitude 50 degrees and longitude 50 degrees, and the real space coordinates of point B are latitude 60 degrees and longitude 60 degrees.

Thereafter, when substituting another virtual object on the virtual reality background 60, the substitution ratio of the virtual object is determined based on the coordinates of points A and B on the virtual reality background 60 and their corresponding coordinates in real space. It is decided.

For example, if a tree, which is a real object, is located at latitude 55 degrees and longitude 55 degrees, the tree is located midway between the real space coordinates of point A and the real space coordinates of point B in real space. In this case, if the substitution ratio of the virtual object 61 representing the tree is determined to be the same as the substitution ratio of points A and B, the virtual object 61 on the virtual reality background 60 is halfway between points A and B. It is located at the location (150, 160).

Meanwhile, the size of the virtual object 61 may also be determined based on the substitution ratio of points A and B. If the ratio of the distance on the virtual reality background 60 and the distance on the real space between points A and B is calculated, since the accumulation of the virtual reality background 60 and the real space can be calculated, the size of the tree is as large as the calculated accumulation ratio. The reduced value of may be the size of the virtual object 61.

FIG. 7 is a flowchart illustrating step S130 of FIG. 5 in more detail. In this embodiment, a specific method for the server to receive the location of the first user terminal is described. In this embodiment, the method for receiving the location of the first user terminal is mainly described, but the same method may be applied to the location receiving method for the second user terminal.

In step S131, Location Based Service (LBS) information and/or Global Positioning System (GPS) information indicating the location of the first user is received.

LBS information is information obtained by measuring the location of the first user terminal through a communication company base station, and detailed configuration and operation of the LBS are well known in the art, and detailed description thereof will be omitted.

The GPS information is information obtained by measuring the position of the first user terminal through a satellite system, and detailed configuration and operation of the GPS are well known in the art, and detailed description thereof will be omitted.

Generally, the accuracy of LBS information is higher than that of GPS information. On the other hand, in terms of reception stability, GPS information is more stable than LBS information. This is because, in the case of LBS information, it is more influenced by geographic features and the connection between the first user terminal and the base station may be interrupted depending on the communication situation.

Either LBS information or GPS information may be received alternatively, but in this embodiment, it is assumed that both LBS information and GPS information are received.

In step S132, an error between LBS information and GPS information is calculated. This will be further described with reference to FIG. 8 .

8 is a diagram illustrating a specific method of calculating an error between LBS information and GPS information. As shown in the table 70 on the left of FIG. 8, it is assumed that LBS information and GPS information indicating the location of the first user terminal are continuously received. In this case, the server calculates an error between LBS information and GPS information at each point in time. Then, an average value of the errors calculated so far, that is, an average error is calculated.

The average error may be used for correcting GPS information when determining the location of the first user terminal based on the GPS information. About this, it demonstrates concretely.

The server provides location information of the first user terminal to the second user terminal based on the LBS information and/or GPS information. For example, as in step S140 above, when the interaction of the first user is provided to the second user terminal, location information of the first user terminal may also be provided.

At this time, when the LBS information can be received, the server provides the location information of the first user to the second user terminal based on the LBS information first. This is because LBS information is more accurate than GPS information.

On the other hand, if the LBS information is not receivable, the server provides the location information of the first user to the second user terminal based on the GPS information. However, in order to supplement the accuracy of the GPS information, the server corrects the GPS information based on an error between the pre-calculated LBS information and the GPS information and provides it as the location information of the first user. For example, when the GPS information is (20, 20), (20, 19) obtained by subtracting the average error (0, 1) calculated in advance is provided as the location information of the first user.

As an embodiment, even after LBS information is unreceivable and the location of the first user is provided based on GPS information, when reception of the LBS information is resumed, the location of the first user may be provided again based on the LBS information.

As an embodiment, when both LBS information and GPS information cannot be received, the server provides location information of the first user based on the LBS information or GPS information that was finally received, or determines that the location information of the first user is not confirmed. The message is provided to the second user terminal. In this case, the second user terminal may output the virtual object at the finally confirmed location of the first user and also output a preset default animation or hand tracking motion.

9 is a flowchart illustrating a digital twin-based user interaction method according to another embodiment of the present invention. The user interaction method of FIG. 9 is performed by the first user terminal 200 of FIG. 1 . Therefore, in the following steps, if the performing entity is not specified, it is assumed that the performing entity is the first user terminal 200 .

In step S210, a virtual reality environment based on the digital twin information is received from the server.

In step S220, a virtual reality background corresponding to the real space is displayed based on the received virtual reality environment.

In step S230, a virtual object representing the user is displayed on the virtual reality background. In this case, the user may be a second user of a second user terminal connected to the first user terminal through a server.

In step S240, the user's interaction is received from the server and displayed on the virtual reality background.

In this case, the virtual reality background may simulate the real space where the second user is located, and the position of the virtual object on the virtual reality background may correspond to the position of the second user in the real space. Meanwhile, since the meanings and correlations of the virtual reality environment, the virtual reality background, and the real space have already been described in detail above, additional descriptions are omitted here to avoid duplication of description.

FIG. 10 is a diagram for amplifying details of steps S230 and S240 of FIG. 9 . Referring to FIG. 10 , an example in which a virtual object 81 representing a second user is displayed on a virtual reality background 80 is illustrated.

In this embodiment, if the position of the virtual object on the virtual reality background is within the viewing angle range set in the first user terminal, the corresponding virtual object is displayed on the virtual reality background as a character model. On the other hand, if the position of the virtual object on the virtual reality background is outside the viewing angle range set in the first user terminal, the corresponding virtual object is displayed as a name or symbol at the edge of the viewing angle range.

For example, referring to FIG. 10 , a virtual object 81 representing a second user is positioned on a virtual reality background 80 within a viewing angle range of a first user terminal, and in this case, a character on the virtual reality background 80 displayed as a model. In this case, the character model may be a 2D or 3D character model.

As an embodiment, the name of the virtual object 81 displayed as a character model on the virtual reality background 80 may be displayed nearby (herein, 'mammo').

On the other hand, the virtual objects 82, 83, and 84 representing other users are positioned on the virtual reality background 80 outside the viewing angle range of the first user terminal, and the corresponding virtual objects 82, 83, and 84 are names or symbols. Displayed at the edge of the viewing angle range.

For example, based on the viewing angle of the first user terminal, the virtual object 82 deviated to the left is displayed on the left end line of the viewing angle, and based on the viewing angle of the first user terminal, the virtual objects 83 and 84 deviated to the right. is displayed on the right end line of the viewing angle. In this case, when the number of virtual objects to be displayed on the end line is plural, each virtual object may be aligned and displayed on the corresponding end line. Referring to FIG. 8 , it can be seen that two virtual objects 83 and 84 deviated from the viewing angle to the right are aligned and displayed on the right end line. Even in the case of the virtual objects 82, 83, and 84 out of the viewing angle, the names of corresponding virtual objects may be displayed similarly to the virtual object 81 within the viewing angle (here, 'Piri', 'Ace', and 'Korean').

Meanwhile, the first user terminal may display the received user interaction in association with the virtual object. For example, when a text output interaction is received from the user terminal corresponding to the 'Fairy' virtual object 82, the first user terminal displays the 'Fairy' virtual object 82 on the virtual reality background 80, A 'speech bubble icon' meaning text output may be displayed in the virtual object 82 together. Alternatively, when an audio output interaction is received from the user terminal corresponding to the 'fairy' virtual object 82, the first user terminal may display a 'loudspeaker icon' indicating audio output instead of a 'speech bubble icon'. It can be displayed in the virtual object 82.

As an embodiment, a virtual object representing a first user who is a user of a first user terminal may also be displayed on a virtual reality background.

Hereinafter, a series of processes in which the first user terminal displays the virtual reality background and virtual objects will be briefly described.

First, the process of displaying the first user on the virtual reality background will be described.

1. When the first user accesses the server, the account of the first user is authenticated through a login procedure.

2. The location coordinates of the first user or previously recorded last coordinates are checked. At this time, the location coordinates of the first user may be simultaneously identified as GPS information and LBS information. When there is no previously recorded last coordinate for the first user, the default location may be determined as the location of the first user.

3. A virtual reality background including the first user's location coordinates or final coordinates is created. If the digital twin is not implemented for the area corresponding to the location coordinates of the first user or the final coordinates, 'not a service area' may be displayed and an arbitrary virtual reality background may be moved.

4. The first user enters a predetermined network channel through which communication between users is possible. When the capacity of the existing channel is exceeded, a new channel is created, and the first user can enter the new channel.

5. Virtual object information corresponding to the first user is loaded. The virtual object information may include information such as design of a 3D character model, clothing, possession money, or an inventory table.

6. A virtual reality background including the first user's location coordinates or final coordinates is displayed, and a virtual object representing the first user is displayed at a location corresponding to the first user's location coordinates or final coordinates.

7. Changes in LBS information and GPS information according to location movement of the first user terminal are recorded. At this time, LBS information or GPS information may be recorded at predetermined time intervals, and may be set to store information only up to a predetermined number of times. For example, if it is set to store LBS information or GPS information up to 10 times in accumulation, when the 11th LBS information or GPS information is received, instead of storing the 11th LBS or GPS information, the 1st LBS or GPS information may be deleted.

8. Meanwhile, movement of the virtual object may be restricted so that the virtual object cannot leave the service range of the server so that it cannot proceed further when it reaches the edge of the service range, or the virtual object may be warped to a specific point within the service range.

Next, the process of displaying the second user on the virtual reality background will be described. The second user may be a user of an augmented reality terminal connected to the first user terminal.

1. A network channel in which the first user participates is scanned, and location coordinates of a second user who participates in the corresponding channel are loaded. In this case, the location coordinates of the second user may be loaded at predetermined time intervals.

2. Virtual object information of the second user is loaded. The virtual object information may include information such as design of a 3D character model or clothing.

2. A virtual object representing the second user is displayed at a location on the virtual reality background corresponding to the second user's location coordinates or final coordinates.

4. Changes in the location coordinates of the second user are periodically checked, and when the location coordinates change, the virtual object of the second user is moved to a corresponding location and displayed.

5. When displaying the movement of the virtual object of the second user, a movement animation may be played along with the display of the movement path.

6. When there is no change in the location coordinates of the second user, 'default animation' may be reproduced on the character model of the virtual object.

Meanwhile, as an embodiment, when a new third user participates in a channel in which the first user has joined, a system message notifying this may be displayed. Alternatively, when the existing third user leaves the channel in which the first user has participated, a system message notifying this may be displayed. Alternatively, when a new channel is created in relation to the virtual reality background displayed by the first user terminal, a system message notifying this may be displayed.

11 is a flowchart illustrating a digital twin-based user interaction method according to another embodiment of the present invention. The user interaction method of FIG. 11 is performed by the second user terminal 300 of FIG. 1 . Therefore, in the following steps, if the performing entity is not specified, it is assumed that the performing entity is the second user terminal 300 .

In step S310, an augmented reality background is displayed on the screen using an image of a real space captured through a camera.

In step S320, an augmented reality object representing a user is displayed on the augmented reality background displayed on the screen. The user may be a first user of a first user terminal connected to a second user terminal via a server.

In step S330, the user's interaction is received from the server and displayed on the augmented reality background.

At this time, the augmented reality background is generated based on the image captured by the second user terminal through the camera, and the position of the augmented reality object on the augmented reality background corresponds to the position of the first user in the real space or virtual reality environment. it could be Meanwhile, since the meanings and correlations of the virtual reality environment, the virtual reality background, and the real space have already been described in detail above, additional descriptions are omitted here to avoid duplication of description.

12 and 13 are diagrams for explaining a method of displaying an object on a second user terminal in the embodiment of FIG. 11 .

In this embodiment, if the position of the augmented reality object on the augmented reality background is within the screen of the second user terminal, the augmented reality object is displayed on the augmented reality background as a character model. On the other hand, if the location of the augmented reality object on the augmented reality background is outside the screen of the second user terminal, the corresponding augmented reality object is displayed as a name or symbol at the edge of the screen.

For example, referring to FIG. 12 , the augmented reality object 91 representing the first user is located within the screen of the second user terminal on the augmented reality background 90, and in this case, the augmented reality object 91 is augmented It is displayed as a character model on the real background 90. In this case, the character model may be a 2D or 3D character model.

As an example, a name of the augmented reality object 91 may be displayed near the augmented reality object 81 displayed as a character model on the augmented reality background 90 (here, 'Six').

On the other hand, the augmented reality objects 92, 93, and 94 representing other users are located outside the screen of the second user terminal on the augmented reality background 90, and the corresponding virtual objects 92, 93, and 94 are names or symbols. displayed on the edge of the screen.

For example, based on the screen of the second user terminal, the augmented reality object 92 deviated to the left is displayed on the left edge of the screen, and based on the screen of the second user terminal, the augmented reality objects 93 and 94 deviated to the right. ) is displayed on the left edge of the screen. In this case, when the number of augmented reality objects to be displayed on the edge of the screen is plural, each augmented reality object may be aligned and displayed on the corresponding edge of the screen. Referring to FIG. 12 , it can be seen that two augmented reality objects 93 and 984 deviated from the screen to the right are aligned and displayed on the right edge of the screen. Even in the case of the augmented reality objects 92, 93, and 94 off the screen, the names of the corresponding augmented reality objects may be displayed similarly to the augmented reality object 91 on the screen (here, 'Fairy', 'Ace', 'Korean'). ).

Meanwhile, the second user terminal may display the received user interaction in association with the augmented reality object. For example, when a text output interaction is received from the user terminal corresponding to the 'Fairy' augmented reality object 92, the second user terminal displays the 'Fairy' augmented reality object 92 on the augmented reality background 90. In addition, a 'speech bubble icon' meaning text output may be displayed in the augmented reality object 92 together. Alternatively, when an audio output interaction is received from a user terminal corresponding to the 'Fairy' augmented reality object 92, the second user terminal may display a 'loudspeaker icon' indicating audio output instead of a 'speech bubble icon'.

As an embodiment, the second user terminal may adjust a display angle of a character model of an augmented reality object by using a gyro sensor. This will be further described with reference to FIG. 13 .

Referring to FIG. 13 , an example of adjusting a display angle of an augmented reality object 91 using a gyro sensor when the second user terminal is tilted is illustrated.

The second user terminal is an augmented reality terminal, and shows an augmented reality background based on an image captured by a camera. Accordingly, when the second user terminal is tilted, the axis D1 of the screen of the second user terminal and the vertical axis D2 of the augmented reality background may be misaligned with each other. In this case, as in the case of general video playback, when the augmented reality object is displayed based on the second user terminal, inconvenience that the augmented reality object is tilted and displayed on the augmented reality background may occur.

Accordingly, the second user terminal detects the direction of gravity using the gyro sensor and then displays the augmented reality object 91 in the opposite direction D2. By doing so, the problem that the augmented reality object is tilted and displayed on the augmented reality background can be solved.

Hereinafter, a series of processes in which the second user terminal displays the augmented reality background and the augmented reality object will be briefly described.

First, a process of displaying an augmented reality background by the second user terminal will be described.

1. When the second user accesses the server, the account of the second user is authenticated through a login procedure.

2. The location coordinates of the second user or previously recorded last coordinates are checked. At this time, the location coordinates of the second user may be simultaneously identified as GPS information and LBS information. When there is no previously recorded last coordinate for the second user, the default location may be determined as the location of the second user.

3. The virtual reality background including the second user's location coordinates or final coordinates is identified, and the camera function of the second user terminal is executed. If the digital twin is not implemented for the area corresponding to the second user's location coordinates or the final coordinates, 'not a service area' may be displayed and the area may be separately classified as an unidentified area.

4. The second user enters a predetermined network channel through which communication between users is possible. When the capacity of the existing channel is exceeded, a new channel is created, and the second user can enter the new channel.

5. User information corresponding to the second user is loaded. The user information may include information such as a design of a 3D character model, clothes, possession money, or an inventory table.

6. Changes in LBS information and GPS information according to the location movement of the second user terminal are recorded. At this time, LBS information or GPS information may be recorded at predetermined time intervals, and may be set to store information only up to a predetermined number of times. For example, if it is set to store LBS information or GPS information up to 10 times in accumulation, when the 11th LBS information or GPS information is received, instead of storing the 11th LBS or GPS information, the 1st LBS or GPS information may be deleted.

7. Meanwhile, when the second user terminal is out of the service range of the server, a message indicating that the service area is not available may be displayed on the screen of the second user terminal.

Next, the process of displaying the first user on the augmented reality background is explained. The first user may be a user of a virtual reality terminal connected to the second user terminal.

1. A network channel in which the second user participates is scanned and the location coordinates of the first user who participates in the corresponding channel are loaded. In this case, the location coordinates of the first user may be loaded at predetermined time intervals.

2. Augmented reality object information of the first user is loaded. Augmented reality object information may include information such as design of a 3D character model or clothing.

2. An augmented reality object representing the first user is displayed at a location on the augmented reality background corresponding to the first user's location coordinates or final coordinates. In this case, the display angle of the augmented reality object of the first user may be adjusted based on the measurement value of the gyro sensor.

4. Changes in the location coordinates of the first user are periodically checked, and when the location coordinates change, the augmented reality object of the first user is moved to a location corresponding to the change and displayed.

5. When displaying the movement of the first user's augmented reality object, a movement animation may be played along with the display of the movement path.

6. When there is no change in the location coordinates of the first user, 'default animation' may be reproduced in the character model of the augmented reality object.

Meanwhile, as an embodiment, when a new third user joins a channel in which the second user has joined, a system message notifying this may be displayed. Alternatively, when the existing third user leaves the channel in which the second user has participated, a system message notifying this may be displayed. Alternatively, when a new channel is created in relation to the augmented reality background displayed by the second user terminal, a system message notifying this may be displayed.

According to the idea of the present invention described above, the present invention can be applied in various ways in real life. For that, a detailed explanation will be given with one specific example.

User P is a virtual reality terminal user and is located on the 10th floor of the Heavenly Building. User Q is an augmented reality terminal user and is located at Exit 2 of Haneul Station near the Heavenly Building.

When the user P accesses the server through the virtual reality terminal, the virtual reality terminal displays a virtual reality environment that simulates Sky Station No. 2 based on the digital twin. Also, the virtual reality terminal is connected to the augmented reality terminal of the user Q, receives the location of the user Q, and displays the user Q at a corresponding location on the virtual reality environment.

The user Q also accesses the server through the augmented reality terminal, and the augmented reality terminal is connected to the virtual reality terminal of the user P to receive the location of the user P and display the user P at the corresponding position on the augmented reality background. In this way, mutual interaction between users may be performed in a situation in which user P and user Q recognize each other through the virtual reality terminal and the augmented reality terminal.

For example, if user P wants to guide user Q to the 10th floor of the Heavenly Building, where he is currently located, user P expresses his intention to follow him, and takes his own route from Exit 2 of Sky Station to the 10th floor of the Heavenly Building. It is possible to guide the movement path of user Q by sequentially moving. The user Q may follow the moving path guided by the user P while viewing the augmented reality object that is moved and displayed according to the movement of the user P through the augmented reality terminal screen of the user.

In addition, various application examples using a similar method may be possible. For example, a non-face-to-face tour guide service in which a guide accesses at home and a tourist connects in real life to interact with each other, a lost child guidance service that guides a lost child to the house or the police station, or a stranger who sees an elderly person with dementia on the road The present invention can also be applied to a guidance service for the elderly with dementia, which allows the elderly with dementia to be easily guided to their homes.

As an embodiment, a map indicating a movement path guided by a user of the virtual reality terminal may be further displayed on the augmented reality terminal. This will be further described with reference to FIG. 14 .

Referring to FIG. 14 , FIG. 14 (a) shows a display screen of a second user terminal 400 that is an augmented reality terminal. As described above, the second user terminal 400 may display an augmented reality background 401 based on an image captured by its own camera and display an augmented reality object 402 representing the first user thereon. there is. In this case, the first user may be a user of a first user terminal that is a virtual reality terminal.

The second user may view the screen of the second user terminal 400 and follow the route to the destination according to the road guidance of the augmented reality object 402 .

Meanwhile, at this time, the second user terminal 400 may display a map indicating a movement path to be guided by the first user. The map displays and guides a movement path of the second user terminal from a first point (eg, a current location) to a second point (eg, a destination).

Exemplary forms thereof are shown in FIGS. 14 (b) and (c). FIG. 14 (b) shows an example in which the map 410 is displayed in a partial area of the augmented reality background, and FIG. 14 (c) shows an example in which the augmented reality background is removed and the map 420 is displayed in a larger size. do.

Various information for road guidance may be displayed on the maps 410 and 420 . For example, as shown in FIG. 14 (c), the map 420 includes a symbol 421 representing the current location of the second user terminal 400, a line 422 representing a route from the current location to the destination, Alternatively, symbols indicating other objects 423, 424, and 425 located around the movement path may be displayed. At this time, the other objects 423, 424, and 425 may be objects representing other users participating in the same channel as the second user, objects representing major milestones installed on the movement route, or objects representing landmarks around the movement route. can

In this way, when the maps 410 and 420 indicating the movement route are displayed on the second user terminal 400, the second user can more easily follow the first user's route guidance by referring to the maps 410 and 420.

In particular, when the digital twin is not implemented or incompletely implemented for the real space where the second user is located, there may be a difference between the virtual reality environment seen by the first user and the augmented reality background seen by the second user. In such a case, it may be difficult for the second user to follow the road guidance by viewing only the user interaction expressed through the augmented reality object 402 . At this time, if the maps 410 and 420 indicating the movement path are displayed, the second user may more easily follow the route guidance of the first user by referring to the maps 410 and 420 .

Meanwhile, in this embodiment, the case of displaying the map on the second user terminal 400, which is an augmented reality terminal, has been described, but it is also possible to display the map on the first user terminal, which is a virtual reality terminal. If the augmented reality screen of the second user and the virtual reality environment that the first user can see do not match each other because the digital twin is not implemented for the corresponding real space, the first user also provides guidance to the second user. Because doing so can be difficult. In this case, the first user terminal may display a map for road guidance on the virtual reality environment, and the first user may send a user interaction for road guidance to the second user while viewing the displayed map.

Hereinafter, referring to FIG. 15 , an exemplary computing device 500 in which methods described in various embodiments of the present invention are implemented will be described. For example, the computing device 500 of FIG. 15 may be the server 100 of FIG. 1 , the first user terminal 200 , or the second user terminal 300 .

15 is an exemplary hardware configuration diagram illustrating a computing device 500 .

As shown in FIG. 15, the computing device 500 loads one or more processors 510, a bus 550, a communication interface 570, and a computer program 591 executed by the processor 510. It may include a memory 530 and a storage 590 for storing the computer program 591. However, only components related to the embodiment of the present invention are shown in FIG. 15 . Therefore, those skilled in the art to which the present invention pertains can know that other general-purpose components may be further included in addition to the components shown in FIG. 15 .

The processor 510 controls the overall operation of each component of the computing device 500 . The processor 510 may include at least one of a Central Processing Unit (CPU), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphic Processing Unit (GPU), or any type of processor well known in the art. can be configured to include Also, the processor 510 may perform an operation for at least one application or program for executing a method/operation according to various embodiments of the present disclosure. Computing device 500 may include one or more processors.

Memory 530 stores various data, commands and/or information. Memory 530 may load one or more programs 591 from storage 590 to execute methods/operations according to various embodiments of the present invention. An example of the memory 530 may be RAM, but is not limited thereto.

The bus 550 provides a communication function between components of the computing device 500 . The bus 550 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 570 supports wired and wireless Internet communication of the computing device 500 . The communication interface 570 may support various communication methods other than Internet communication. To this end, the communication interface 570 may include a communication module well known in the art.

Storage 590 may non-temporarily store one or more computer programs 591 . The storage 590 may be a non-volatile memory such as read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or the like, a hard disk, a removable disk, or a device well known in the art. It may be configured to include any known type of computer-readable recording medium.

Computer program 591 may include one or more instructions for implementing methods/operations according to various embodiments of the present invention.

For example, the computer program 591 may include an operation of storing digital twin information corresponding to a real space, an operation of providing a virtual reality environment to a first user terminal based on the digital twin information, and a control for the virtual reality environment. It may include instructions for performing an operation of receiving an interaction of the first user and an operation of providing the received interaction of the first user to a second user terminal.

Alternatively, the computer program 591 may include an operation of receiving a virtual reality environment based on digital twin information from a server, an operation of displaying a virtual reality background corresponding to a real space based on the virtual reality environment, and a user on the virtual reality background. It may include instructions for performing an operation of displaying an object representing , and an operation of receiving the user's interaction from the server and displaying it on the virtual reality background.

Alternatively, the computer program 591 may perform an operation of displaying an augmented reality background on a screen using an image of a real space captured through a processor camera, an operation of displaying an object representing a user on the augmented reality background, and an operation of displaying an object representing a user on the augmented reality background, and from the server It may include instructions for performing an operation of receiving the user's interaction and displaying it on the augmented reality background.

When computer program 591 is loaded into memory 530, processor 510 may execute the one or more instructions to perform methods/acts according to various embodiments of the present invention.

The technical idea of the present invention described so far can be implemented as computer readable code on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet, installed in the other computing device, and thus used in the other computing device.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art can implement the present invention in other specific forms without changing the technical spirit or essential features. can understand that there is Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the technical ideas defined by the present invention.

Claims (20)

In the digital twin-based user interaction method performed by a computing device,
Storing digital twin information corresponding to real space;
Providing a virtual reality environment to a first user terminal based on the digital twin information;
receiving a first user's interaction with the virtual reality environment; and
Providing the received interaction of the first user to a second user terminal;
The interaction of the first user,
Including movement of the location of the first user,
The second user terminal,
In response to the interaction of the first user, displaying a first object corresponding to the first user on a display of the second user terminal to a position corresponding to the position movement;
Receiving the interaction of the first user,
Receiving Location Based Service (LBS) information or Global Positioning System (GPS) information indicating the location of the first user; and
Comprising the step of calculating the error between the LBS information and the GPS information,
User interaction method based on digital twin. According to claim 1,
The first user terminal,
Displaying a virtual reality background corresponding to the real space based on the virtual reality environment and displaying a second object representing a second user on the virtual reality background;
A position on the virtual reality background where the second object is displayed corresponds to a position of the second user in the real space.
User interaction method based on digital twin. According to claim 1,
The second user terminal,
An augmented reality background is displayed using an image captured through a camera of the second user terminal, and the first object is displayed on the augmented reality background,
A location on the augmented reality background where the first object is displayed corresponds to a location of the first user in the virtual reality environment.
User interaction method based on digital twin. According to claim 1,
The interaction of the first user,
Including the first user's chat, conversation, gesture, or status display,
User interaction method based on digital twin. delete According to claim 1,
The step of providing to the second user terminal,
When the LBS information is receivable, providing location information of the first user to the second user terminal based on the LBS information,
User interaction method based on digital twin. According to claim 1,
The step of providing to the second user terminal,
If the LBS information is not receivable, providing location information of the first user to the second user terminal based on the GPS information,
User interaction method based on digital twin. According to claim 7,
The location information of the first user,
Based on the error between the pre-calculated LBS information and the GPS information, the GPS information is corrected,
User interaction method based on digital twin. According to claim 1,
Further comprising receiving a second user's interaction from the second user terminal and providing it to the first user terminal,
User interaction method based on digital twin. According to claim 1,
The first user terminal or the second user terminal,
Displaying a map guiding the movement path of the second user terminal from a first point to a second point,
User interaction method based on digital twin. In the digital twin-based user interaction method performed by a computing device,
Receiving a virtual reality environment based on digital twin information from a server;
displaying a virtual reality background corresponding to a real space based on the virtual reality environment;
displaying an object representing a user on the virtual reality background; and
Receiving the user's interaction from the server and displaying it on the virtual reality background;
The virtual reality background,
It simulates the real space where the user is located,
The position of the object on the virtual reality background corresponds to the position of the user in the real space;
The step of displaying an object representing the user,
If the position of the object on the virtual reality background is within a predetermined viewing angle range, the object is displayed as a character model, and if the position of the object on the virtual reality background is outside the predetermined viewing angle range, the object is designated as a name or symbol at the viewing angle Marked on the edge of the range,
User interaction method based on digital twin. delete delete In the digital twin-based user interaction method performed by a computing device,
displaying an augmented reality background on a screen using an image of a real space captured through a camera;
displaying an object representing a user on the augmented reality background; and
Receiving the user's interaction from a server and displaying it on the augmented reality background;
A position on the augmented reality background where the object is displayed corresponds to a position of the user on a virtual reality space corresponding to the real space;
The step of displaying an object representing the user,
If the location of the object on the augmented reality background is within the screen, the object is displayed as a character model, and if the location of the object on the augmented reality background is outside the screen, the object is displayed as a name or symbol at the edge of the screen doing,
User interaction method based on digital twin. delete According to claim 14,
The step of displaying an object representing the user,
Adjusting the display angle of the character model using a gyro sensor,
User interaction method based on digital twin. delete processor;
a memory for loading a computer program executed by the processor; and
Including a storage for storing the computer program,
The computer program,
An operation of storing digital twin information corresponding to real space;
An operation of providing a virtual reality environment to a first user terminal based on the digital twin information;
receiving a first user's interaction with the virtual reality environment; and
Includes instructions for performing an operation of providing the received interaction of the first user to a second user terminal;
The interaction of the first user,
Including movement of the location of the first user,
The second user terminal,
In response to the interaction of the first user, displaying a first object corresponding to the first user on a display of the second user terminal to a position corresponding to the position movement;
The operation of receiving the interaction of the first user,
Receiving LBS (Location Based Service) information or GPS (Global Positioning System) information indicating the location of the first user, and calculating an error between the LBS information and the GPS information,
User interaction device based on digital twin. processor;
a memory for loading a computer program executed by the processor; and
Including a storage for storing the computer program,
The computer program,
Receiving a virtual reality environment based on digital twin information from a server;
displaying a virtual reality background corresponding to a real space based on the virtual reality environment;
displaying an object representing a user on the virtual reality background; and
Includes instructions for performing an operation of receiving the user's interaction from the server and displaying it on the virtual reality background;
The virtual reality background,
It simulates the real space where the user is located,
The position of the object on the virtual reality background corresponds to the position of the user in the real space;
The operation of displaying an object representing the user,
If the position of the object on the virtual reality background is within a predetermined viewing angle range, the object is displayed as a character model, and if the position of the object on the virtual reality background is outside the predetermined viewing angle range, the object is designated as a name or symbol at the viewing angle Marked on the edge of the range,
User interaction device based on digital twin. processor;
a memory for loading a computer program executed by the processor; and
Including a storage for storing the computer program,
The computer program,
An operation of displaying an augmented reality background on a screen using an image taken of a real space through a camera;
displaying an object representing a user on the augmented reality background; and
Includes instructions for performing an operation of receiving the user's interaction from a server and displaying it on the augmented reality background;
A position on the augmented reality background where the object is displayed corresponds to a position of the user on a virtual reality space corresponding to the real space;
The operation of displaying an object representing the user,
If the location of the object on the augmented reality background is within the screen, the object is displayed as a character model, and if the location of the object on the augmented reality background is outside the screen, the object is displayed as a name or symbol at the edge of the screen doing,
User interaction device based on digital twin.

Images