KR20220125540A - A method for providing a virtual space client-based mutual interaction service according to location interlocking between objects in a virtual space and a real space - Google Patents

Abstract

Presented is a method for providing a mutual interaction service between a virtual reality user and an augmented reality user, which can provide various mutual interaction services between the virtual reality user and the augmented reality user based on a virtual space built using digital twin technology. The present invention relates to the method for providing a VR client standard service. The method comprises the steps of: generating a channel by loading the virtual space (VR scene) by mapping coordinates of a real world corresponding to the virtual space built for a service target area, and outputting my character of a VR user character (VR character) to the VR scene; outputting other characters (AR characters) corresponding to one or more augmented reality users corresponding to each VR scene in response to the location of one or more augmented reality user terminals connected to the channel; and among the location information of the AR character, recording LBS coordinates and GPS coordinate changes, calculating a coordinate change direction and an average movement speed, and when the coordinates cannot be recognized, generating a virtual movement path to correct a location change error of the AR character by reproducing a movement animation. Action values are transmitted and received through a network between objects, of which locations are output, to generate mutual interactions. Therefore, the present invention can improve user satisfaction.

Description

A method for providing a virtual space client-based mutual interaction service according to location interlocking between objects in a virtual space and a real space }

The present invention provides a mutual interaction service based on digital twin technology based on a virtual space client-based interaction between objects in a virtual space and real space, which can provide a mutual interaction service by linking the positions between virtual reality users and augmented reality users. It relates to a method of providing a service.

Recently, virtual reality (VR) and augmented reality (AR) are attracting attention as tools for sharing and expanding human experience.

Virtual reality (VR) creates a virtual space with a virtual image composed of a background and an object, whereas augmented reality (AR) is a technology that shows a three-dimensional virtual image over a real image as a single image.

The virtual reality application provides an experience environment to the user by interacting in a virtual space according to the user's actions delivered through separate equipment such as virtual reality glasses and motion recognition devices. Augmented reality applications have a high sense of reality because objects in the real world and virtual objects interact.

Recently, due to the development of software and hardware technologies such as computer graphics technology such as 3D rendering and high-speed/parallel processing technology, various applications using virtual reality and augmented reality technology are being developed. Various industries such as games and industries are also interested in developing applications using virtual and augmented reality technologies.

Korean Patent Laid-Open Patent Publication No. 10-2005-0061856 (June 23, 2005) A tour guide service system using virtual reality and a service method therefor.

The present invention has been proposed in view of the above-described conventional circumstances, and between objects in virtual space and real space to provide various interactive interaction services between virtual reality users and augmented reality users based on a virtual space built with digital twin technology. An object of the present invention is to provide a method for providing a virtual space client-based mutual interaction service according to location linkage.

In order to achieve the above object, a method according to a preferred embodiment of the present invention is a method of providing a VR client standard service, by mapping the coordinates of the virtual space constructed for the service target area and the real world corresponding to the virtual space (VR). scene) to create a channel, and outputting a VR user's my character (VR character) to the VR scene; outputting other characters (AR characters) corresponding to the one or more augmented reality users to a VR scene in response to the positions of one or more augmented reality user terminals connected to the channel; Records LBS coordinates and GPS coordinate changes among the location information of the AR character, calculates the coordinate change direction and average movement speed, and creates a virtual movement path when coordinates cannot be recognized to do; It is a service providing method for generating interaction with each other by exchanging action values through a network between objects, including the location output.

An apparatus for implementing the method includes: a digital twin image storage unit for storing one or more digital twin images; A virtual reality user terminal that processes mutual interaction with an augmented reality user corresponding to a virtual reality space, and an augmented reality user terminal that processes mutual interaction with a virtual reality user corresponding to an augmented reality space. synchronizer; and extracting a digital twin image corresponding to the position and coordinate system synchronized through the synchronization unit from the digital twin image storage unit and transmitting it to the virtual reality user terminal, with a real image as a background in the augmented reality user terminal, and the virtual In the real user terminal, with the digital twin image as a background, an interaction service providing unit that provides a mutual interaction service between the augmented reality user and the virtual reality user based on the location and coordinate system synchronization; can be configured.

The location and coordinate system synchronizer may transmit the synchronized location information and coordinate system information to the virtual reality user terminal and the augmented reality user terminal, respectively.

It may further include; a network channel connection unit for connecting a network channel between the virtual reality user terminal and the augmented reality user terminal to interact with each other.

The digital twin image may be an image produced by digital twin technology that implements real equipment or space in the virtual world.

The interaction service providing unit may download a service app capable of mutual interaction with the augmented reality user to the virtual reality user terminal.

The interaction service providing unit may download a service app capable of mutual interaction with the virtual reality user to the augmented reality user terminal.

On the other hand, a system for providing a mutual interaction service between a virtual reality user and an augmented reality user according to a preferred embodiment of the present invention includes: a virtual reality user terminal for processing mutual interaction with an augmented reality user performed in a virtual reality space; Augmented reality user terminal for processing the interaction with the virtual reality user performed in the augmented reality space; and providing a mutual interaction service with an augmented reality user performed in a virtual reality space through the virtual reality user terminal, and providing a mutual interaction service with a virtual reality user performed in an augmented reality space through the augmented reality user terminal server; includes.

The server synchronizes the positions and coordinate systems of the virtual reality user terminal and the augmented reality user terminal, respectively, and transmits them to the virtual reality user terminal and the augmented reality user terminal, and the location information synchronized with each other belongs to the digital region A twin image may be extracted from a pre-stored digital twin image and transmitted to the virtual reality user terminal and the augmented reality user terminal.

The virtual reality user terminal may include: a location tracking unit for tracking the current location of the virtual reality user; and an interaction service providing unit that, upon receiving the synchronized location and coordinate system information between the virtual reality user and the augmented reality user, displays the characters of the augmented reality users whose positions are synchronized with each other on the display unit based on the coordinate system synchronized with each other ; may be included. In this case, the location tracking unit may include: an LBS module unit for tracking the current location of the virtual reality user in real time using a latitude and longitude location measurement technique through a communication base station; GPS module unit for tracking the current location of the virtual reality user in real time using a latitude and longitude location measurement technique through satellite; and an error correction unit for continuously calculating and correcting an error between the LBS coordinate value from the LBS module unit and the GPS coordinate value from the GPS module unit. The location tracking unit may initially track the location of the virtual reality user with the LBS, and when tracking with the LBS is impossible, track the location with the GPS.

The augmented reality user terminal, a location tracking unit for tracking the current location of the augmented reality user; and an interaction service providing unit that, upon receiving the synchronized location and coordinate system information between the virtual reality user and the augmented reality user, displays the characters of the virtual reality users whose positions are synchronized with each other on the display unit based on the coordinate system synchronized with each other ; may be included. At this time, the location tracking unit LBS module unit for tracking the current location of the augmented reality user in real time by a latitude and longitude location measurement technique through a communication base station; GPS module unit for tracking the current location of the augmented reality user in real time using a latitude and longitude location measurement technique through satellite; and an error correction unit for continuously calculating and correcting an error between the LBS coordinate value from the LBS module unit and the GPS coordinate value from the GPS module unit. The location tracking unit may track the location of the augmented reality user with the first LBS, and if tracking with the LBS is not possible, track the location with GPS.

According to the present invention having such a configuration, various interactive interaction services are possible between a virtual reality user and an augmented reality user based on a virtual space built with digital twin technology, and thus user satisfaction can be improved.

1 is a block diagram of a system for providing a mutual interaction service between a virtual reality user and an augmented reality user according to an embodiment of the present invention.
FIG. 2 is an internal configuration diagram of the virtual reality user terminal shown in FIG. 1 .
3 is an internal configuration diagram of the augmented reality user terminal shown in FIG. 1 .
FIG. 4 is an internal configuration diagram of the server shown in FIG. 1 .
5 and 6 are examples of screens displayed on the augmented reality user terminal shown in FIG. 1 .
7 is an explanatory diagram of an interaction service providing method of a mutual interaction service system according to the present invention;
8 is an explanatory diagram of a service providing method of a VR client of the mutual interaction service system according to the present invention.
9 is an explanatory diagram of a service providing method of an AR client of the mutual interaction service system according to the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a block diagram of a system for providing a mutual interaction service between a virtual reality user and an augmented reality user according to an embodiment of the present invention.

A system for providing a mutual interaction service between a virtual reality user and an augmented reality user according to an embodiment of the present invention includes a virtual reality user terminal 10 , an augmented reality user terminal 20 , a network 30 , and a server 40 . may include

The virtual reality user terminal 10 may provide a virtual reality service performed through a virtual reality (VR) space.

In particular, the virtual reality user terminal 10 may process interaction with an augmented reality user (ie, the augmented reality user terminal 20 ) performed in a virtual reality (VR) space.

The virtual reality user terminal 10 may download and use a service app capable of mutual interaction with the augmented reality user from the server 40 .

The virtual reality user terminal 10 may display a character (eg, a three-dimensional object) of the augmented reality user performing mutual interaction on the screen based on the current location of the augmented reality user.

In addition, the virtual reality user terminal 20 may display the characters of a plurality of other virtual reality users connected to the same network channel and characters corresponding to the plurality of other augmented reality user terminals by performing a mutual interaction on the screen.

In addition, in the case of displaying a plurality of characters, it is possible to display all the participant characters in a separate map window correspondingly displayed.

The augmented reality user terminal 20 may provide an augmented reality service performed through an augmented reality (AR) space.

Especially. The augmented reality user terminal 20 may process a mutual interaction with a virtual reality user (ie, the virtual reality user terminal 10) performed in an augmented reality (AR) space.

The augmented reality user terminal 20 may download and use a service app capable of mutual interaction with the virtual reality user from the server 40 .

The augmented reality user terminal 20 may display a character (eg, a three-dimensional object) of a virtual reality user performing mutual interaction on the screen based on the current position of the character in the 3D scene of the virtual reality user.

The augmented reality user terminal 20 may display the characters of a plurality of virtual reality users connected to the same network channel and characters corresponding to a plurality of other augmented reality user terminals by performing a mutual interaction on the screen.

In addition, in the case of displaying a plurality of characters, it is possible to display all the participant characters in a separate map window correspondingly displayed.

The above-described virtual reality user terminal 10 and augmented reality user terminal 20 may be implemented as a portable terminal or a portable computer. Here, the portable terminal is a wireless communication device that guarantees portability and mobility, and includes a Personal Communication System (PCS), a Global System for Mobile communications (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a Personal Digital (PDA). Assistant), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminals, etc. ) based wireless communication device. Portable computers may include notebooks, laptops, and the like.

In addition, the virtual reality user terminal 10 and the augmented reality user terminal 20 may be various smart devices such as a smart phone, a smart note, a tablet PC, and a wearable computer.

Meanwhile, the aforementioned virtual reality user terminal 10 and augmented reality user terminal 20 may access the server 40 through the network 30 .

Here, the network 30 may be implemented as a wired network such as a local area network (LAN), a wide area network (WAN), or a value added network (VAN). On the other hand, the network 30 is a mobile communication network (mobile radio communication network), satellite communication network, Bluetooth (Bluetooth), Wibro (Wireless Broadband Internet), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), such as all It can be implemented as a kind of wireless network. If necessary, the network 30 may be a network in which wired and wireless are mixed.

The server 40 may provide a mutual interaction service with an augmented reality user performed in a virtual reality space through the virtual reality user terminal 10 .

In addition, the server 40 may provide a mutual interaction service with the virtual reality user performed in the augmented reality space through the augmented reality user terminal 20 .

In particular, the server 40 synchronizes the current positions and coordinate systems of the virtual reality user terminal 10 and the augmented reality user terminal 20 with each other for mutual interaction between the virtual reality user and the augmented reality user terminal 10. And it can be transmitted to the augmented reality user terminal (20).

Accordingly, while the virtual reality user and the augmented reality user interact with each other, the virtual reality user terminal 10 displays a digital twin image and a character (three-dimensional model) of the other party (ie, augmented reality user), augmented reality The user terminal 20 displays a camera image (background image) and a character (three-dimensional model) of the other party (ie, virtual reality user).

In addition, when a user moves while the virtual reality user and the augmented reality user interact with each other, the image displayed on the screen of each terminal and the character of the other party may also be moved according to the moving position of the corresponding user.

Although the virtual reality user terminal 10 and the augmented reality user terminal 20 are shown one by one in FIG. 1 , it should be considered that there are actually a plurality of the virtual reality user terminal 10 and the augmented reality user terminal 20 , respectively.

FIG. 2 is an internal configuration diagram of the virtual reality user terminal shown in FIG. 1 .

The virtual reality user terminal 10 may include a location tracking unit 11 , an interaction service providing unit 12 , and a display unit 13 .

The location tracking unit 11 tracks the current location of the virtual reality character corresponding to the virtual reality user and the augmented reality character corresponding to the augmented reality user terminal.

The position tracking unit 11 may include an internal position tracking unit 11a, another character position receiving unit 11b, and an error correcting unit 11c.

My location tracking unit 11a tracks the location of my character loaded into the 3D scene in correspondence with the virtual reality user. This maps the coordinates of the 3D scene to the corresponding coordinates in the real world, and tracks the position of my character in the 3D scene.

The other character location receiving unit 11b receives the location information of the other character corresponding to the other user connected to the corresponding network channel. That is, the LBS coordinate value and GPS coordinate value information of the AR user terminal are received.

The error correcting unit 11c may continuously calculate and correct the error between the LBS coordinate value and the GPS coordinate value among the position information of the other character received through the other character position receiving unit 11b.

In addition, the error between the LBS coordinate value and the GPS coordinate value is continuously calculated to predict the average error rate of the two after the program is started. keep it down

The interaction service providing unit 12 may log in by accessing the server 40 through the network 30 .

The interaction service providing unit 12 may check the current connection coordinates and the final recorded coordinates through the location tracking unit 11 . At this time, the interaction service providing unit 12 can be checked simultaneously with GPS and LBS, and in the case of the first execution, the default is loaded.

The interaction service providing unit 12 may generate a three-dimensional world scene including the final recorded coordinates (eg, the coordinates at which the connection with the server 40 was last terminated in advance). Here, the scene means a unit of virtual space. Virtual space refers to a three-dimensional world created by digital twins. Coordinates mean location information (X, Y, Z) in the scene. Digital twin refers to technology that implements real equipment or space in the virtual world (three-dimensional world) like twins, and refers to digital reproduction of physical objects (real objects, etc.) in real space.

When the final recorded coordinates belong to a digital twin unimplemented area, the interaction service providing unit 12 may display "not a service area" and move to an arbitrary virtual scene.

The interaction service providing unit 12 may enter a network channel or may create a network channel. For example, the interaction service providing unit 12 can quickly enter the same world, and when there are n or more participants who have entered the corresponding network channel, it can create a new network channel to the corresponding world.

The interaction service providing unit 12 may load character information of the corresponding virtual reality user. In this case, the interaction service providing unit 12 may load a 3D model, clothes, possessions, an inventory table, and the like.

The interaction service providing unit 12 may create a character (3D model) of the virtual reality user by matching the location of the VR HMD on the 3D world scene that matches the final recorded coordinate value.

The interaction service providing unit 12 may express the virtual reality user's character as well as the surroundings in three-dimensional form on the display unit 13 from a first-person view. In addition, the interaction service providing unit 12 may be able to track hand motion in addition to the first-person view, and may play a designated default animation in the case of a PC or mobile.

The interaction service providing unit 12 may record a change in coordinates according to a movement input value. It can be used as coordinate information when moving and reconnecting later by recording coordinate changes for a certain number of times for my character's position information. For example, for the coordinate information of other characters, the interaction service providing unit 12 may record the LBS coordinate values and the GPS coordinate values that change at n second intervals, or may be recorded flexibly according to the base station type. On the other hand, the interaction service providing unit 12 may store the recorded LBS coordinate values and the GPS coordinate values, respectively, only 10 cumulative times, and may automatically delete records exceeding 11 times. Here, the LBS coordinate value means a coordinate value by a latitude and longitude position measurement technique through a communication base station, and the GPS coordinate value means a coordinate value by a latitude and longitude position measurement technique through a satellite.

The interaction service providing unit 12 may restrict movement so that the coordinates of the player (eg, virtual reality user and augmented reality user connected to a network channel) cannot deviate from the service range. For example, the interaction service providing unit 12 may check a character collision between the virtual reality user and the augmented reality user, and the player's warp is treated as an exception.

Meanwhile, the interaction service providing unit 12 may scan the network channel and load the coordinate values of the participating players (eg, load at n second intervals).

The interaction service provider 12 may load character information (eg, appearance center (3D model, clothes)) of an augmented reality user who has entered (participated) a network channel.

The interaction service providing unit 12 may generate a character (3D model) of the augmented reality user on the 3D world scene that matches the final recorded coordinate value.

The interaction service providing unit 12 may check the coordinate change of the augmented reality user according to the network channel scan result, and may change the output position of the character of the corresponding augmented reality user accordingly.

The interaction service provider 12 may compare the difference with the previous output value when recording new coordinates for the augmented reality user.

The interaction service providing unit 12 may reproduce a "moving animation" on the character (3D model) of the augmented reality user when a coordinate change event of the augmented reality user occurs, and connect the corresponding path.

If there is no difference between the new coordinates of the augmented reality user and the previous coordinates, the interaction service providing unit 12 may play a "default animation" on the character (3D model) of the augmented reality user. If the player is virtual reality (VR), HMD and hand tracking motion can be output.

The interaction service provider 12 exchanges action values through the network 30 between the virtual reality user whose location is output and the augmented reality user and generates mutual interactions (eg, voice conversation, motion, chatting, movement, etc.) .

The interaction service providing unit 12 loads the recorded GPS data when the LBS coordinates cannot be recognized. On the other hand, when it is impossible to load GPS data, the interaction service providing unit 12 may output a 3D model to the final recorded value and reproduce a default animation or output an HMD and hand tracking motion.

The interaction service providing unit 12 may calculate the coordinate change direction and the average moving speed of the recorded GPS 10 times.

The interaction service provider 12 may generate a virtual movement path of coordinates and reproduce a "moving animation" by applying it to a 3D model in a state where LBS coordinates cannot be recognized.

The interaction service providing unit 12 stops the virtual path processing upon receiving the new LBS coordinate value, and overwrites the correction process with the normal output process (ie, deletes the existing calculated value).

The interaction service providing unit 12 may output, for example, a system message such as “Channel Creation”, in the center of the screen of the display unit 13 when the first network channel is created.

The interaction service providing unit 12 may output, for example, a system message such as “#name participation” in the center of the screen of the display unit 13 when the corresponding channel player participates.

The interaction service providing unit 12 may output, for example, a system message such as “#name leaving” in the center of the screen of the display unit 13 when the corresponding channel player leaves.

In particular, as the interaction service provider 12 receives the synchronized location and coordinate system information between the virtual reality user and the augmented reality user from the server 40, the location of the synchronized virtual reality user's character and the augmented reality user Characters can be displayed on the display unit 13 on the basis of the same coordinate system.

The display unit 13 displays a mark such as “#name” on the head of the character (3D model) as illustrated in FIG. 3 when the player exists inside the standard of the viewing angle of the screen of my character (default: ±60 degrees). can

In the case of a player out of the standard of the viewing angle of my character screen (default: ±60 degrees), the display unit 13 shows 61 degrees to 180 degrees as illustrated in FIG. 4 by arranging the right arrow and "# name" can be displayed. In this case, the display unit 13 may display a status icon in front of each arrow rather than the “#name” of the player.

In the case of a player out of the standard of the viewing angle of my character screen (default: ±60 degrees), the display unit 13 is 181 degrees to 300 degrees as illustrated in FIG. name" can be displayed. In this case, the display unit 13 may display a status icon in front of each arrow rather than the “#name” of the player.

The display unit 13 may display a "loudpeaker icon" in front of the player #name when the player is in an audio output state.

The display unit 13 may display a "speech bubble icon" in front of the player #name when the player is in a text chat box output state.

3 is an internal configuration diagram of the augmented reality user terminal shown in FIG. 1 , and FIGS. 5 and 6 are examples of screens displayed on the augmented reality user terminal shown in FIG. 1 .

The augmented reality user terminal 20 may include a location tracking unit 21 , an interaction service providing unit 22 , and a display unit 23 .

The location tracking unit 21 tracks the current location of the augmented reality user.

The location tracking unit 21 may include an LBS module unit 21a, a GPS module unit 21b, and an error correction unit 21c.

The LBS module unit 21a may track the current location of the corresponding augmented reality user in real time by a latitude and longitude location measurement technique through a communication base station.

The LBS module unit 21a outputs an LBS coordinate value (eg, XYZ) for the corresponding augmented reality user.

The GPS module unit 21b may track the current position of the corresponding augmented reality user in real time by using a latitude and longitude position measurement technique through a satellite.

The GPS module unit 21b outputs a GPS coordinate value (eg, XYZ) for the corresponding augmented reality user.

The error correction unit 21c may continuously calculate and correct the error between the LBS coordinate value from the LBS module unit 21a and the GPS coordinate value from the GPS module unit 21b.

In other words, the location tracking unit 21 tracks the first LBS module unit 21a in tracking the location of the augmented reality user, and when tracking with the LBS module unit 21a is not possible, the GPS module unit 21b track That is, the location tracking unit 21 tracks the location of the augmented reality user based on the LBS, but the GPS value is continuously measured and recorded from behind.

In addition, the location tracking unit 21 continuously calculates the error between the LBS coordinate value and the GPS coordinate value to predict the average error rate of the two after the program is driven. By substituting it, the position output is maintained naturally.

The interaction service providing unit 22 may log in by accessing the server 40 through the network 30 .

The interaction service providing unit 22 may check the current connection coordinates and the final recorded coordinates through the location tracking unit 21 . At this time, the interaction service providing unit 22 can simultaneously check with GPS and LBS, and loads a default in case of first execution.

The interaction service providing unit 22 can classify the three-dimensional world region including the final recorded coordinates (eg, the coordinates where the connection with the server 40 was last terminated in advance), and turn on the front camera function. can

When the final recorded coordinates belong to a digital twin unimplemented area, the interaction service providing unit 22 may display "not a service area" and separately classify it as an unconfirmed area.

The interaction service providing unit 22 may enter a network channel or may create a network channel. For example, the interaction service providing unit 12 can quickly enter the same world, and when there are n or more participants who have entered the corresponding network channel, it can create a new network channel to the corresponding world.

The interaction service providing unit 22 may load character information of the corresponding virtual reality user. In this case, the interaction service providing unit 22 may load a 3D model, clothes, possessions, an inventory table, and the like.

The interaction service providing unit 22 may record changes in LBS and GPS coordinates according to movement input values. For example, the interaction service providing unit 22 may record the changed LBS coordinate values and GPS coordinate values at n second intervals, or may be recorded flexibly according to the base station type. On the other hand, the interaction service providing unit 22 may store the recorded LBS coordinate values and the GPS coordinate values each cumulatively only 10 times, and the record exceeding 11 times may be automatically deleted. Here, the LBS coordinate value means a coordinate value by a latitude and longitude position measurement technique through a communication base station, and the GPS coordinate value means a coordinate value by a latitude and longitude position measurement technique through a satellite.

The interaction service providing unit 22 may output a message “not in the service area” to the center of the screen of the display unit 23 when the coordinates of the player are out of the range of the digital twin realization area.

The interaction service providing unit 22 may automatically move to an unidentified area and change the network channel when staying in an area other than the service area for more than n seconds.

Meanwhile, the interaction service providing unit 22 may scan the network channel and load the coordinate values of the participating players (eg, load at n second intervals).

The interaction service provider 22 may load character information (eg, appearance center (3D model, clothes)) of a virtual reality user who has entered (participated) a network channel.

The interaction service providing unit 22 may generate a character (3D model) of the virtual reality user at the coordinates on the screen corresponding to the actual point that matches the final recorded coordinate value. At this time, the interaction service providing unit 22 adjusts the level so that the character (3D model) of the virtual reality user can stand vertically through the gyro sensor (not shown).

The interaction service providing unit 22 may check the coordinate change of the virtual reality user according to the network channel scan result, and may change the output position of the virtual reality user's character accordingly.

The interaction service providing unit 22 may compare a difference with a previous output value when recording new coordinates for the virtual reality user.

When the virtual reality user's coordinate change event occurs, the interaction service providing unit 22 may play a "moving animation" on the virtual reality user's character (3D model) and connect the corresponding path.

When there is no difference between the new coordinates of the virtual reality user and the previous coordinates, the interaction service providing unit 22 may play a "default animation" on the virtual reality user's character (3D model). If the player is virtual reality (VR), HMD and hand tracking motion can be output.

The interaction service provider 22 exchanges action values through the network 30 between the virtual reality user whose location is output and the augmented reality user to generate mutual interactions (eg, voice conversation, motion, chatting, movement, etc.) .

The interaction service providing unit 22 loads the recorded GPS data when the LBS coordinates cannot be recognized. On the other hand, when the loading of GPS data is also impossible, the interaction service providing unit 22 may output a 3D model to the final recorded value and reproduce a default animation or output an HMD and hand tracking motion.

The interaction service providing unit 22 may calculate the coordinate change direction and average moving speed of the recorded GPS 10 times.

The interaction service providing unit 22 may generate a virtual movement path of coordinates and reproduce a "moving animation" by applying it to the 3D model in a state where the LBS coordinates cannot be recognized.

The interaction service providing unit 22 stops the virtual path processing upon receiving the new LBS coordinate value, and overwrites the correction process with the normal output process (ie, deletes the existing calculated value).

The interaction service providing unit 22 may output, for example, a system message such as “Channel Creation”, as a whole, in the center of the screen of the display unit 23 when the first network channel is created.

The interaction service providing unit 22 may output, for example, a system message such as “#name participation” in the center of the screen of the display unit 23 when the corresponding channel player participates.

The interaction service providing unit 22 When the corresponding channel player leaves, for example, a system message such as “#name leaving” may be entirely output on the center of the screen of the display unit 23 .

In particular, as the interaction service provider 22 receives the synchronized location and coordinate system information between the virtual reality user and the augmented reality user from the server 40, the location of the synchronized virtual reality user's character and the augmented reality user Characters can be displayed on the display unit 23 based on the same coordinate system.

When the player exists in the application screen of the augmented reality user terminal 20, the display unit 23 may display a mark such as "#name" on the head of the corresponding character (3D model) as illustrated in FIG. 6 . .

The display unit 23, when there is a player out of the application screen of the augmented reality user terminal 20, 0 degrees to 180 degrees based on the camera median value, as illustrated in FIG. An indication such as "#name" can be displayed. In this case, the display unit 23 may display a status icon in front of each arrow rather than the "#name" of the player.

As illustrated in FIG. 7 , when a player outside the application screen of the augmented reality user terminal 20 exists, the display unit 23 aligns the 181 degrees to 360 degrees based on the camera median in the left end line of the viewing angle in order to the left arrow and An indication such as "#name" can be displayed. In this case, the display unit 23 may display a status icon in front of each arrow rather than the "#name" of the player.

The display unit 23 may display a "loudpeaker icon" in front of the player #name when the player is in the voice output state.

The display unit 23 may display a "speech bubble icon" in front of the player #name when the player is in a text chat box output state.

FIG. 4 is an internal configuration diagram of the server shown in FIG. 1 .

The server 40 includes a digital twin image storage unit 41 , a receiving unit 42 , a network channel connection unit 43 , a location and coordinate system synchronization unit 44 , an interaction service providing unit 45 , and a control unit 46 . can do. Such a server 40 may be an apparatus for providing a mutual interaction service between a virtual reality user and an augmented reality user described in the claims of the present invention.

The digital twin image storage unit 41 stores digital twin images by region (by region). Here, the digital twin image is an image produced by digital twin technology that implements real equipment or space in the same way as twins in the virtual world (three-dimensional world). means the video was created. In an embodiment of the present invention, a virtual space (eg, a three-dimensional world) is constructed with such a digital twin image.

It is desirable to maintain the scale and ratio of the digital twin image for each region described above.

For example, if arbitrary coordinates A and B in the scene are specified, latitude and longitude corresponding to the real world can be substituted for coordinates A and B to match the substitution ratio of the entire scene. Here, the scene means a unit of virtual space. Virtual space refers to a three-dimensional world created by digital twins.

The receiver 42 may receive login information from the virtual reality user terminal 10 and the augmented reality user terminal 20 .

In addition, the receiver 42 may receive real-time location information from the virtual reality user terminal 10 and the augmented reality user terminal 20 .

In addition, the receiving unit 42 is a network channel creation request from the virtual reality user terminal 10 and the augmented reality user terminal 20, a request to participate in the created network channel, a connection disconnection (withdrawal) request from the participating network channel, etc. can receive

The network channel connection unit 43 creates a network channel of the corresponding virtual reality user terminal 10 or augmented reality user terminal 20 according to the network channel creation request received by the receiving unit 42, and another player (ie, other A network channel connection with the user terminal) may be performed.

Of course, the network channel connection unit 43 may disconnect the network channel of the corresponding user terminal when there is a connection disconnection (withdrawal) request from the virtual reality user terminal 10 or the augmented reality user terminal 20 .

The location and coordinate system synchronization unit 44 is the current location of the virtual reality user terminal 10 and the augmented reality user terminal 20 for more effective mutual interaction between the virtual reality user terminal 10 and the augmented reality user terminal 20 Synchronize with each other and synchronize the coordinate systems of the virtual reality user terminal 10 and the augmented reality user terminal 20 with each other.

The location and coordinate system synchronization unit 44 may transmit the synchronized location and coordinate system information to the virtual reality user terminal 10 and the augmented reality user terminal 20 for mutual interaction.

In other words, in order to accurately display the position of the character of the augmented reality user on the virtual reality user terminal 10 and accurately display the position of the character of the virtual reality user on the augmented reality user terminal 20, the virtual reality user and the augmented reality user The position information of the characters of should be synchronized with each other, and the coordinate system of the 3D scene of the virtual reality user terminal 10 and the coordinate system of the augmented reality user terminal 20 should be identical to each other. Therefore, for this purpose, the position and coordinate system synchronization unit 44 synchronizes the current positions of the virtual reality user terminal 10 and the augmented reality user terminal 20 and the coordinate systems of each other with each other.

In this way, the positions of the virtual reality user's character and the augmented reality user's character are synchronized with each other, and the virtual reality user's character and the augmented reality user's character, whose positions are synchronized with each other, are synchronized with each other in the same coordinate system to display the screen on the other's terminal. can be displayed.

If the coordinate systems of each other are different, the characters of the other party will not be properly synchronized with each other's terminals, which will make it difficult to identify the position of the other party.

For example, a virtual reality user who is online at Exit 2 of "Sky Station" created as a virtual space through a digital twin in Office A and an augmented reality app on a smartphone near the real "Sky Station" that is a real space It is assumed that augmented reality users are communicating through networks through each other's characters and explained further.

The positions of the virtual reality user's character and the augmented reality user's character are synchronized with each other, and the synchronized virtual reality user's character and the augmented reality user's character are synchronized with each other in the same coordinate system, and the virtual reality user connects to virtual reality. In the virtual space, he sees the augmented reality user's character and greets him with voice and motion, and moves his character to guide the augmented reality user to the office A to follow him.

Accordingly, the augmented reality user listens to the virtual reality user's voice in the real world with the smartphone AR app turned on, answers the chat, and walks along the virtual reality user's character projected on the screen to guide the way to the office A will receive

In this way, the augmented reality user will be able to visit Office A properly.

The interaction service providing unit 45 may provide a mutual interaction service with the augmented reality user performed in the virtual reality space through the virtual reality user terminal 10 . To this end, the interaction service provider 45 may download a service app capable of mutual interaction with the augmented reality user to the virtual reality user terminal 10 .

In addition, the interaction service providing unit 45 may provide a mutual interaction service with the virtual reality user performed in the augmented reality space through the augmented reality user terminal (20). To this end, the interaction service provider 45 may download a service app capable of mutual interaction with the virtual reality user to the augmented reality user terminal 20 .

The interaction service providing unit 45 extracts the digital twin image of the region to which the location information (eg, XYX coordinate values) synchronized with each other in the location and coordinate system synchronization unit 44 belongs from the digital twin image storage unit 41, It may be transmitted to the virtual reality user terminal 10 and the augmented reality user terminal 20 .

The control unit 46 controls the overall operation of the server 40 .

This time, the operation of the interactive interaction service providing system between the virtual reality user and the augmented reality user configured as described above will be described.

7 is an explanatory diagram of a mutual interaction synchronization method between a virtual reality service and an augmented reality service according to the present invention.

A step (S11) of constructing a virtual space digital twin image with a 3D or 2D scene modeled in 3D for the service target area (S11);

A step of mapping the coordinate system by substituting the coordinates of the virtual space VR scene to the latitude and longitude coordinates of the real world, and matching the substitution ratio of the scene (S12);

Loading the VR character and the AR client character into the VR scene, and loading the VR character on the AR screen (S13);

A step of synchronizing the positions of the VR character in the VR scene, the AR client character, and the VR character in the AR screen by tracking the position of the VR character and the AR client character (S14);

and providing an interaction service by exchanging action values through a network between the VR character whose location is outputted and the AR client character (S15).

In the step of synchronizing the position (S14), the VR character extracts coordinate information in the VR scene and provides it as output position information of the VR character on the AR screen. The AR client character is the location information of the user terminal of the AR client, and is loaded into the LBS. That is, the LBS location information of the AR client user terminal is extracted and the AR client character location is loaded in the VR scene. If LBS load is not possible, load it with GPS. In addition, the position of the object is tracked and output based on the LBS, and the GPS value is continuously measured and recorded. By continuously calculating the error between the LBS output value and the GPS measurement value, the average error rate of the two is predicted after the program is started. .

8 is an explanatory diagram of the operation of the VR client according to the present invention.

Looking at the operation of the VR client, it performs my character output (S21), other character output (S22), error correction output (S23), and controls including channel participant recognition UX (S24) and player character display UX (S25) do.

In the step of outputting my character (S21), the 3D scene is created by checking the login coordinates, the network channel entry/generation is controlled, the VR character is loaded into the 3D scene, and the screen is controlled in response to the VR user's movement input value. and record the coordinates. In this case, my character can be output in a first-person view or a third-person view.

The other character output S22 refers to outputting a character corresponding to the AR client user terminal as another character in the VR scene. Of course, another VR user's character can be displayed as another character. It loads the network chattel scan and the coordinates of participating players, creates other character models in the 3D scene, changes the output position according to the coordinate change, and plays animations and connects the path when the coordinate change event occurs.

The error correction output (S23) records LBS coordinates and GPS coordinate changes among the location information of other characters, calculates the coordinate change direction and average movement speed, and plays a movement animation by creating a virtual movement path when coordinate recognition is not possible. , stop the virtual path processing upon receiving the coordinates and return to the normal state by processing the overwrite by the correction process.

Channel participant recognition UX (S24), based on the center of the channel screen, the player participation and departure, the player name and the participation / departure is displayed on the screen.

The player character display UX (S25) displays #name based on the viewing angle of my character screen. If the viewing angle is out of the way, arrows and #names are displayed to the left and right of the outward direction to recognize other characters off the screen and out of the way.

Meanwhile, when displaying a VR character and at least one AR client character in a VR scene of one channel, a minimap may be further included, and the positions of all characters may be tracked and displayed on the minimap corresponding to the VR scene.

9 is an operation explanatory diagram of an AR client according to the present invention.

As an operation flowchart of the AR user terminal, internal screen output (S31), other character output (S32), error correction output (S33) are performed, and channel participant recognition UX (S34) and player character display UX (S35) are included do.

My screen output (S31) checks the log-in coordinates, divides the 3D scene area and turns on the front camera, enters the network channel, loads the VR character and outputs it on the screen, and records LBS and GPS coordinate changes according to movement and control according to the range of the service area. That is, the VR character is displayed on the inner screen, which is a video screen shot by the front camera, and the location coordinates of the AR user terminal are measured and recorded, and the VR character output is controlled according to the service area by comparing it with the service area coordinates of the 3D scene. do.

Other character output (S32) loads the network channel participating player coordinate values, loads other characters, horizontally adjusts other characters, changes the output position by checking coordinate changes, and performs animation path connection when coordinates are moved do.

The error correction output (S33) records the coordinate change, calculates the coordinate change direction and the average movement speed, generates a virtual movement mirror when the coordinate is not recognized, plays the movement animation, and overwrites the correction process when receiving the coordinate information .

In addition, the channel participant recognition UX ( S34 ) displays the player name and participation/departure of the player joining/leaving based on the center of the channel screen on the screen.

The player character display UX (S35) displays #name based on the viewing angle of the my character screen. If the viewing angle is out of the way, arrows and #names are displayed to the left and right of the outward direction to recognize other characters off the screen and out of the way.

In addition, when displaying a VR character and at least one AR client character on the screen of the AR client, a mini map may be further included, and the positions of all characters may be tracked and displayed on the mini map.

First, the server 40 preloads the digital twin image for each region (by region) produced by digital twin technology that implements real equipment or space in the virtual world (three-dimensional world) in the same way as twins in the digital twin image storage unit 41 in advance. Save it.

The virtual reality user activates the interactive interaction service app with the augmented reality user for mutual interaction with the augmented reality user, and the augmented reality user activates the interactive interaction service app with the virtual reality user for the interactive interaction with the virtual reality user do.

Then, each accesses the server 40 through the network 30 and performs a login or the like, and then requests a network channel connection (formation).

Accordingly, the server 40 connects (forms) a network channel between the virtual reality user terminal 10 and the augmented reality user terminal 20 .

At this time, the virtual reality user terminal 10 transmits the error-corrected current location information of the virtual reality user to the server 40, and the augmented reality user terminal 20 is the error-corrected augmented reality user's current location information is transmitted to the server 40 .

Accordingly, the server 40 extracts the digital twin image for the region including the current location of the virtual reality user and the current location of the augmented reality user from the digital twin image storage unit 41 to extract the virtual reality user terminal 10 ) and transmitted to the augmented reality user terminal 20 . At the same time, the server 40 synchronizes the current positions of the virtual reality user terminal 10 and the augmented reality user terminal 20 with each other, and the coordinate system of the virtual reality user terminal 10 and the augmented reality user terminal 20 with each other. synchronize Then, the server 40 transmits the synchronized location and coordinate system information to the user who wants to mutually interact (that is, the virtual reality user terminal 10 and the augmented reality user terminal 20 that have formed a network channel). .

In this way, the positions of the virtual reality user's character and the augmented reality user's character are synchronized with each other, and the virtual reality user's character and the augmented reality user's character, whose positions are synchronized with each other, are synchronized with each other in the same coordinate system to display the screen on the other's terminal. is displayed That is, while the virtual reality user and the augmented reality user interact with each other, the virtual reality user terminal 10 contains the current location of the virtual reality user and a digital twin image around it and the character 3 of the other party (that is, the augmented reality user). dimensional model) is displayed, and the augmented reality user terminal 20 displays the current location of the augmented reality user and a camera image (background image) around it and a character (three-dimensional model) of the other party (ie, virtual reality user). .

In this way, when the virtual reality user's character and the augmented reality user's character, whose positions are synchronized with each other, are synchronized with each other in the same coordinate system and displayed on the other's terminal, more effective mutual interaction between the virtual reality user and the augmented reality user can be performed. have.

In other words, a virtual reality user who is online at Exit 2 of "Sky Station" made into a virtual space through a digital twin in Office A in the example above, and a smartphone augmented reality near the real "Sky Station" which is a real space Augmented reality users connected to the app can perform very convenient mutual interactions, such as network communication through each other's characters.

On the other hand, the embodiment of the present invention as described above is also applicable to non-face-to-face tour guide tours. For example, a guide can connect at home and a tourist can connect in real life to perform a mutual interaction.

In addition, the embodiment of the present invention as described above is applicable to tracking the location of the elderly living alone with dementia and preventing lost children. For example, the user may perform a mutual interaction, such as coming home after seeing the mother character.

In addition, the embodiment of the present invention as described above is applicable to remote local events and festival participation, and is also applicable to a local direct transaction platform through live commerce and the like.

In addition, the embodiment of the present invention as described above is also applicable to online virtual reality smart city construction.

As described above, the best embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention, and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

10: virtual reality user terminal
11, 21: location tracking unit
12, 22: Interaction service provider
13, 23: display
20: augmented reality user terminal
30: network
40 : server

Claims (9)

A method of providing a VR client reference service providing a mutual interaction service between objects between a virtual reality user terminal providing a virtual reality (VR) service and an augmented reality user terminal providing an augmented reality (AR) service,
Creating a channel by loading a virtual space (VR scene) by mapping the coordinates of the virtual space constructed for the service target area and the corresponding real world, and outputting the VR user's my character (VR character) to the VR scene ;
outputting other characters (AR characters) corresponding to the one or more augmented reality users to a VR scene in response to the positions of one or more augmented reality user terminals connected to the channel;
Records LBS coordinates and GPS coordinate changes among the location information of the AR character, calculates the coordinate change direction and average movement speed, and creates a virtual movement path when coordinates cannot be recognized to do; including
Through the network between the objects whose positions are outputted, action values are exchanged to generate interaction with each other.
HOW TO PROVIDE SERVICES. The method of claim 1,
The step of outputting my character is,
generating a 3D scene by checking the login connection coordinates and the final recorded coordinates;
entering or creating a network channel;
Loading my character (VR character) information, matching a VR HMD position on a 3D scene that matches the coordinate values, and generating a 3D model of my character;
controlling my character while recording the coordinate change according to the movement input value; containing
HOW TO PROVIDE SERVICES. 3. The method of claim 2,
Creating the 3D model includes:
Controlled by first-person view and hand motion tracking, or by playing a specified default animation for PC or mobile
HOW TO PROVIDE SERVICES. 3. The method of claim 2,
Movement is restricted so that the player's coordinates cannot go out of the service range, but collision check and warp are exceptions.
HOW TO PROVIDE SERVICES. The method of claim 1,
The other character output is
loading the participating player coordinates by scanning the network channel at preset time intervals;
Loading information about the appearance-oriented 3D model and clothes of other characters;
generating a 3D model of the other character on a 3D scene that matches the participating player's coordinates;
changing the coordinate change check and output position based on the scan result;
Comparing the difference with the previous output value when new coordinates are recorded, and when a coordinate change event occurs, moving animation playback and path connection processing are performed on the 3D model;
Playing a default animation on the 3D model when there is no coordinate change; containing
HOW TO PROVIDE SERVICES. 6. The method of claim 5,
When the other character's corresponding player is VR, the HMD and tracking motion are output.
HOW TO PROVIDE SERVICES. The method of claim 1,
The error correction output step is
Loading the recorded GPS data when LBS coordinate recognition is not possible, but outputting a 3D model to the final recorded value when loading GPS data is not possible, and playing a default animation or outputting HMD and hand tracking motion;
calculating a coordinate change direction and an average moving speed for a predetermined number of recorded GPS coordinates;
generating a virtual movement path of the coordinates and playing a 'moving animation' by applying the calculation result of the coordinate change direction and the average movement speed to the 3D model in a state where the LBS coordinates cannot be recognized;
stopping the virtual path processing upon receiving a new LBS coordinate value, overwriting the correction process with a normal output process; containing
HOW TO PROVIDE SERVICES. The method of claim 1,
Channel participant recognition UX, based on the center of the channel screen, when players join and leave, the player name and join/leave are displayed on the screen,
The player character display UX displays the #name based on the viewing angle of my character's screen, and if it is out of the viewing angle, arrows and #names are displayed to the left and right of the outward direction to recognize other characters off the screen and the outward direction.
How to provide services. The method of claim 1,
When displaying a VR character and at least one AR client character in a VR scene of one channel, a mini map is further included, and the positions of all characters are tracked and displayed on the mini map corresponding to the VR scene.
HOW TO PROVIDE SERVICES.

Images