KR20220122349A - Cloud-based metaverse content collaboration system - Google Patents

Abstract

Disclosed is a cloud-based metaverse content collaboration system. The cloud-based metaverse content collaboration system includes a behavior recognition algorithm that can analyze a structural feature of a person when recognizing a motion of a user and accordingly recognize and remember motion actions.

Description

Cloud-based metaverse content collaboration system

The present invention relates to a cloud-based metaverse content collaboration system.

Metaverse is a compound word of virtual/transcendent (meta) and world/universe, meaning a three-dimensional virtual world. More specifically, it is widely used in the meaning of a life-type and game-type virtual world where both real and unreal can coexist in the overall aspects of politics, economy, society, and culture. This metaverse refers to an environment in which the boundary between the real and the virtual world is broken because the motions of the real world are reflected in the virtual world and the results of the virtual world are applied to the virtual world. Virtual reality technology, the most core technology that implements this metaverse, has been applied as a technology that allows users to enter and immerse in the virtual reality world and perform specific tasks since the 1970s. It has been divided into After that, as technologies such as metaverse contents were applied, it was applied to a mixed reality environment that could be virtual reality or augmented reality depending on the situation, and an extended reality environment with AI technology and services.

Recently, many metaverse solutions are being released as solutions that combine these virtual and augmented reality technologies with existing online games and SNS. Metaverse solutions include solutions such as Fortnite [5], Roblox, and Minecraft, which are solutions that have become metaverses in online games, and solutions such as VRChat and Naver's ZEPETO, which contain gamification content in the center of social media and exchanges. have. Among them, Fortnite is a special mode of a network-based FPS game, and released a game mode where users can gather to dance and talk, and based on this, a mode where users can enjoy a virtual concert was released. More than 10 million people participated and enjoyed the metaverse concert together.

The solutions constituting the metaverse largely consist of mimicking the actions of users in the real world and transferring them to avatars in the virtual world, and providing a system that enables users in the virtual world to collaborate through a network. The extended reality environment, which is the most technical basis of the metaverse, is a technology that can provide a new experience to users by augmenting virtual objects in the real world, and has been widely used not only in the game field but also in the industrial field and simulation education. In particular, a lot of contents such as games and simulators have been released in virtual reality and augmented reality, respectively. In the case of virtual reality content, “Beat Saber” is one of the representative examples of providing an experience of virtual reality that many people cannot easily experience due to the fun of the game through virtual reality and the conventional cost or technical limitations. . 1 is a play example of “Beat Saber,” which is one of the popular virtual reality games. In addition, “Black Box VR,” the first virtual reality gym in 2019 that combines exercise and games in virtual reality, was released. showed off However, these virtual reality-based contents tend to be dependent on the device of virtual reality, and do not measure all the user's movements, but restrict movement (tracking 3D information of the hand from the head and the controller while wearing an HMD). Since it can be recognized, there is a disadvantage in that it does not provide complete freedom to users.

Studies on technologies that can recognize user's motion in existing virtual reality devices have been implemented using motion tracking technology mainly used in movie shooting. The motion tracking technology matches the information recognized through the markers with the main joint parts of a person when the user attaches the infrared-reflected markers to the user's joints and moves around the space of several infrared cameras as follows. Therefore, it has been used as a technology to recognize the motion of a virtual character in real time, and has been widely used due to its accurate and fast measurement method. However, this method has a characteristic that it is difficult to popularize except in special fields such as the movie industry because the usability is inferior in that the user has no choice but to always attach a specific marker to the body to recognize it, and the equipment is expensive.

Among the methods of tracking the user's body movement by attaching such a specific device to the user's body, HTC Vive's Full Body Tracking is widely used in the metaverse system in that it allows general users to recognize the capture and movement of the mouse at a relatively low price. has been The feature of this device is that the user attaches the HMD and trackers to the body in a designated space and synchronizes the movement of the real world with the virtual world by attaching the attached tracker to a specific joint area of the avatar of the virtual world. . These technologies are linked with solutions such as VR Chat among metaverses. However, since only some of the trackers that recognize the user's body movements are attached, the accuracy is greatly reduced, and using this equipment is also somewhat costly and has a low usability.

On the other hand, a representative game in the field of augmented reality is a game called “Pokemon Go.” This game tracks the user's location with GPS when the user goes outside using a mobile phone, and when a monster appears or a battle occurs, Visualize monsters in augmented reality using a mobile phone and play a game using touch, and a game called “Roboraid” is a game that fights aliens in an augmented reality environment.

This kind of augmented reality game also has the disadvantage of being limited by limited devices. In the case of augmented reality games using a smartphone, the user can play the game by touching the screen in order to interact with the augmented virtual game object. In this case, since the user has to hold the smartphone with one hand and touch it with the other hand, there is a limit to the input that the user can input. The device's interaction restrictions have a disadvantage in that it does not provide users with a realistic game because it is more similar to the real world. To solve this problem, the user supports a special touch controller. However, in this case, the user continuously grips the heavy controller, which increases the user's fatigue, and the control method of holding the controller is mainly used in games in a virtual reality environment in which the game environment is fixed.

In order to track user's body motion in mixed reality, studies using sensors such as IMUs to track motion have also been proposed. However, even in this method, the user has difficulty in attaching several complex devices, which makes usability less useful and the sensors are expensive.

To solve this problem, there is a method that uses a camera to track the user's motion without attaching a sensor. A representative method is the case of using a depth camera to extract motion from a user's body motion image and apply it to virtual reality. However, in the case of this method, there is a disadvantage that recognition is not good when the user's actions are slightly complicated or when interacting with objects. In addition, in the case of a depth camera, it has disadvantages that it is difficult to recognize in the outdoors, where infrared tracking is difficult, and that it is difficult to recognize through a general camera. In order to solve this problem, studies that can recognize the user's motion based on deep learning have been conducted. “Open Pose” made studies that can recognize 2D human motion and facilitated tracking for multiple users. There is a disadvantage that there are overlapping problems in the process of user interaction with the part where the accuracy of information is lowered. “Deeplabcut” conducted recognition studies on moving people, but there is a problem that the recognition rate is somewhat low for parts that are also covered or overlapped. In addition, when the user's motion is recognized, only the user's joints are measured in real time, so when the user performs meaningful actions, that is, specific gestures, there are ways to track and recognize the user's motion according to the passage of time. need. Meanwhile, an attempt to apply this to 3D characters was made in XNect. The system showed the possibility of metaverse connection by applying the information captured in motion to 3D characters, but it also showed problems in that characters overlapped or motions were obscured by objects.

In addition, since most existing metaverse contents provide technologies to manipulate characters with a keyboard or mouse, except for VR Chat that supports Vive trackers, users can see their body movements reflected in the virtual world, so that the boundary between reality and virtual reality is blurred. There are problems that it is difficult to enjoy the obscure metaverse contents.

In the metaverse synchronization collaboration system through deep learning-based motion recognition proposed in this system, we propose to solve existing problems in the following steps. 1) The Attentional GRU-RNN method was applied to classify the upper and lower body movements according to the characteristics of the user's physical structure, and to recognize the movements according to the passage of time. 2) Complex Gesture-GAN network is applied to separate the user's motion and the classification and interference of the interaction object when the user grabs a specific object or performs a complex action between users. 3) Finally, when the user recognizes motion in the client, it provides a cloud-based motion recognition and metaverse application system that recognizes and provides it through the network.

The cloud-based metaverse content collaboration system according to the present invention includes a behavior recognition algorithm capable of analyzing structural characteristics of a person when recognizing a user's motion, and recognizing and remembering motion actions accordingly.

In addition, in order to predict an output of a corresponding value by processing a portion of displacements of motion with respect to the human spine, upper body, and lower body as attention, it can be managed through an attention mechanism.

In addition, it may include a Complex Gesture GAN algorithm that can discriminate the user's meaningful gestures based on the data learned in the previous step, and distinguish them after tracking a physical object.

The effect of the present invention can be largely composed of three types. First, it is possible to solve the problem that the user's motion is not recognized in the existing metaverse system or is recognized through high cost. Using the system proposed in the present invention, it is possible to respond to the avatar of the metaverse by recognizing the user's motion using a camera worth tens of thousands of won, without the need for a motion capture device, which costs tens of thousands to millions of won, so the cost of the system can be reduced.

Second, if the basic motion recognition system had a problem that the usability was somewhat inferior in that the user had to attach a motion capture sensor and a device such as a tracker to the body, the system proposed in the present invention can recognize motion only if it has a webcam. In that sense, usability can be significantly improved.

Third, when compared with the existing camera-based gesture recognition technology, the present invention simultaneously extracts the user's meaningful gestures in the metaverse compared to extracting the user's skeleton through gesture recognition and separately recognizing the gestures. You can use content to help you connect with other users.

Finally, the system proposed in the present invention solves the accuracy problems such as occlusion of the human body that may occur in the process of holding and operating objects such as dumbbells during home training as well as body movements by the user through the GRU_RNN and Complex Gesutre GAN algorithms. Therefore, it can be applied more usefully to users to create more complex metaverse content.

1 is a play example of “Beat Saber,” which is one of the popular virtual reality games.
2 is a diagram illustrating a cloud-based metaverse content collaboration system according to the present invention.
3 is a diagram for explaining a user's motion tracking system to which GRU_RNN is applied.
4 is a diagram for explaining a process of guessing and combining the attention of the operations.
5 is a diagram for explaining a recognition method to which Complex-Gesutre GAN is applied.
6 and 7 are diagrams illustrating scenes in which motion information extracted through the system proposed in the present invention is applied to metaverse content.
8 is a diagram for explaining metaverse health contents.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the technical spirit of the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art.

In this specification, when a component is referred to as being on another component, it may be directly formed on the other component or a third component may be interposed therebetween. In addition, in the drawings, the thicknesses of the films and regions are exaggerated for effective description of technical contents.

Also, in various embodiments of the present specification, terms such as first, second, third, etc. are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another. Accordingly, what is referred to as a first component in one embodiment may be referred to as a second component in another embodiment. Each embodiment described and illustrated herein also includes a complementary embodiment thereof. In addition, in this specification, 'and/or' is used in the sense of including at least one of the elements listed before and after.

In the specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as "comprise" or "have" are intended to designate that a feature, number, step, element, or a combination thereof described in the specification exists, and one or more other features, numbers, steps, or configurations It should not be construed as excluding the possibility of the presence or addition of elements or combinations thereof. In addition, in this specification, "connection" is used in a sense including both indirectly connecting a plurality of components and directly connecting a plurality of components.

In addition, in the following description of the present invention, if it is determined that a detailed description of a related well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

2 is a diagram illustrating a cloud-based metaverse content collaboration system according to the present invention.

Referring to FIG. 2 , the cloud-based metaverse content collaboration system according to the present invention consists of two environments: a server and a client. First, in the server, the deep learning-based user motion recognition and metaverse avatar transformation system and the part that processes the interaction between users recognized as metaverse by user area and task, and interactive contents according to the user's client request, are adaptively processed. It consists of a part that converts content into AR or VR through conversion. On the other hand, the client used by the user basically consists of a part that recognizes the user's motion using an RGB camera and transmits it to the server, and a viewer that provides a metaverse in VR or AR depending on the user's environment.

First, the deep learning-based user motion recognition and metaverse avatar transformation system performs deep learning-based recognition in two steps. Here, we developed a behavior recognition algorithm that can recognize and remember behaviors by analyzing the structural characteristics of a person when recognizing a user's motion. The structural characteristic of a person is that the head is turned upward with the spine as the center, the arms are on the upper body, and the feet are on the lower body with the spine in the center. In order to recognize such a human motion, it must be possible to recognize the motion by observing the change information of the human motion for a certain period of time. should be able to For this, a Gated Recurrent Unit (GRU)-RNN (Recurrent Neural Network) is used.

3 is a diagram for explaining a user's motion tracking system to which GRU_RNN is applied.

Referring to FIG. 3 , while recognizing skeleton information based on image information about a recognized person, based on the anatomical structure, in the case of small animals, the movement of the legs is based on the movement of the spine, so the motion information of the animal is By learning the information on each GRU-RNN, it is possible to finally recognize a dynamic movement.

At this time, in order to predict the output of the corresponding value by processing the part about the displacement of the human spine, the upper body and the lower body as attention, it is managed through the attention mechanism.

4 is a diagram for explaining a process of guessing and combining the attention of the operations.

Referring to FIG. 4 , the attention mechanism requires another value to predict the result of an operation over time, which is a new value called an attention value. An attention value for predicting the t-th word may be defined as at. To obtain this at value, a value called attention score is used. This value is a score for determining how similar each of all the hidden states of the encoder is to the hidden state st of the current time of the decoder in order to predict the next pose at the current pause time t. In dot-product attention, to obtain this score value, st is transposed and a dot product is performed with each hidden state. That is, all attention score values are scalars. For example, the method of calculating the attention score of st and the i-th hidden state of the encoder is as follows. Let et be the collection value of st and the attention scores of all hidden states of the encoder. Equation 1 of e ^t is as follows.

[Formula 1]

When the attention distribution, which is a collection value of attention weights at the time t of the decoder, is α ^t , α ^t is defined as Equation 2 as shown in Equation 2.

[Equation 2]

Now, in order to obtain the final result of attention, it is a process of multiplying the hidden state of each encoder and the attention weight values, and finally adding them all together, that is, in summary, a weighted sum is obtained. Equation 3 below is the final result of attention. That is, the expression for the attention value at, which is the output value of the attention function, is shown.

[Equation 3]

Afterwards, the recognized user's appearance is recognized as a gesture with a specific meaning such as dance or sports in the metaverse, or the user interacts with a physical object such as a baseball bat, a stick, or a chair to express the metaverse content. In this case, it should be possible to prevent a decrease in the recognition rate due to the occlusion of the user's body motion by recognizing the corresponding objects. The system proposed in the present invention proposes a Complex Gesture GAN algorithm that can discriminate a user's meaningful gesture based on the data learned in the previous step, track a physical object, and then distinguish it. In general, when a user's gesture is complex or when motion recognition using objects is obscured, there is a problem in that information on the corresponding part is not recognized properly. We propose a Complex Gesture GAN algorithm that can not only recognize the user's body motion recognized as a gesture over time by creating it, but also precisely infer the gesture even when the motion is obscured by other obstacles. 5 is a diagram for explaining a recognition method to which Complex-Gesutre GAN is applied.

6 and 7 are diagrams illustrating scenes in which motion information extracted through the system proposed in the present invention is applied to metaverse content.

6 and 7, the system made in the present invention integrates algorithms so that it can be applied to various metaverse clients so that users can recognize it through augmented reality devices such as smartphones, web cams and HoloLens through a cloud environment. We propose a cloud-based real-time metaverse pose motion recognition system that can In the proposed system, as shown in the following figure, when the user inputs voice information through an input device at the same time as the user's walking motion, the cloud-based server receives the information through the sensor manager and receives the input. After matching the user's desired avatar, It expresses the avatar information obtained through deep learning algorithms.

8 is a diagram for explaining metaverse health contents.

Referring to FIG. 8 , the system proposed in the present invention can be applied to various metaverse contents. As shown in the following figure, it can be applied to metaverse health contents, that is, in the process of exercising with dumbbells.

As mentioned above, although the present invention has been described in detail using preferred embodiments, the scope of the present invention is not limited to specific embodiments and should be construed according to the appended claims. In addition, those skilled in the art should understand that many modifications and variations are possible without departing from the scope of the present invention.

Claims (3)

A cloud-based metaverse content collaboration system that includes a behavior recognition algorithm that can analyze the structural characteristics of a person when recognizing a user's motion, and recognize and remember motion actions accordingly. The method of claim 1,
A cloud-based metaverse content collaboration system that manages the part about movement displacements of a person's spine, upper body, and lower body as attention, and manages it through an attention mechanism to predict the output of the corresponding value. The method of claim 1,
A cloud-based metaverse content collaboration system that includes the Complex Gesture GAN algorithm that can discriminate the user's meaningful gestures based on the data learned in the previous step and distinguish them after tracking the physical object.

Images