KR102234646B1 - Apparatus and method for experience of virtual reality - Google Patents

Abstract

According to an embodiment of the present invention, a virtual reality experience apparatus and method for recognizing the motion of a user wearing a head-mounted display (HMD) in a simulation space and providing virtual reality to the user. The virtual reality experience apparatus comprises: a motion data reception unit for receiving, from a plurality of camera modules, pieces of motion data obtained by capturing the motion of the user at different angles; a body data generation unit for generating pieces of body data by extracting the body excluding the head of the user from each of the pieces of motion data; a motion recognition unit which divides the pieces of body data for each body part to generate a plurality of candidate regions, extracts features for each candidate area by using a deep learning algorithm to recognize joint points included in the candidate area, and recognizes the motion of the user based on the recognized joint point; a three-dimensional data generation unit for generating three-dimensional data by combining the pieces of motion data for which motion is recognized based on the same body parts and joint points; a coordinate conversion unit for converting the coordinates of the three-dimensional data to match those in the simulation space; and a virtual character generation unit for generating a virtual character in the simulation space based on the three-dimensional data of which the coordinates are converted. The virtual character takes the same motion in real time in response to the motion of the user.

Description

Virtual reality experience device and method {APPARATUS AND METHOD FOR EXPERIENCE OF VIRTUAL REALITY}

The present invention relates to an apparatus and method for experiencing virtual reality, and more specifically, through a deep learning algorithm, a character that takes the same motion as the user's motion is created by merging image data taken from a plurality of cameras that photograph the user at different angles. Thus, it relates to a virtual reality experience device and method for allowing a user to experience virtual reality.

Virtual reality (VR) refers to a technology that makes users feel as if they are actually interacting in a virtual world created by a computer. As the virtual reality market matures, competition among global ICT companies to preoccupy the emerging virtual reality game market is also fierce. In general, virtual reality games are developed so that game users can directly become characters in virtual reality in a virtual space and enjoy the game like a real situation.

In this specification, we will introduce a method of implementing such a virtual reality using a deep learning algorithm.

[Prior literature]

Publication Patent 10-2018-0086938

The present invention relates to an apparatus and method for experiencing virtual reality, and more specifically, through a deep learning algorithm, a character that takes the same motion as the user's motion is created by merging image data taken from a plurality of cameras that photograph the user at different angles. Thus, it relates to a virtual reality experience device and method for allowing a user to experience virtual reality.

A virtual reality experience device for providing a virtual reality experience to the user by recognizing the motion of a user wearing a head mount display (HMD) in a simulation space according to an embodiment of the present invention is a motion of the user from a plurality of camera modules. A motion data receiving unit receiving a plurality of motion data photographed at different angles, a body data generating unit generating a plurality of body data by extracting a body excluding the user's head from each of the plurality of motion data, the plurality of The body data is divided by body part to generate a plurality of candidate regions, and a feature for each of the candidate regions is extracted using a deep learning algorithm to recognize joint points included in the candidate region, and the joint points are referenced. A motion recognition unit that recognizes the user's motion, a three-dimensional data generation unit that generates three-dimensional data by merging the plurality of motion data on which the motion is recognized based on the same body part and joint point, and the three-dimensional data A coordinate conversion unit that converts the coordinates of to match the coordinates of the simulation space, and a virtual character generation unit that generates a virtual character in the simulation space based on the three-dimensional data whose coordinates are converted, wherein the virtual character is It is characterized in that the same motion is taken in real time in response to the motion of the user.

The virtual reality experience apparatus according to an embodiment of the present invention further includes an HMD coordinate matching unit for matching the coordinates of the HMD to the coordinates of the head of the virtual character, and the plurality of camera modules is at least four.

The virtual character generation unit generates the virtual character to correspond to each of the coordinates of each body part of the 3D data whose coordinates are converted, and the virtual character takes the same motion in real time according to the motion of each body part of the user. It is characterized by that.

The 3D data generator generates 3D data by synchronizing boundary coordinates between candidate regions with respect to each of the plurality of motion data for which motion is recognized, and then overlapping coordinates corresponding to the joint points.

A virtual reality experience method using a virtual reality experience device that provides a virtual reality experience to the user by recognizing the motion of a user wearing a head mount display (HMD) in a simulation space according to an embodiment of the present invention is a motion data receiving unit. Receiving a plurality of motion data photographing the motion of the user at different angles from a plurality of camera modules, a body data generation unit extracting a body other than the user's head from each of the plurality of motion data Generating data, the motion recognition unit generates a plurality of candidate regions by dividing the plurality of body data for each body part, and extracts features for each of the candidate regions using a deep learning algorithm, and is included in the candidate region. Recognizing an existing joint point, recognizing the motion of the user based on the corresponding joint point, converting the coordinates of the 3D data to match the coordinates of the simulation space by a coordinate conversion unit, and the coordinates of the virtual character generation unit And generating a virtual character in the simulation space based on the converted 3D data, wherein the virtual character takes the same motion in real time in response to the motion of the user.

The virtual reality experience method according to an embodiment of the present invention further includes the step of matching, by an HMD coordinate matching unit, the coordinates of the HMD with the coordinates of the head of the virtual character, and the plurality of camera modules is at least four.

In the step of creating the virtual character, the virtual character generation unit generates the virtual character to correspond to each of the coordinates of each body part of the 3D data whose coordinates have been converted, and the virtual character is a motion of each body part of the user. In the step of taking the same motion in real time according to and generating the 3D data, the 3D data generator synchronizes boundary coordinates between candidate regions for each of the plurality of motion data for which motion is recognized, and then the joint point 3D data is created by overlapping the corresponding coordinates.

The virtual reality experience device according to the present invention creates motion data by photographing a user from all directions through a plurality of camera modules, merges the motion data through a deep learning algorithm, and takes the user's motion through 3D data. By creating a character of, it allows users to experience a more vibrant virtual reality.

1 is a schematic diagram of a virtual reality experience system according to an embodiment of the present invention.
2 is a block diagram of a virtual reality experience device according to an embodiment of the present invention.
3 is a flowchart of a virtual reality experience method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. Refer to the accompanying drawings. Hereinafter, the present invention will be described in detail.

1 is a schematic diagram of a virtual reality experience system 1000 according to an embodiment of the present invention.

Referring to FIG. 1, a virtual reality experience system 1000 according to an embodiment of the present invention includes a camera module 100, a display terminal 300, and a virtual reality experience device 200 connected thereto through a network 400. It may include.

The camera module 100 may be a module installed in a simulation space in which a user experiences virtual reality. The camera module 100 may be configured in plural. The plurality of camera modules 100 may each photograph the user at a different angle to generate motion data of the user. This will be described in more detail in FIG. 2.

The virtual reality experience device 200 may receive data from the camera module 100 through the network 400 and output the processed result data to the display terminal 300. The virtual reality experience device 200 may be a server.

The display terminal 300 may be a terminal installed in the simulation space. A user can experience a virtual experience through an image from the display terminal 300.

The display terminal 300 may be a display device having a large display unit such as a TV. However, the display terminal 300 is not limited thereto, and the display terminal 300 is an electronic device such as a general desktop computer, a navigation system, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), and a tablet PC. It may include. An electronic device may have one or more general or special purpose processors, memory, storage, and/or networking components (wired or wireless).

The communication method of the network 400 is not limited, and examples of the communication network that the network 400 may include include not only a mobile communication network, a wired online, a wireless online, a communication method using a broadcasting network, but also short-range wireless communication between devices. Can be included. For example, the network 400 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Online, and the like.

2 is a block diagram of a virtual reality experience apparatus 200 according to an embodiment of the present invention, and FIG. 3 is a flow chart of a virtual reality experience method according to an embodiment of the present invention.

2 and 3, a virtual reality experience device 200 according to an embodiment of the present invention includes a motion data receiving unit 201, a body data generating unit 202, a motion recognition unit 203, and 3D data. A generation unit 204, a coordinate conversion unit 205, a virtual character generation unit 206, and an HMD coordinate matching unit 207 may be included.

The motion data receiving unit 201 may receive a plurality of motion data photographing a user's motion from different angles from the plurality of camera modules 100 (S10).

There may be at least four camera modules 100. For example, the viewing angle of the camera module 100 may be about 120 degrees, and in this case, the user may be photographed from all directions through the four camera modules 100.

The body data generation unit 202 may generate a plurality of body data by extracting a body excluding the user's head from each of the plurality of motion data. (S11) The body data generation unit 202 may generate a plurality of body data from the received motion data. It detects whether there is a human body, and if there is no human body, the process of motion recognition does not proceed because body data is not generated.

As an example of the present invention, a method in which the body data generator 202 detects and extracts a subject's body from motion data may employ a known object detection algorithm. The algorithm may detect a body using visible ray image information or thermal image information, and improve accuracy of body detection by using both image information at the same time.

The motion recognition unit 203 may generate a plurality of candidate regions by dividing a plurality of body data for each body part.

The motion recognition unit 203 may generate a candidate region by detecting a user's body part included in the body data and dividing it for each detected body part. The body part excludes the user's head.

The generation of the candidate region may be performed through a Selective Search Algorithm for regions in motion data where there may be body parts.

After generating a plurality of candidate regions, the motion recognition unit 203 extracts features for each of the candidate regions using a deep learning algorithm, recognizes joint points included in the candidate regions, and refers to the joint points. The user's motion can be recognized (S12).

The deep learning algorithm may be an algorithm through a region proposal-convolutional neural network (R-CNN) neural network. R-CNN is a Region Proposal-based CNN technique, which classifies what objects are in the candidate region and corrects the position of the object by using the features of the region proposal extracted from the image. The motion recognition unit 203 uses the cell labeling information given for each cell included in the motion data as training data, and performs a pre-processing process to determine the cells of each of the candidate regions whose data screen size is adjusted. The body part included in the corresponding candidate region can be recognized through the algorithm through this.

The algorithm through the R-CNN neural network can proceed like the selective search algorithm described above. In other words, after extracting candidate regions that may have objects in motion data using a selective search algorithm, each candidate region is input to a separate CNN to extract features of the candidate region, thereby classifying the types of objects present in each candidate region. , By narrowing the bounding box for the object, the detection position of the object can be precisely corrected.

The 3D data generation unit 204 may generate 3D data by merging a plurality of motion data for which motion is recognized based on the same body part and joint point (S13). For example, 3D data is generated. The unit 204 may generate 3D data by synchronizing boundary coordinates between candidate regions for each of a plurality of motion data in which motion is recognized, and then overlapping coordinates corresponding to joint points.

As mentioned above, since the user's motion is photographed in all directions through the plurality of camera modules 100, the 3D data generation unit 204 merges the common coordinates from the plurality of motion data so that the user’s all directions It is possible to generate 3D data that reflects the motion of.

The coordinate conversion unit 205 may convert the coordinates of the 3D data to match the coordinates of the simulation space (S13).

The virtual character generation unit 206 may generate a virtual character in the simulation space based on the 3D data in which the coordinates are converted (S14), and the corresponding virtual character may be displayed on the display unit of the display terminal 300 as well. The virtual character generation unit 206 may generate a virtual character to correspond to each of the coordinates of each body part of the 3D data in which the coordinates have been converted. In addition, a background image in which a simulation space is implemented may be displayed on the display unit of the display terminal 300. A background image and a virtual character included in the background image may be output together on the display unit of the display terminal 300.

The virtual character can take the same motion in real time in response to the user's motion.

Users can experience virtual reality in a simulation space by wearing a head mounted display (HMD). The HMD coordinate matching unit 207 may match the coordinates of the HMD with the coordinates of the head of the virtual character.

As described above, the virtual reality experience device 200 according to the present invention captures a user from all directions through a plurality of camera modules 100 to generate motion data, merges the motion data through a deep learning algorithm, and generates 3D data. Through the creation of a virtual character that takes the user's motion, the user can experience a more vibrant virtual reality.

The above-described embodiments are for illustrative purposes only, and those of ordinary skill in the art to which the above-described embodiments belong can easily transform into other specific forms without changing the technical spirit or essential features of the above-described embodiments. You can understand. Therefore, it should be understood that the above-described embodiments are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope to be protected through the present specification is indicated by the claims to be described later rather than the detailed description, and should be interpreted as including all changes or modified forms derived from the meaning and scope of the claims and the concept of equivalents thereof.

100: camera terminal 202: body data generation unit
200: virtual reality experience device 203: motion recognition unit
201: motion data receiving unit 204: 3D data generating unit

Claims (7)

In a virtual reality experience device for providing a virtual reality experience to the user by recognizing a motion of a user wearing a head mount display (HMD) in a simulation space,
A motion data receiving unit for receiving a plurality of motion data photographing the motion of the user from different angles from a plurality of camera modules;
A body data generator configured to generate a plurality of body data by extracting a body excluding the user's head from each of the plurality of motion data;
The plurality of body data is divided for each body part to generate a plurality of candidate regions, and a feature for each of the candidate regions is extracted using a deep learning algorithm to recognize joint points included in the candidate region, and the corresponding joint A motion recognition unit for recognizing a motion of the user based on a point;
A 3D data generator configured to generate 3D data by merging the plurality of motion data for which motion is recognized based on the same body part and joint point;
A coordinate conversion unit converting the coordinates of the 3D data to match the coordinates of the simulation space;
A virtual character generation unit that generates a virtual character in the simulation space based on the 3D data whose coordinates have been converted; And
An HMD coordinate matching unit matching the coordinates of the HMD to the coordinates of the head of the virtual character,
The plurality of camera modules is at least 4,
The virtual character generation unit generates the virtual character to correspond to each of the coordinates of each body part of the 3D data whose coordinates have been converted,
The virtual character takes the same motion in real time according to the motion of each body part of the user,
The 3D data generator generates 3D data by synchronizing boundary coordinates between candidate regions for each of the plurality of motion data for which motion is recognized, and then overlapping coordinates corresponding to the joint points to generate 3D data. Experience device. delete delete delete In a virtual reality experience method using a virtual reality experience device that recognizes a motion of a user wearing a head mount display (HMD) in a simulation space and provides a virtual reality experience to the user,
Receiving, by a motion data receiving unit, a plurality of motion data photographing a motion of the user from a plurality of camera modules at different angles;
Generating a plurality of body data by extracting a body excluding the user's head from each of the plurality of motion data by a body data generation unit;
A motion recognition unit divides the plurality of body data for each body part to generate a plurality of candidate regions, extracts features for each of the candidate regions using a deep learning algorithm, and recognizes joint points included in the candidate region. Recognizing the motion of the user based on the joint point;
Generating 3D data by merging the plurality of motion data for which motion is recognized by a 3D data generator based on the same body part and joint point;
Converting, by a coordinate conversion unit, the coordinates of the 3D data to match the coordinates of the simulation space;
Generating, by a virtual character generation unit, a virtual character in the simulation space based on the 3D data whose coordinates are converted; And
An HMD coordinate matching unit comprising the step of matching the coordinates of the HMD to the coordinates of the head of the virtual character,
The plurality of camera modules is at least 4,
In the step of generating the virtual character, the virtual character generation unit generates the virtual character so as to correspond to each of the coordinates of each body part of the 3D data whose coordinates have been converted,
The virtual character takes the same motion in real time according to the motion of each body part of the user,
In the step of generating the 3D data, the 3D data generation unit synchronizes boundary coordinates between candidate regions with respect to each of the plurality of motion data for which motion is recognized, and then overlaps the coordinates corresponding to the joint points to generate 3D data. Virtual reality experience method, characterized in that to generate a.
The virtual reality experience method, characterized in that the virtual character takes the same motion in real time in response to the motion of the user.

delete delete

Images