KR20200008400A - System for providing virtual experience space having contents generated by user - Google Patents

Abstract

The present invention relates to a system for providing a virtual experience space including user-generated content, capable of providing a virtual experience space, which provides an individual interaction effect of a user, which comprises: a plurality of output surfaces for outputting output content by projectors; a plurality of projectors installed in different directions; a Kinect camera; an input display for creating virtual content; a head mount worn on the head of a user; and a control system.

Description

System for providing virtual experience space containing user-generated content {SYSTEM FOR PROVIDING VIRTUAL EXPERIENCE SPACE HAVING CONTENTS GENERATED BY USER}

The present invention relates to a system for providing a virtual experience space for creating a mixed reality including contents generated by an experienced person and providing the experienced person. Among the 2015 industry-academic cooperation business enterprise growth technology development support project, 'Interactive Realistic Media Technology Convergence Science' Learning Experience Center Service Development 'is the invention about the result that is produced in performing the task.

In recent years, with the rapid development of the multimedia environment, a technology for providing a virtual image to the user has already been widely spread.

Recently, a participant directly participates in the provided virtual space, and a technology for interacting with the contents of the virtual space has been developed.

Accordingly, the virtual space is provided with a physical space for the user to participate, the wall of the physical space is implemented to output the content, output the content on the space, the user participates in the space And interact with the content, and the reproduced content is output according to the result of the interaction.

In order to provide such a realistic experience space, a technology for basically outputting content on a physical three-dimensional space and a technology for processing a user command through an action of a participant should be provided.

At this time, the method for recognizing a participant's user command for the participant interaction includes a method in which a user wears a wearable device and a method of recognizing participant's behavior using a depth camera such as a Kinect camera commercialized with the user's action.

The user's command input using such a depth camera (Kinect camera) has been disclosed in Korean Patent Application Laid-Open No. 10-2016-0075884 and the prior patent application No. 10-2016-0104053 of the present patent.

The present invention relates to a method of recognizing a user's motion by a depth camera, in the case of a method of recognizing a user's motion by using a depth camera, one user (player, experienced person) is assigned and used in one space. The way is main.

This is because in order to provide a user interaction, the motion of the user must be recognized, because when a plurality of users are active in one experience space, errors are frequently generated in user specification due to overlapping recognition results of the users.

In particular, in the case of providing individual virtual content for each user, when an error occurs in user-specific, virtual content generated for another user is provided, thereby providing a virtual content that is not suitable for context.

(001) Republic of Korea Patent Publication No. 10-2016-0075884 (002) Republic of Korea Patent Application No. 10-2016-0104053

The present invention has been made to solve the above conventional problems, the present invention provides a display device that can create a virtual content from the user in a virtual reality experience space that can be experienced by multiple users, the user It is to provide a virtual experience space providing system that can provide each user with a mixed reality added to the virtual content created by himself.

In addition, the present invention is to provide a virtual experience space providing system that can minimize the error of recognition in providing the virtual content to the user who created it, the user who created the virtual content among a plurality of users. .

In addition, the present invention generates the mixed reality to be provided to the user from the output content output on the multi-screen, so that it is possible to accurately and quickly determine the edge area of the output area through a plurality of projectors to obtain an integrated reality image It is to provide a system for providing a virtual experience space.

According to a feature of the present invention for achieving the above object, the present invention comprises: a plurality of output surfaces are installed in different directions to form an experience space therein, the output content is output by the projector; A plurality of projectors installed in different directions to output output contents on the output surface; A Kinect camera installed toward the experience space to recognize a user object located in the experience space; An input display installed at a rear side of the experience space to output the output content, and to generate virtual content to form mixed reality content according to an input value input from the user; A head mounted display which is worn on the head of the user, acquires a real image, and provides the generated mixed reality to the user; And outputting the output contents of the projectors, receiving the position and shape of the dynamic object recognized from the Kinect camera, and merging the virtual image generated from the input image with the real image from the head mount to generate a mixed reality. And a control system for providing the generated mixed reality to the head mounted display.

In this case, the control system may be characterized by generating a user of the virtual content generated through the input display, and may generate and provide the mixed reality for each user.

And the control system is configured to generate skeleton data from depth information of the user included in the shooting information of the Kinect camera; The shape and operation of the user object may be grasped from the skeleton data.

The control system may detect an overlapping area in which output content is output by two or more projectors on the output surface, and specify only one of a plurality of projectors for the detected overlapping area to output the output content. You can also control the output of the projector.

The detection of the overlapping area of the control system may include outputting colors of different colors to a plurality of projectors for detecting the overlapping area, and generating colors by overlapping the output colors of the image obtained from the Kinect camera. It may be detected by determining whether there is an area where the in edge color is output.

In addition, the output color may be selected from among R (Red), G (Green), and B (Blue) light source having a clear contrast according to the addition mixing principle of light.

And the control system changes the color of the output light source from the output color to the edge color for some of the output pixels of any one of the projectors, and examines the output color increase / decrease rate on the recognition screen of the Kinect camera. The output area of the projector may be set by detecting the outgoing pixel in which the output color disappears.

In addition, the control system may identify the user of the input event of the input display in contrast to the front direction of the user and the screen direction of the input display.

In addition, the front direction of the user may be determined by calculating an average vector value with respect to the amount of vector movement of a position corresponding to a specific body part among the stored and stored skeletons.

In addition, the input display may include a touch panel.

In the virtual experience space providing system including user-generated content according to the present invention as described above, the following effects can be expected.

That is, in the present invention, while providing a complete virtual space by outputting the content to the rear through an additional display device in a virtual reality experience space that can be experienced by multiple users at the same time, the virtual content generated directly by the user through the additional display device Since the mixed reality can be provided to each user individually, there is an effect that can provide a virtual experience space in which the individual interaction effect of the user is provided.

In the present invention, it is possible to accurately specify the user who created the virtual content among the plurality of users, it is possible to provide the generated virtual content to the user individually, there is an effect of improving the accuracy of the interaction.

In addition, in the present invention, when outputting content using a plurality of projectors on a multi-screen, the generated edge area can be detected by dividing by the pixel position of the projector, thereby ensuring a perfectly processed edge area. .

1 is an exemplary view showing the configuration of a virtual experience space providing system according to the present invention.
Figure 2 is an exemplary view showing a form in which a user is recognized by the Kinect camera according to the present invention.
3 is an exemplary view illustrating a form in which a user is recognized by a Kinect camera when a plurality of users are located on an input display of the present invention.
Figure 4 is a flow diagram illustrating a specific embodiment of a method for providing a mixed reality through a virtual experience space providing system according to the present invention.
5 is an exemplary view showing a user specific principle in the method for providing mixed reality according to the present invention.
6 is an exemplary view showing a principle of output area allocation in the method for providing mixed reality according to the present invention.

Hereinafter, a virtual experience space providing system including user-generated content according to a specific embodiment of the present invention will be described with reference to the accompanying drawings.

Prior to the description, the effects, features and methods of achieving the same of the present invention will be apparent in the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the embodiments are to make the disclosure of the present invention complete, and the general knowledge in the technical field to which the present invention belongs. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims.

In describing the embodiments of the present disclosure, when it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted, and the following terms are used in the embodiments of the present disclosure. Terms defined in consideration of the function of the may vary depending on the user or operator's intention or custom. Therefore, the definition should be made based on the contents throughout the specification.

Combinations of each block and each step of the flowchart in the accompanying block diagrams may be performed by computer program instructions (execution engine), which may be executed on a processor of a general purpose computer, special purpose computer or other programmable data processing equipment. As such, instructions executed through a processor of a computer or other programmable data processing equipment create means for performing the functions described in each block of the block diagram or in each step of the flowchart.

These computer program instructions may be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, and thus the computer usable or computer readable memory. It is also possible for the instructions stored therein to produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram or in each step of the flowchart.

And computer program instructions can also be mounted on a computer or other programmable data processing equipment, such that a series of operating steps can be performed on the computer or other programmable data processing equipment to create a computer-executable process that can be executed by the computer or other programmable data processing equipment. Instructions for performing data processing equipment may also provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart.

In addition, each block or step may represent a portion of a module, segment or code that includes one or more executable instructions for executing specific logical functions, and in some alternative embodiments referred to in blocks or steps It is also possible that functions occur out of order.

That is, the two blocks or steps shown may in fact be executed substantially concurrently, and the blocks or steps may be performed in the reverse order of the corresponding functions as required.

1 is an exemplary view showing a configuration of a system for providing a virtual experience space according to the present invention, FIG. 2 is an exemplary view showing a form in which a user is recognized by a Kinect camera according to the present invention, and FIG. When a plurality of users is located on the input display constituting the configuration, it is an exemplary view showing the user is recognized by the Kinect camera, Figure 4 is a method for providing a mixed reality through the virtual experience space providing system according to the present invention 5 is a flowchart illustrating a specific embodiment of the present invention, FIG. 5 is an exemplary diagram illustrating a user specific principle in the method of providing mixed reality according to the present invention, and FIG. 6 is a method of providing the mixed reality according to the present invention. It is an exemplary diagram showing the principle of output area allocation.

First, as shown in FIG. 1, the system for providing a virtual experience space according to the present invention includes multiple output surfaces 110, 120, 130,..., An input display 200, and a projector 300 constituting the experience space. , Kinect camera 400, the control system 500, the head mounted display 600 is configured.

The experience space is a three-dimensional space composed of multi-directional walls, as shown, each wall is composed of output surfaces on which the content is output.

At this time, the output surface is basically configured to include a front (first output surface, 110), left and right side (second output surface, 120), preferably further comprises a bottom surface (third output surface, 130). It is configured by.

In addition, the ceiling surface may be further configured according to the installation environment, but the ceiling surface may be excluded according to the installation space of the projector 300 and the Kinect camera 400 which will be described later.

On the other hand, in general, the rear surface of the experience space is configured as an open surface for the user's access and the internal observation of the administrator, in the present invention by placing the input display 200 consisting of a large touch screen on the back of the input display Through 200, the output content can be output on the rear surface.

In this case, in addition to outputting the output content, the input display 200 allows the user to directly create content and add it to the mixed reality through the touch screen.

That is, the user generates a virtual content preferred by the user through the input display 200 and is provided with a mixed reality including the generated virtual content.

For example, a user who creates a butterfly as virtual content on the input display 200 among a plurality of experience space users is provided with a mixed reality including virtual content about a butterfly flying in a real image, and in the input display 200. The user who created the car as the virtual content is provided with the mixed reality including the virtual content of the moving car in the real image.

Here, the reality image is an image captured by the head mounted display (hereinafter, referred to as 'HMD') 600 worn by the user on each output surface on which the output content is output. The reality image is generated by the user. One virtual content is merged to create mixed reality content.

To this end, a projector 300 for outputting output content is provided on each output surface. Preferably, the projector 300 includes the entire experience space according to the output range and installation position according to the angle of view of each projector 300. Installation position and number are set to cover.

On the other hand, the Kinect camera 400 measures the user's motion and the user's depth, to recognize and track the user (experienced person) located in the space.

Here, the Kinect camera 400 is a representative example of a commercially available depth camera that recognizes the shape and distance of an object (subject) and detects movement of the object, and includes various depth cameras capable of performing the same function. Used in the sense.

Here, the Kinect camera 400 may be configured as a single Kinect camera, or may be configured with a plurality of Kinect cameras in order to improve the detection accuracy of the object.

On the other hand, the control system 500 controls to output the output content on the output surface by controlling the driving of each projector 300, matching the virtual content generated through the input display 200 with the user, This part allows the user to receive mixed reality content including virtual content.

To this end, the control system 500 is configured to include a database (not shown), and the database stores various content data for generating output content and virtual content.

In addition, the control system 500 recognizes the user recognized by the Kinect camera 400 as an object, specifically, the control system 500 is a user through the photographing data received from the Kinect camera 400 Skeleton data of the (dynamic object) is calculated, and the position and operation of the user are determined based on the skeleton data.

That is, the control system 500 generates skeleton data of the dynamic object using the depth information included in the photographing data received from the Kinect camera 400, and grasps the time-series change of the generated skeleton data. To determine the location and operation of the user.

As such, an example of the skeleton data of the user recognized by the control system 500 is illustrated in FIG. 2.

Such grasping operation using skeleton data has excellent advantages in terms of accuracy and data processing efficiency. However, when another object is located close to the object, it is recognized in specifying the object due to cross recognition between the objects. There is a problem that errors are easily generated.

In detail, as illustrated in FIG. 3, when a plurality of users are located in front of the input display 200, due to the cross recognition between the skeleton data 810 of the first user and the skeleton data 820 of the second user, There is a high possibility that an error may occur in specifying whether the creator of the virtual content generated through the input display 200 is a first user or a second user.

Therefore, there is a fear that the generated virtual content may be wrongly provided to the adjacent user, not the user who created the virtual content.

The driving method of the control system for preventing such a user specific error will be described in detail again.

Meanwhile, the HMD 600 is a personalized display device worn by a user. The HMD 600 is worn on the user's head to acquire a reality image according to the user's gaze direction, and adds the generated virtual content to the obtained reality image to add mixed reality content. This is the part to create and provide to the user.

The HMD 600 is commercially provided with various kinds of products, and various types of HMD 600 may be applied to the present invention.

Hereinafter, a virtual experience space providing method including user-generated content according to the present invention as described above will be described in detail with reference to the accompanying drawings.

First, as shown in FIG. 4, the method for providing a virtual experience space according to the present invention starts by selecting the output content to be output on each output surface composed of multiple surfaces by the control system 500 (S110).

Thereafter, the control system 500 drives the projector 300 to output output contents by dividing the output areas allocated to the plurality of projectors 300 (S120).

At this time, the output content output to each output surface is the data that is input through the HMD (600) to form a reality image, the edge processing for the contact area generated between each output surface to maximize the reality (edge blending) Is very important.

Accordingly, in the present invention, the control system 500 detects an overlapping area, and designates only one of the plurality of projectors 300 to the detected overlapping area, so that only one projector outputs output content to the corresponding area. Remove it.

Here, the overlapping area detection method of the control system 500 is a method of utilizing the addition mixing principle of light, and selects a plurality of projectors 300 to detect the overlapping area, and outputs different colors to each projector 300. Set the color to print.

In addition, the Kinect camera 400 recognizes an output screen (spatial display), and whether there is an area in which the color generated by the superposition of the output colors (hereinafter referred to as 'edge color') exists in the recognized output screen. Determine.

In this case, as shown in FIG. 6, when the edge color is recognized, the corresponding areas become overlapping areas D13 and D12. This means that the overlapped edge regions are generated by the projector 300. If the edge colors are not recognized, the edge regions are not generated.

Therefore, it is preferable that at least two of R (Red), G (Green), and B (Blue) light sources having a clear contrast with the edge color are selected according to the addition mixing principle of light.

Meanwhile, when the control system briefly describes the process of allocating the projector to the edge area, as shown in FIG. 6, when an edge color exists in the output surface, the control system 500 outputs the pixels of the projector 300. Among them, an output area is set by selecting pixels constituting an edge area, and the process is performed by the control system 500 by selecting a part of one of the projector pixels (hereinafter, referred to as a “first projector”) of the projectors 300. The color of the output light source is changed from the output color to the edge color, but gradually the pixel output color of the projector is changed to the edge color until all the output color disappears in the output surface.

More specifically, the control system 500 outputs an output color through the first projector, and then changes some pixels from the output color to the edge color. Then, the rate of increase or decrease of the output color on the recognition screen of the Kinect camera is examined.

The pixel color of the projector is set to an output color or an edge color in a direction in which the increase / decrease rate of the output color decreases, and these processes are repeatedly performed until the output color is not recognized.

Subsequently, when the output color disappears on the recognition screen, pixels in which the output color is output among the output pixels of the first projector in the corresponding state are set as overlap regions.

This process is repeated for all the projectors, and the detected overlapping area is set such that only one of the projectors is responsible for the output.

Accordingly, in the present invention, while outputting one content using a plurality of projectors, there is an effect that can effectively remove the overlapping area.

Thereafter, the control system 150 determines whether a user input is generated in the input display 200 (S130).

When an input event occurs in the input display 200, the control system 500 identifies and specifies a user who generates a corresponding input event among users who are tracked using the Kinect camera 400 (S140).

In this case, when there is only one user adjacent to the input display 200, the identification of the user is easy. However, as illustrated in FIG. 3, when a plurality of users are located, an identification error may be high.

Accordingly, the present invention can determine the front direction of the user adjacent to the screen of the input display 200, the user located behind the input display 200 can reduce the identification error by excluding from identification.

Here, the grasp of the front direction of the user may be determined from the vector direction of the skeleton motion information of the user (dynamic object) received from the Kinect camera 400.

That is, by calculating an average vector value with respect to the vector movement amount of the accumulated and stored skeleton, the direction of the dynamic object can be determined according to the direction in which the average vector value is directed.

For example, as shown in FIG. 5, when the determination portion is an arm, a vector average of the arm portion of the skeleton is calculated, and the direction of the average vector becomes the front surface of the dynamic object.

On the other hand, when the judgment site is the foot, a vector average of the foot site of the skeleton is calculated, and the opposite direction of the average vector becomes the front face of the dynamic object.

Of course, such a criterion is applied when the dynamic object is a human, and the criterion may be set differently according to the type of the dynamic object.

As such, when user identification and specification of an input on the input display 200 are made, virtual content to be provided to the specified user is generated by the input of the input display 200 (S150).

The control system 500 or the HMD 600 worn by the user merges the acquired real image and the specified virtual content to generate mixed reality content (S170).

The mixed reality thus generated is provided to the user through the HMD 600.

As such, since the user directly generates the virtual content and the generated virtual content is included in the mixed reality, which is specific to the producer, a plurality of experiences simultaneously experience the experience in one experience space while providing a customized mixed reality according to the user's preference. There is an effect that can be provided separately.

The rights of the present invention are not limited to the embodiments described above, but are defined by the claims, and those skilled in the art can make various modifications and adaptations within the scope of the claims. It is self-evident.

The present invention relates to a virtual experience space providing system for generating a mixed reality including the content generated by the experience to provide to the experience, in the present invention, an additional display in the virtual reality experience space that can be experienced by multiple users at the same time By providing contents to the rear through the device to provide a complete virtual space, an additional display device can provide each user with a mixed reality to which the virtual content created by the user is added. There is an effect that can provide the provided virtual experience space.

110, 120, 130: output surface 200: input display
300: projector 400: Kinect camera
500: control system 600: head mounted display

Claims (5)

A plurality of output surfaces installed in different directions to form an experience space therein and outputting output contents by a projector;
A plurality of projectors installed in different directions to output output contents on the output surface;
A Kinect camera installed toward the experience space to recognize a user object located in the experience space;
An input display installed at a rear side of the experience space to output the output content, and to generate virtual content to form mixed reality content according to an input value input from the user;
A head mounted display which is worn on the head of the user, acquires a real image, and provides the generated mixed reality to the user; And
Outputting the output contents of the projectors, receiving the position and the shape of the dynamic object recognized from the Kinect camera, generating the mixed reality by merging the virtual image generated from the input image and the real image from the head mount; And a control system for providing the generated mixed reality to the head mounted display.
The method of claim 1,
The control system,
Characterized by the user of the generation of the virtual content generated through the input display, and a virtual experience space providing system comprising the user-generated content, characterized in that for generating and providing the mixed reality for each user.
The method of claim 1,
The control system,
Generating skeleton data from depth information of a user included in the shooting information of the Kinect camera;
And a user-generated content, characterized in that determining the shape and operation of a user object from the skeleton data.
The method according to any one of claims 1 to 3,
The control system,
Detecting an overlapping area in which output content is output by two or more projectors on the output surface, and controlling the output of the projector to output the output content by designating only one of a plurality of projectors for the detected overlapping area; A system for providing a virtual experience space comprising user-generated content.
The method of claim 4, wherein
The input display,
System for providing a virtual experience space including user-generated content, characterized in that configured to include a touch panel.

Images