KR102430463B1 - Preemptive mixed reality experience system and control method thereof - Google Patents

Abstract

The present invention relates to a preemptive mixed reality experience system and a control method therefor. A method for controlling a preemptive mixed reality experience system according to the present invention comprises the steps of: determining a location or behavior of a mixed reality experiencer within a preset area; extracting virtual content corresponding to the determined location or behavior of the mixed reality experiencer; and controlling the extracted virtual content to be displayed on the display device of the mixed reality experiencer, and extracting specific virtual content by additionally considering the number of preemptions for each virtual content.

Description

Preemptive mixed reality experience system and its control method

The present invention relates to a mixed reality experience system and a control method therefor, and more particularly, to a mixed reality experience system and a control method thereof for constructing storytelling using a preemptive method.

Virtual reality (VR) literally means a virtual space that is implemented so that an experience similar to reality can be experienced. It refers to a technology for creating the illusion of seeing the relevant object or object in a virtual space.

In this case, the user (experiencer) may directly or indirectly move or interact with a specific thing or object in the virtual space.

Augmented reality (AR: Augment Reality) is a form of adding a virtual image to real reality. For example, augmented reality collects real information (position information is obtained through GPS, tilt or the speed of the device moving). is obtained through a gyroscope, etc.) It has a high sense of reality because it shows a virtual image based on the collected real information, and there is no or less dizziness when wearing a VR device.

On the other hand, mixed reality (MR: Mixed Reality) is also called convergence reality, which means that virtual reality and augmented reality are integrated and operated for a large number of users or only one is operated as needed.

In this way, in providing the mixed reality service, virtual content such as predetermined virtual reality content or augmented reality content is provided to the experiencers according to the location or behavior of the experiencer, and the overall storytelling is achieved.

For example, when a mixed reality service is applied to a theme park, various virtual contents can be viewed in a virtual or augmented reality method as the experiencers move to various places in the theme park. The interactions of the experiencers with these virtual contents are gathered to form a single storytelling.

However, if the same virtual contents are provided according to the current location of the experiencers or the specific actions of the experiencers, visitors who visit the space that provides the mixed reality service (the above-mentioned theme park) will have the same storytelling experience, which reduces the fun or interest. , especially those who have revisited the experience space feel bored by the same storytelling.

Patent Publication No. 10-2019-0032084

The present invention has been devised to solve the problems of the prior art, and its purpose is to provide a different storytelling configuration through provision of different virtual contents on the premise that predetermined conditions are satisfied even for those who have performed the same action in the same location. It is to provide a preemptive mixed reality experience system that makes this possible.

In order to achieve the above object, a preemptive mixed reality experience system according to the present invention includes: a determination unit for determining the location or behavior of a mixed reality experiencer within a preset area; a content extraction unit for extracting virtual content corresponding to the location or behavior of the mixed reality experiencer determined by the determination unit; and a display control unit for controlling the display device of the mixed reality experiencer to display the virtual content extracted by the content extraction unit, wherein the content extraction unit selects specific virtual content by additionally considering the number of preemptions for each virtual content. characterized by extraction.

Here, the display control unit may accumulate and store the number of preemptions for the corresponding virtual content whenever the virtual content extracted by the content extraction unit is controlled to be displayed.

Here, the content extraction unit may include: a confirmation unit for confirming the number of preemptions for virtual content that primarily corresponds to the location or behavior of the mixed reality experiencer determined by the determination unit; and an extraction unit for extracting virtual content secondary to the location or behavior of the mixed reality experiencer when the number of preemption times confirmed by the confirmation unit exceeds a preset value.

Here, the content extraction unit may include: a confirmation unit configured to confirm the number of preemptions for virtual content corresponding to the location or behavior of the mixed reality experiencer determined by the determination unit; When the number of preemptions checked by the confirmation unit exceeds a preset value, the corresponding virtual content is deactivated, and other virtual content according to the preset order is converted into virtual content corresponding to the location or behavior of the mixed reality experiencer. It may include an extraction unit for extracting by setting.

Here, the virtual content may be at least one of augmented reality content and virtual reality content.

In addition, in order to achieve the above object, a control method of a preemptive mixed reality experience system according to the present invention includes the steps of: determining the location or behavior of a mixed reality experiencer in a preset area; extracting virtual content corresponding to the determined location or behavior of the mixed reality experiencer; and controlling the extracted virtual content to be displayed on the display device of the mixed reality experiencer, and extracting specific virtual content by additionally considering the number of preemptions for each virtual content.

Here, whenever the control to display the extracted virtual content is controlled, the number of preemptions for the virtual content may be accumulated and stored.

Here, checking the number of preemptions for virtual content that primarily corresponds to the determined location or behavior of the mixed reality experiencer; The method may include extracting virtual content secondarily corresponding to the location or behavior of the mixed reality experiencer when the confirmed number of preemption exceeds a preset value.

Here, checking the number of preemptions for virtual content corresponding to the determined location or behavior of the mixed reality experiencer; When the confirmed number of preemption exceeds a preset value, the corresponding virtual content is deactivated, and other virtual content according to the preset order is set as virtual content corresponding to the location or behavior of the mixed reality experiencer. It may include the step of extracting.

Here, the virtual content may be at least one of augmented reality content and virtual reality content.

As described above, according to the present invention, mixed reality experiencers can significantly increase satisfaction in using augmented reality content.

That is, by managing information on the number of preemption times for virtual contents constituting each storytelling, and deactivating or replacing virtual contents exceeding a preset value with other virtual contents, the same place, the same It is possible to provide different virtual contents to mixed reality participants who are acting, so that they can have various and different experiences.

This also achieves the effect of increasing customer satisfaction by providing a storytelling experience that is not the same every time to customers who have visited the theme park or the like in which the system according to the present invention is constructed.

1 is a schematic configuration diagram of an entire system including a mixed reality experience system according to an embodiment of the present invention;
Figure 2 is a functional block diagram of the mixed reality system of Figure 1,
3 is a view showing a state in which a part is inactive in consideration of the combination state of each virtual content constituting the storytelling and the number of preemption,
4 and 5 are control flowcharts of a mixed reality system according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, each embodiment according to the present invention is merely an example for helping understanding of the present invention, and the present invention is not limited to these embodiments. In particular, the present invention may be composed of a combination of at least any one of individual components, individual functions, or individual steps included in each embodiment.

In particular, for convenience, some of the claims include an alphabet such as '(a)', but the alphabet does not define the order of each step.

In addition, each signal referred to in each embodiment according to the present invention may mean one signal transmitted by one connection, etc., but may also mean a series of signal groups transmitted for the purpose of performing a specific function to be described later. . That is, in each embodiment, a plurality of signals transmitted at a predetermined time interval or transmitted after receiving a response signal from a counterpart device may be expressed as one signal name for convenience.

A schematic configuration of the entire system including the mixed reality experience system 10 according to an embodiment of the present invention is shown in FIG. 1 .

First, in describing the present embodiment, it is assumed that a specific space, that is, a specific space for experiencing mixed reality exists, and that at least one experiencer is included therein.

A specific space to experience mixed reality may have a boundary, for example, a wall 51 to separate it from the outside or other experiential space 50 . Content may be displayed.

In order to display virtual reality content, the wall 51 of the experience space 50 may be configured to perform its own display function by means of LEDs, or an image by a projector provided outside is displayed on the wall of the experience space 50 . (51) may also be indicated.

As such, the display of the predetermined virtual reality content on the predetermined wall 51 corresponds to a known technology, and thus a more detailed description thereof will be omitted.

In FIG. 1 , a Head Mounted Display (HMD) 40 is worn by the experiencer, that is, the mixed reality experiencer according to the present invention, and performs a function of displaying content received from the mixed reality experience system 10 through wireless communication. .

For example, when the mixed reality experience system 10 transmits augmented reality content to the HMD 40 worn by each mixed reality experiencer, the HMD 40 displays the augmented reality content received from the mixed reality experience system 10 on the display unit. mark it so that the mixed reality experiencer can see it.

Here, since the display unit of the HMD 40 is made of transparent glass, it may be configured to view the actual situation outside.

In FIG. 1 , the HMD 40 and the mixed reality experience system 10 communicate through a predetermined access point 20 as an example, but the communication means is not limited thereto.

Meanwhile, in FIG. 1 , the surveillance camera 30 captures the location of the mixed reality experiencer and transmits it to the mixed reality experience system 10 .

Of course, coordinate information received from a terminal equipped with a mixed reality experiencer may be used for the method of determining the location of the mixed reality experiencer, but as shown in FIG. 1 , by analyzing the image captured by the surveillance camera 30 It is also possible to judge the position of the experiencer. Since the technique of determining the position coordinates of a specific object through analysis of the captured image also corresponds to a known technique, a more detailed description thereof will be omitted.

In addition, the image captured by the monitoring camera 30 may be used for analyzing a specific behavior of the experiencer.

On the other hand, in FIG. 1 , the mixed reality experience system 10 provides virtual reality content and/or augmented reality content to an experiencer in a preset area to have a new experience, and in this embodiment, as described above, It is a task to display virtual content on a wall (for example, a wall surrounding the corresponding area) 51 and provide augmented reality content to a display device (HMD 40 in this embodiment) carried or worn by the experiencer. do for example

An example of a specific functional block of the mixed reality experience system 10 is shown in FIG. 2 .

As shown in the drawing, the mixed reality experience system 10 may include a determination unit 11 , a content extraction unit 12 , a display control unit 13 , and a storage unit 14 .

Here, the storage unit 14 functions to store applications, setting data, and contents necessary for the operation of the mixed reality experience system 10 , as well as newly generated or received from the outside during the operation of the mixed reality experience system 10 . It performs a function of storing information, data, or content.

Hereinafter, even if there is no separate expression for the storage unit 14 , it is assumed that the storage unit 14 is used when processing functions such as extracting necessary content or setting predetermined data.

Here, the determination unit 11 performs a function of determining the location or behavior of the mixed reality experiencer in the preset area (ie, the above-described experience space).

For example, the determination unit 11 may analyze the image captured by the monitoring camera 30 as described above to determine the current location coordinates of the experiencer and the specific behavior of the experiencer. In particular, although not shown in FIG. 1 , the specific behavior of the experiencer may be determined by receiving a sensing signal from a sensor provided in gloves or clothes worn by the experiencer.

In addition, judging the location of the subject that did not exist before, that is, the location of the participants who entered the set area through the analysis of the image captured by the surveillance camera 30, or judging the action itself corresponds to a known technology, so a more detailed description is omit

The content extraction unit 12 performs a function of extracting virtual content corresponding to the location or behavior of the mixed reality experiencer determined by the determination unit 11 .

Here, the virtual content may be augmented reality content displayed on the HMD 40 worn by the experiencer, or may be virtual reality content displayed on the wall of the experience space.

For example, assuming that a specific virtual content is matched to the location or specific action of the mixed reality experiencer and stored in the storage unit 14, the content extraction unit 12 detects the location or specific action of the mixed reality experiencer. In this case, the matching virtual content can be extracted.

That is, the content extraction unit 12 can change the type of virtual content provided according to the behavior or location of each mixed reality experiencer even if the mixed reality experiencer is in the same area (ie, the experience space). will be.

Here, the content extraction unit 12 may additionally consider the number of preemptions for each virtual content when extracting the virtual content.

In order to perform this function, the content extraction unit 12 may be configured to include a confirmation unit 15 and an extraction unit 16 .

Here, the confirmation unit 15 performs a function of confirming the number of preemptions for virtual content that primarily corresponds to the location or behavior of the mixed reality experiencer determined by the determination unit 11 .

Here, 'responding primarily' means that it is most closely related to the location or behavior of the mixed reality experiencer.

And the number of preemption for virtual content means the number of times the virtual content has already been extracted or provided so that other users can use it.

The extraction unit 16 performs a function of extracting the virtual content if the number of preemptions checked by the verification unit 15 does not exceed a preset value, whereas the number of preemptions checked by the verification unit 15 is set If it exceeds the set value, it performs a function of extracting virtual content that is secondary to the location or behavior of the mixed reality experiencer.

Here, the meaning of 'secondarily responding' means that it is secondarily closely related to the location or behavior of the mixed reality experiencer.

For example, the content 'C1' matched with the highest priority for the specific behavior of the mixed reality experiencer and the content 'C2' matched with the next priority are stored in the storage unit 14, and the determination unit 11 When the specific action of the mixed reality experiencer is detected according to the determination of If the number of preemption of the checked C1 content does not exceed the preset value, the C1 content is extracted, but if the number of preemption of the C1 content exceeds the preset value, the C2 content matched with the next priority is extracted. .

Of course, here, before the extraction unit 16 extracts the C2 content, the confirmation unit 15 checks the number of preemption of the C2 content, and only when the number of preemption of the corresponding C2 content does not exceed a preset value, the extraction unit ( 16) may extract C2 content.

As another example, when the number of preemptions checked by the confirmation unit 15 exceeds a preset value, the extractor 16 inactivates the corresponding virtual content, and adds other virtual content according to the preset order to the mixed reality. It can also be extracted by setting it as virtual content corresponding to the location or behavior of the experiencer. A more detailed description thereof will be provided later.

On the other hand, the display control unit 13 performs a function of controlling to display the virtual content extracted by the content extraction unit 12 on the display device of the mixed reality experience.

For example, the display control unit 13 may perform a function of controlling the virtual reality content in the corresponding zone to be displayed on the wall 51 of the preset zone. Here, the display method of the virtual reality content is as described above. Various methods such as direct display through LED or indirect display through a projector may be used.

In addition, the display controller 13 may control the augmented reality content displayed on the HMD 40 worn by the experiencer to be displayed.

In particular, the display control unit 13 may accumulate and store the number of preemptions for the corresponding virtual content whenever the virtual content extracted by the content extraction unit 12 is controlled to be displayed. As described above, can be used in the content extraction unit 12 .

Of course, the accumulated storage of the number of preemptions for each virtual content may be performed by the content extraction unit 12 instead of the display control unit 13 .

3 shows a process in which different storytelling is performed by combining each virtual content according to the operation of the mixed reality experience system 10 as described above.

First, FIG. 3(a) shows a process in which different storytelling occurs through sequential combinations of virtual content provided according to the location or specific behavior of the experiencer.

Briefly describing FIG. 3( a ), each circular figure such as A1 and B1 corresponds to a specific event (eg, location-based or action-based event, etc.) Augmented reality contents), and storytelling can be made by a combination of these virtual contents, and if we look at some of them, storytelling that leads to A1->B1->C1->D1 and A1->B1-> Storytelling from C2->D2, storytelling from A1->B2->C3->D3, storytelling from A1->B2->C3->D6, storytelling from A1->B2->C4->D5 Continuing storytelling is possible.

For example, after the virtual content A1 is provided to the experiencer, the contents of B1, B2, B3, B4, and B5 may be provided according to the specific behavior of the user, and the subsequent process is also the same.

For example, after content B1 is provided to the experiencer, one of C1, C2, and C3 may be provided according to the additional behavior of the experiencer, and after the B2 content is provided to the experiencer, C3, C4, One of C5's content can be provided.

3(b) shows a process in which other virtual content is selected and provided instead of the virtual content to be originally selected according to the number of preemption times.

For example, after providing the virtual content A1 to the experiencer, if the user raises one arm, the content B1 is matched, and if the user raises both arms, the content B2 is provided. Assuming that they are matched, even if the experiencer raises one arm, if the number of preemptions for B1 content exceeds a preset value, the mixed reality experience system 10 can provide the experiencer with B2 content instead of B1 content. there will be

In this case, since it is impossible to connect to B1 following A1, storytelling that leads to A1->B1->C1->D1 and A1->B1->C2->D2 among storytelling by virtual content combination Storytelling, A1->B1->C1->D3 storytelling, and A1->B1->C2->D4 storytelling cannot be provided to the experiencer temporarily or continuously.

On the other hand, storytelling that leads from A1->B2->C3->D3 or storytelling from A1->B2->C4->D5 is possible.

Hereinafter, the overall control flow of the preemptive mixed reality experience system 10 according to an embodiment of the present invention will be described with reference to FIG. 4 .

The preemptive mixed reality experience system 10 determines the location or behavior of the mixed reality experiencer (step S1), and determines whether the location or action of the mixed reality experiencer meets the event occurrence condition (step S3).

As a result of the determination, when the event occurrence condition is satisfied, the preemptive mixed reality experience system 10 selects the first augmented reality content corresponding to the event (step S5).

Next, the preemptive mixed reality experience system 10 corresponds to the next order (here, the order may be predetermined) when the number of preemption for the selected first augmented reality content exceeds a preset value (step S7). Augmented reality content, that is, the second augmented reality content is selected (step S9), and it is determined whether the number of preemptions for the selected second augmented reality content exceeds a preset value (step S7).

As such, the process of selecting the augmented reality content and determining the number of preemption may be repeated until the determined number of preemption does not exceed a preset value.

Thereafter, the preemptive mixed reality experience system 10 extracts the finally selected augmented reality content (step S11), and at this time, the number of preemptions for the selected augmented reality content may be updated (that is, increased by the accumulated value by 1). (Step S13).

The preemptive mixed reality experience system 10 may control the extracted augmented reality content to be displayed on the user's HMD 40 (step S15).

5 shows the control flow of the preemptive mixed reality experience system 10 according to another embodiment of the present invention.

The process in which the preemptive mixed reality experience system 10 determines the location or behavior of the mixed reality experiencer (step S21) and determines whether the location or action of the mixed reality experiencer meets the event occurrence condition (step S23) is shown in FIG. as described in

As a result of the determination, when the event occurrence condition is satisfied (step S23), the preemptive mixed reality experience system 10 selects preset augmented reality content corresponding to the event (step S25).

Next, the preemptive mixed reality experience system 10 determines whether the number of preemption for the selected augmented reality content exceeds a preset value (step S27). Step S29), the augmented reality content of the next sequence according to a preset algorithm is matched to the corresponding event (step S31).

Thereafter, the preemptive mixed reality experience system 10 extracts the augmented reality content matched to the event (step S33) and controls it to be displayed on the HMD 40, etc. of the experiencer (step S35), in which case the final extracted augmented reality content is displayed (step S35). Updating the preemption count of real content is the same as described above.

Meanwhile, it goes without saying that the process of performing each of the above-described embodiments may be performed by a program or application stored in a predetermined recording medium (eg, computer-readable). Here, the recording medium includes an electronic recording medium such as a random access memory (RAM), a magnetic recording medium such as a hard disk, and an optical recording medium such as a compact disk (CD).

In this case, the program stored in the recording medium may be executed on hardware such as a computer or smart phone to perform each of the above-described embodiments. In particular, at least one of the functional blocks of the preemptive mixed reality experience system according to the present invention described above may be implemented by such a program or application.

In addition, the present invention is not limited to the specific embodiments described above, but can be practiced with various modifications and modifications within the scope without departing from the gist of the present invention. It will be apparent that such modifications and variations are included in the present invention provided they fall within the scope of the appended claims.

10: mixed reality experience system 20: access point
30: surveillance camera 40: HMD
11: judgment unit 12: content extraction unit
13: display control unit 14: storage unit
15: confirmation unit 16: extraction unit

Claims (12)

In the control method performed by the preemptive mixed reality experience system,
(a) determining the location or behavior of the mixed reality experiencer in the preset area;
(b) extracting virtual content corresponding to the location or behavior of the mixed reality experiencer determined in step (a);
(c) controlling the display device of the mixed reality experiencer to display the virtual content extracted in step (b),
In step (b), a specific virtual content is extracted by additionally considering the number of preemptions for each virtual content,
In step (c), whenever the virtual content extracted in step (b) is controlled to be displayed, the number of preemptions for the virtual content is accumulated and stored,
The step (b) includes: (b1) checking the number of preemptions for virtual content that primarily corresponds to the location or behavior of the mixed reality experiencer determined in step (a); (b2) when the number of preemption checked in step (b1) exceeds a preset value, extracting virtual content secondarily corresponding to the location or behavior of the mixed reality experiencer Control method of type mixed reality experience system. delete delete According to claim 1,
In step (b2), if the number of preemptions checked in step (b1) exceeds a preset value, the corresponding virtual content is deactivated, and other virtual content according to the preset order is added to the mixed reality experiencer’s A control method of a preemptive mixed reality experience system, characterized in that it is set and extracted as virtual content corresponding to a location or action. According to claim 1,
The control method of the preemptive mixed reality experience system, characterized in that the virtual content is at least one of augmented reality content and virtual reality content. A computer-readable recording medium recording a program for executing the method of any one of claims 1, 4 to 5. An application program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 5. a determination unit for determining the location or behavior of the mixed reality experiencer within the preset area;
a content extraction unit for extracting virtual contents corresponding to the location or behavior of the mixed reality experiencer determined by the determination unit;
and a display control unit for controlling the display device of the mixed reality experiencer to display the virtual content extracted by the content extraction unit,
The content extraction unit extracts specific virtual content by additionally considering the number of preemptions for each virtual content,
Each time the display control unit controls to display the virtual content extracted by the content extraction unit, the display control unit accumulates and stores the number of preemptions for the virtual content,
The content extraction unit may include: a confirmation unit for confirming the number of preemptions for virtual content that primarily corresponds to the location or behavior of the mixed reality experiencer determined by the determination unit; and an extraction unit for extracting virtual content secondary to the location or behavior of the mixed reality experiencer when the number of preemption times confirmed by the confirmation unit exceeds a preset value. delete delete 9. The method of claim 8,
The extraction unit of the content extraction unit inactivates the corresponding virtual content when the number of preemptions checked by the confirmation unit exceeds a preset value, and adds other virtual content according to the preset order to the location or behavior of the mixed reality experiencer. A preemptive mixed reality experience system, characterized in that it is set and extracted as corresponding virtual content. 9. The method of claim 8,
The preemptive mixed reality experience system, characterized in that the virtual content is at least one of augmented reality content and virtual reality content.

Images