KR20210133344A - Apparatus for controlling mixed reality attraction and method thereof - Google Patents

Abstract

The present invention relates to an apparatus for controlling a mixed reality experience device, which provides a user located in a space separated from a predefined target space with a mixed reality image, which is generated on the basis of surrounding image information of a moving object acquired by an image sensor provided in the moving object when the moving object moves in the target space, on the basis of a communication structure among a central control unit, a mediator unit, and an agent unit. According to the present invention, the central control unit generates a mixed reality image by matching the surrounding image information of a moving object with stored mixed reality content and transfers the generated mixed reality image to the mediator unit along with the motion information of the moving object. The mediator unit controls motion of one or more motion simulators provided to enable a user to ride thereon on the basis of the motion information of the moving object received from the central control unit and displays the mixed reality image received from the central control unit in synchronization with the motion of each motion simulator through a display device possessed by the user riding in each motion simulator. The agent unit updates the mixed reality image displayed through the display device on the basis of an interaction with the user riding in each motion simulator.

Description

Apparatus and method for controlling a mixed reality experience apparatus

The present invention relates to an apparatus and method for controlling a mixed reality experience apparatus for controlling a mixed reality experience apparatus that provides a mixed reality image to a user riding a motion simulator.

Virtual reality (VR) refers to a technology that enables a user to experience real interaction with a virtual surrounding environment by producing a specific virtual environment or situation as three-dimensional content based on a computer. (AR: Augmented Reality) refers to a mixture of a real image and a virtual image by inserting an image or object such as computer graphics into the real surrounding environment.

Recently, by combining virtual reality and augmented reality, mixed reality (MR) provides the advantages of augmented reality in that it can interact with the real surrounding environment and the advantage of virtual reality in that it can convey a realistic immersion feeling to virtual objects. : Mixed Reality) is attracting attention, and this mixed reality is making a leap forward in a wide range of fields such as games, movies, performances, exhibitions, education and tourism.

On the other hand, an attraction is a play equipment installed in a theme park, etc., and provides a pleasure to the user by providing various combinations of rotational movement, tilting movement, constant velocity movement, acceleration movement, horizontal movement, and vertical movement while the user is on board. In recent years, by applying the above-mentioned mixed reality to an attraction, the user can experience an immersive mixed reality by driving the device the user rides in a mechanical way and simultaneously providing the user with an image according to a predetermined scenario. A mixed reality experience mechanism (or mixed reality attraction) is developing.

Republic of Korea Patent Publication No. 10-2018-0060451 (title of the invention: virtual space experience device using a miniature, published on June 7, 2018, hereinafter referred to as the prior art document) discloses that an image taken through a vehicle running in miniature land uses an HMD. Disclosed is a virtual space experience device displayed through However, in the above prior art document, there is no specific operation and control topology for controlling the virtual space experience apparatus, such as a communication structure applied to the virtual space experience apparatus and a method of displaying an image.

The apparatus and method for controlling a mixed reality experience apparatus according to the present invention are communication that can be applied to a mixed reality experience apparatus that provides a mixed reality image according to a predetermined scenario to a user at the same time as driving the apparatus boarded by the user in a mechanical manner An object of the present invention is to provide a control topology in which a mixed reality experience mechanism is controlled based on the structure and the communication structure.

A control apparatus for a mixed reality experience apparatus according to an aspect of the present invention recognizes an image provided in a moving object in a process of moving the moving object in a predefined target space based on a communication structure between a central control unit, a mediator unit, and an agent unit A device for controlling a mixed reality experience apparatus that provides a mixed reality image generated based on image information around the moving object acquired by a sensor to a user located in a space separated from the target space, wherein the central control unit comprises: The mixed reality image is generated by matching the surrounding image information of the moving object with the pre-stored mixed reality content, and the generated mixed reality image is transmitted to the mediator unit together with the motion information of the moving object, and the mediator unit is provided from the central control unit. Controls the motion of one or more motion simulators provided for the user's boarding based on the received motion information of the moving object, and synchronizes the mixed reality image received from the central control unit with the motion of each motion simulator. It is displayed through a display device possessed by a user who rides the , and the agent unit updates the mixed reality image displayed through the display device based on an interaction with the user riding each of the motion simulators. .

In the present invention, the central control unit performs a master function of integrated monitoring and integrated control for each operation of the moving object, the motion simulator, the display device, the mediator unit, and the agent unit, and supports the master function. It is characterized in that it includes an integrated motion database, an integrated image database, and an integrated content database as an integrated database.

In the present invention, the integrated contents database stores the mixed reality contents, and the central control unit extracts the mixed reality contents matching the currently acquired surrounding image information of the moving object from the integrated contents database, and the extracted contents are extracted from the integrated contents database. and generating the mixed reality image by matching the mixed reality content with the currently acquired surrounding image information of the moving object.

In the present invention, the image recognition sensor provided in the moving object includes one or more cameras, and the central control unit uses the image information surrounding the moving object acquired by the image recognition sensor as monocular image information or binocular image information as the integrated image. It is characterized in that it is stored in the database.

In the present invention, when an abnormal motion of the moving object is detected based on at least one of image information and motion information of the surrounding of the moving object, the central control unit stops each operation of the moving object and the motion simulator, and It is characterized in that at least one of image information and motion information is stored in the integrated motion database.

In the present invention, the mediator unit synchronizes and controls the motions of the respective motion simulators based on the motion information of the moving object received from the central control unit.

In the present invention, the motion information of the moving object is multi-axis motion information in which the translational and rotational motions of the moving object are reflected, and the one or more motion simulators include at least two heterogeneous motion simulators each having a different axis configuration, and the mediator unit , selects motion information corresponding to an axis configuration of the heterogeneous motion simulator from among the multi-axis motion information, and controls the motion of the heterogeneous motion simulator using the selected motion information.

In the present invention, when controlling the motion of the heterogeneous motion simulator using the selected motion information, the mediator unit synchronizes and controls the motion control timing of the heterogeneous motion simulator.

In the present invention, further comprising an action sensor sensing an action of a user riding on the motion simulator, wherein the agent unit is a local content database storing mixed reality content corresponding to the user's action sensed by the action sensor It is characterized in that it includes.

In the present invention, the agent unit extracts the mixed reality content corresponding to the user's action sensed by the action sensor from the local content database, and displays the mixed reality image currently displayed through the display device from the local content database. It is characterized by reflecting the extracted mixed reality content.

A method for controlling a mixed reality experience apparatus according to an aspect of the present invention is based on a communication structure between a central control unit, a mediator unit, and an agent unit, and an image recognition sensor provided in the moving object in the process of moving the moving object in a predefined target space. A method of controlling a mixed reality experience mechanism that provides a mixed reality image generated based on the surrounding image information of the moving object obtained by generating the mixed reality image by matching surrounding image information with pre-stored mixed reality content, and transmitting the generated mixed reality image together with motion information of the moving object to the mediator unit; The mediator unit controls the motion of one or more motion simulators provided for the user's boarding based on the motion information of the moving object received from the central control unit, and transmits the mixed reality image received from the central control unit to each of the motion simulators. displaying through a display device possessed by a user riding in each of the motion simulators in synchronization with the motion; and updating, by the agent unit, the mixed reality image displayed through the display device based on the interaction with the user riding on each of the motion simulators.

According to one aspect of the present invention, the present invention provides a triple communication structure consisting of a central control unit, a mediator unit, and an agent unit, and provides an integrated database for integrated monitoring and control of a mixed reality experience apparatus in the central control unit, and the mediator By providing a local database for real-time monitoring and image processing of the mixed reality experience apparatus in the department and the agent department, it is possible to improve the processing and management efficiency of a large amount of information generated in real time for the operation of the mixed reality experience apparatus, Based on such a triple communication structure, the control efficiency can be improved by dividing the control of the mixed reality experience device into the central control unit, the mediator unit, and the agent unit.

1 is an exemplary diagram schematically illustrating a mixed reality experience apparatus to which a control apparatus for a mixed reality experience apparatus according to an embodiment of the present invention is applied.
2 is a block diagram illustrating a control apparatus for a mixed reality experience apparatus according to an embodiment of the present invention.
3 is a block diagram illustrating an organic operation between a central control unit, a mediator unit, and an agent unit including a control device for a mixed reality experience apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating a method of controlling a mixed reality experience apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of an apparatus and method for controlling a mixed reality experience apparatus according to the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is an exemplary diagram for generally explaining a mixed reality experience apparatus to which a control device of a mixed reality experience apparatus according to an embodiment of the present invention is applied, and FIG. 2 is a mixed reality experience apparatus according to an embodiment of the present invention. is a block diagram for explaining the control device of the , and FIG. 3 is a block for explaining the organic operation between the central control unit, the mediator unit, and the agent unit including the control device of the mixed reality experience apparatus according to an embodiment of the present invention It is a configuration diagram, and FIG. 4 is a flowchart for explaining a method of controlling a mixed reality experience apparatus according to an embodiment of the present invention.

First, a mixed reality experience mechanism to which this embodiment is applied will be generally described with reference to FIG. 1 .

The mixed reality experience apparatus of this embodiment is generated based on the surrounding image information of the moving object VEH acquired by the image recognition sensor provided in the moving object VEH while the moving object VEH moves in a predefined target space. It may correspond to a mixed reality attraction installed in a theme park, etc. that provides a mixed reality image to a user located in a space separated from the target space.

Specifically, as shown in Fig. 1 (b), the above-mentioned movable body (VEH) may be a miniature type of mobile miniature model vehicle, and may be implemented as a miniature model vehicle of various types such as automobiles, railroad vehicles, ships, or gondolas. Yes (FIG. 1(b) shows an example in which the moving body VEH is implemented as a railroad vehicle). The moving object VEH may be provided with an image recognition sensor for acquiring surrounding image information, and the image recognition sensor includes at least one of a monocular camera and a binocular camera, so that the moving object VEH moves around the image information. (eg, a front image, a rear image, a side image, or a surround view image) can be acquired. The surrounding image information of the moving object VEH acquired by the image recognition sensor may be utilized in the process of generating a mixed reality image and detecting abnormal motion of the moving object VEH as described later, and also monocular image information or As binocular image information, it may be stored in the integrated image database 140 provided in the central controller 100 .

In addition, the moving object VEH may be provided with a motion sensor for obtaining motion information of the moving object VEH, and the motion information of the moving object VEH obtained by the motion sensor may include translational motion and rotation of the moving object VEH. It may be multi-axis motion information in which motion is reflected, for example, 6 degrees of freedom (DOF) motion information. As a motion sensor, various sensors such as an acceleration/gyro sensor, a depth sensor, a magnetic field sensor, and a camera sensor can be employed to sense 6 DOF motion information. As such, it can be utilized in the process of controlling the motion of the motion simulator MS and detecting the abnormal motion of the moving object VEH.

A time stamp reflecting the time each information was acquired may be added to the surrounding image information and motion information respectively acquired through the image recognition sensor and motion sensor provided in the moving body (VEH), and added to each information This time stamp may be utilized for synchronization between the mixed reality image and the motion of the motion simulator (MS).

As shown in FIG. 1( a ), the target space to which the mobile body VEH moves may be a miniature model village in the form of a miniature model. Miniature sculptures located nearby may be provided. In the example of FIG. 1 in which the moving body VEH is implemented as a railroad vehicle, the moving body VEH may secure power by receiving its operating power through a miniature rail, but is not limited thereto. For example, the moving body VEH. In an example in which is implemented as a gondola, it may be implemented as an embodiment in which it is hung on a cable that moves by a separate power and is passively moved according to the movement of the cable.

In the process of moving the moving object VEH in the target space, image information around the moving object VEH is obtained by an image recognition sensor provided in the moving object VEH, and the central control unit 100 to be described later based on the obtained surrounding image information. ) and the agent unit 300, a mixed reality image is generated, and the generated mixed reality image can be provided to a user riding in a motion simulator (MS) located in a space separated from the target space through a display device (DISPLAY). have. The motion simulator (MS) is provided with a display device (DISPLAY, for example, a Head Mounted Display (HMD)), and provides translational and rotational motions to correspond to the mixed reality image provided through the display device (DISPLAY) worn by the user. By doing so, it refers to a mechanical device that provides a feeling of experience as if the user actually boarded the moving body VEH. A plurality of motion simulators (MS) may be provided for each user so that a plurality of users can ride in a position spaced apart from the above-described target space, and as will be described later, a plurality of motion simulators (MS) each have a different axis configuration. It may correspond to a heterogeneous motion simulator.

The mixed reality experience mechanism as described above is controlled by the control device of this embodiment based on the communication structure between the central control unit 100, the mediator unit 200, and the agent unit 300 shown in FIG. Based on this, the operation of the control device of the mixed reality experience apparatus according to the present embodiment will be described in detail with reference to FIG. 3 .

The central control unit 100 generates a mixed reality image by matching the surrounding image information of the moving object VEH with pre-stored mixed reality content, and the mediator unit 200 uses the generated mixed reality image together with the motion information of the moving object VEH. can be passed to The central control unit 100 performs integrated monitoring (eg, abnormal operation monitoring) and integration for each operation of the moving object (VEH), the motion simulator (MS), the display device (DISPLAY), the mediator unit 200 and the agent unit 300 . It can be implemented as a control server that performs a master function of control (eg, power on/off control). As shown in FIG. 3 , the central control unit 100 may include an integrated motion database 130 , an integrated image database 140 and an integrated content database 150 as an integrated database for supporting the master function, In addition, the first image processing module 120 for generating a mixed reality image, the integrated databases 130 , 140 , 150 , and the first image processing module ( 120 ) and a first controller 110 performing a function to mediate with the mediator unit 200 . Although the first image processing module 120 and the first controller 110 are illustrated in separate configurations in FIG. 3 , they may be implemented as an integrated configuration within the central control unit 100 according to an exemplary embodiment. Hereinafter, an embodiment in which the first image processing module 120 and the first controller 110 are separately implemented according to the example of FIG. 3 will be described.

If the operation of the central control unit 100 for generating the mixed reality image is described in detail as a sub-configuration thereof, the first controller 110 receives the surrounding image information of the moving object VEH from the moving object VEH in real time and receives the first transmitted to the image processing module 120, and the first image processing module 120 includes VR content and AR resources as mixed reality content (3D object model information as 3D object model information) matching the currently acquired surrounding image information of the moving object VEH. ) from the integrated content database 150 , and the extracted mixed reality content may be matched with the currently acquired surrounding image information of the moving object VEH to generate a mixed reality image.

Specifically, the overall scenario of the mixed reality image provided to the user through the display device DISPLAY of the motion simulator MS from the point in time when the movement of the moving object VEH is started to the point in time when the movement is completed may be predefined, According to the predefined scenario as described above, the mixed reality content matched to the surrounding image information of the moving object VEH is matched by the surrounding image information of the moving object VEH at a specific point in time or at a specific location and stored in the integrated content database 150 . It may be pre-saved. Accordingly, the first image processing module 120 extracts, from the integrated content database 150 , the mixed reality content that is matched with the currently acquired surrounding image information of the moving object VEH and is stored (ie, VR content and AR resource). A mixed reality image can be generated by selectively extracting one or more of them according to the currently acquired surrounding image information of the moving object VEH) and matching it with the surrounding image information of the moving object VEH through real-time stitching. .

The first image processing module 120 may store peripheral image information of the moving object VEH acquired by an image recognition sensor provided in the moving object VEH as monocular image information or binocular image information in the integrated image database 140, , in an embodiment in which both the monocular image information and the binocular image information are obtained, the monocular image information and the binocular image information may be separated and stored in the integrated image database 140 . The information stored in the integrated image database 140 may be utilized for maintenance or debugging work by a designer later.

Meanwhile, the first controller 110 may receive motion information of the moving object VEH together with real-time surrounding image information of the moving object VEH from the moving object VEH, and the received peripheral image information and motion of the moving object VEH. An abnormal operation of the moving object VEH may be detected based on one or more of the information.

Specifically, since the moving object VEH moves along a miniature rail that is fixed in the target space, surrounding image information acquired through an image recognition sensor provided in the moving object VEH is specific depending on the location of the moving object VEH. image is obtained, and therefore, the first controller 110 determines whether the currently acquired image information of the surrounding area of the moving object VEH is different from the image expected to be obtained in the course of moving the moving object VEH. It can be determined that an abnormal operation such as In this case, image information stored in the integrated image database 140 may be used as reference image information to be compared with surrounding image information of the moving object VEH in order to detect an abnormal operation of the moving object VEH.

In addition, as mentioned above, since the moving body VEH moves along a miniature rail that is fixed in the target space, motion information (ie, multi-axis motion information) obtained through a motion sensor provided in the moving body VEH is It exists within a predefined reference range according to the shape and structure of the miniature rail. Therefore, when the currently acquired motion information of the moving object VEH is out of the aforementioned reference range, the moving object VEH derails. Alternatively, it may be determined that an abnormal operation such as overturning has occurred.

Accordingly, the first controller 110 may detect an abnormal operation of the moving object VEH by using any one of the surrounding image information and the motion information of the moving object VEH, or by using both information, and the moving object VEH. ) is detected, the operation of the moving object VEH and the motion simulator MS is stopped, and one or more of the surrounding image information and motion information of the moving object VEH (that is, the basis for detecting the abnormal operation of the moving object VEH) information) may be stored in the integrated motion database 130 . The information stored in the integrated motion database 130 may be utilized for maintenance or debugging work by a designer later.

In addition to the abnormal operation of the moving object VEH, the first controller 110 detects an abnormal operation of the motion simulator MS, such as a synchronization error between the motion of the motion simulator MS and a mixed reality image, a motion synchronization error between heterogeneous motion simulators, And a malfunction of the motion simulator MS may be detected, and when an abnormal operation of the motion simulator MS is detected, the motion information may be stored in the integrated motion database 130 .

Next, the mediator unit 200 controls the motion of one or more motion simulators (MS) provided for the user's boarding based on the motion information of the moving body (VEH) received from the central control unit 100, and the central control unit ( 100) is synchronized with the motion of each motion simulator (MS) and displayed through a display device (DISPLAY) possessed by a user riding on each motion simulator (MS) (including the concept of wearing). can The mediator unit 200 may be implemented as a computing device (eg, a processor) that is integrated and provided inside the motion simulator MS, and stores the motion information of the moving object VEH received from the central control unit 100 by itself. It may include a local motion database 220 and a second controller 210 for motion control of the motion simulator (MS).

When the operation of the mediator unit 200 is described in detail with its sub-configuration, the second controller 210 is a display device (DISPLAY) in which the user possesses (wears) the mixed reality image received from the central control unit 100 (eg, : HMD), and at the same time control the motion of the motion simulator MS based on the motion information of the moving object VEH received from the central control unit 100, and at this time, the mixture displayed through the display device DISPLAY It is possible to synchronize the motion of the real image and the motion simulator (MS). In this case, as described above, the time stamp added to the mixed reality image (that is, the time stamp added to the surrounding image information based on the generation of the mixed reality image) and the time stamp added to the motion information are synchronized during the synchronization process. For example, the second controller 210 synchronizes the display time of the mixed reality image having the time stamp of the same time and the motion control time of the motion simulator (MS) to the mixed reality image and the motion simulator. It is possible to synchronize the motion of (MS).

Meanwhile, the second controller 210 may synchronize and control the motion of each motion simulator MS based on the motion information of the moving object VEH received from the central controller 100 .

Specifically, one or more motion simulators MS to which this embodiment is applied may include at least two heterogeneous motion simulators each having a different axis configuration, for example, a 6-axis motion simulator having 6 degrees of freedom (6 DOF). And, it may include a 3-axis motion simulator with 3 degrees of freedom (3 DOF). That is, the present embodiment assumes that a plurality of types of motion simulators (MS) having different axis configurations are employed in order to secure a realistic degree of design freedom for the motion simulator (MS) in the process of designing a mixed reality experience mechanism. Accordingly, the second controller 210 synchronizes the motions of a plurality of types of motion simulators MS based on the motion information of the moving body VEH (that is, multi-axis motion information in which the translational and rotational motions of the moving body VEH are reflected). can be controlled

In this case, the second controller 210 may select motion information corresponding to an axis configuration of the heterogeneous motion simulator from among the multi-axis motion information, and control the motion of the corresponding heterogeneous motion simulator using the selected motion information.

Specifically, as described above, the multi-axis motion information of the moving body VEH may be 6 DOF motion information of the moving body VEH, and accordingly, the second controller 210 controls the motion of the 6-axis motion simulator among heterogeneous motion simulators. Motion of 6-axis motion simulator can be controlled by using all 6 DOF motion information (Heterogeneous motion simulator motion standardization). The motion control of each of the heterogeneous motion simulators must be synchronized with the mixed reality image displayed through the display device (DISPLAY), and accordingly, the motion control timing of each of the heterogeneous motion simulators must also be synchronized. It is possible to synchronize and control the motion control timings of heterogeneous motion simulators through a method of generating a motion synchronization point for synchronization of a start time and an end time for the motion control of .

In the process in which the mixed reality image is displayed by the second controller 210 of the mediator unit 200 through the display device DISPLAY, the agent unit 300 interacts with a user who rides each motion simulator MS. Based on the , the mixed reality image displayed through the display device DISPLAY may be updated. The agent unit 300 may be implemented as a computing device (eg, PC, etc.) capable of wired/wireless communication with the mediator unit 200 and the display device (DISPLAY), and mixed reality content (VR content) required for updating the mixed reality image. and a local content database 320 storing AR resources and the mixed reality content stored in the aforementioned integrated content database 150), and a second image processing module ( 310) may be included.

When the mixed reality image is updated based on the interaction with the user, it means that the mixed reality content corresponding to the user's action is reflected in the mixed reality image. Various actions, such as a motion, a user's gesture, and a user's manipulation of a separate manipulation device, may be included. The user's action as described above may be obtained through a separately provided action sensor (eg, a pupil sensor for detecting a field of view, a haptic sensor for detecting a haptic, a gesture sensor for detecting a gesture, a separately provided mechanical manipulation device, etc.), The action sensor may be installed in various locations within a range capable of sensing a user's action, such as a display device (DISPLAY) (ie, HMD) or a motion simulator (MS).

Accordingly, the second image processing module 310 extracts the mixed reality content corresponding to the user's action sensed by the action sensor from the local content database 320, and the mixed reality currently displayed through the display device DISPLAY. The mixed reality image may be updated by reflecting the mixed reality content extracted from the local content database 320 in the image.

The mixed reality content applied to the update of the mixed reality image may be stored in advance in the local content database 320 in correspondence with each user's action. Alternatively, mixed reality content corresponding to each case, such as when a gesture is made or when a specific manipulation is applied to a separate manipulation device, may be pre-stored in the local content database 320 . Accordingly, the second image processing module 310 extracts the mixed reality content stored in response to the current user's action from the local content database 320 (that is, one or more of the VR content and the AR resource as the user's action (selectively extracted according to ) and reflected on the mixed reality image currently displayed through the display device (DISPLAY), interaction between the user and the call mixed reality image can be ensured. Meanwhile, the process of selecting the mixed reality image to be reflected according to the user's action may be performed by a logic module implemented separately from the second image processing module 310 .

According to the present embodiment described above, the central control unit 100, which can be applied to a mixed reality experience device that provides a mixed reality image according to a predetermined scenario to the user at the same time as driving the device the user rides in a mechanical way , a triple communication structure composed of the mediator unit 200 and the agent unit 300, and a control topology in which a mixed reality experience mechanism is controlled based on the communication structure may be provided. Furthermore, an integrated database is provided in the central control unit 100 for integrated monitoring and control of the mixed reality experience apparatus, and the mediator unit 200 and the agent unit 300 are local for real-time monitoring and image processing of the mixed reality experience apparatus. By providing a database, it is possible to improve the processing and management efficiency of large amounts of information generated in real time for the operation of the mixed reality experience apparatus, and based on such a triple communication structure, the central control unit to control the mixed reality experience apparatus (100), it is possible to improve the control efficiency by dividing the mediator unit 200 and the agent unit 300.

4 is a flowchart illustrating a method of controlling a mixed reality experience apparatus according to an embodiment of the present invention.

A method of controlling a mixed reality experience apparatus according to an embodiment of the present invention will be described with reference to FIG. 4 . First, the central control unit 100 matches the surrounding image information of the moving object VEH with pre-stored mixed reality contents to create mixed reality. An image is generated, and the generated mixed reality image is transmitted to the mediator unit 200 together with motion information of the moving object VEH (S100).

In step S100 , the central control unit 100 extracts the mixed reality content matching the currently acquired surrounding image information of the moving object VEH from the integrated content database 150 , and extracts the extracted mixed reality content from the currently obtained mobile object VEH. ) to generate a mixed reality image by matching it with the surrounding image information.

In addition, in step S100, the central control unit 100 stores peripheral image information of the moving object VEH obtained by the image recognition sensor as monocular image information or binocular image information in the integrated image database 140, and When an abnormal operation of the moving object VEH is detected based on at least one of the surrounding image information and the motion information, each operation of the moving object VEH and the motion simulator MS is stopped, and the surrounding image information and motion of the moving object VEH One or more of the information is stored in the integrated motion database 130 .

After step S100 , the mediator unit 200 controls the motion of one or more motion simulators (MS) provided for the user's boarding based on the motion information of the moving object VEH received from the central control unit 100, and the central control unit The mixed reality image received from 100 is synchronized with the motion of each motion simulator MS and displayed through a display device DISPLAY possessed by a user riding on each motion simulator MS (S200).

In step S200, the mediator unit 200 synchronizes and controls the motion of each motion simulator MS based on the motion information of the moving object VEH received from the central control unit 100, and specifically, the movement of the moving object VEH. Among the multi-axis motion information, motion information corresponding to an axis configuration of the heterogeneous motion simulator is selected, and the motion of the heterogeneous motion simulator is controlled using the selected motion information, but the motion control timing is synchronized and controlled.

After the step S200, the agent unit 300 updates the mixed reality image displayed through the display device DISPLAY based on the interaction with the user riding on each motion simulator MS (S300).

In step S300, the agent unit 300 extracts the mixed reality content corresponding to the user's action sensed by the action sensor from the local content database 320, and adds it to the mixed reality image currently displayed through the display device (DISPLAY). It reflects the mixed reality content extracted from the local content database 320 .

Implementations described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the discussed features may also be implemented in other forms (eg, in an apparatus or a program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, or the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants ("PDAs") and other devices that facilitate communication of information between end-users.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and it is understood that various modifications and equivalent other embodiments are possible by those skilled in the art. will understand Accordingly, the true technical protection scope of the present invention should be defined by the following claims.

VEH: mobile
MS: Motion Simulator
DISPLAY: display device
100: central control
110: first controller
120: first image processing module
130: integrated motion database
140: integrated image database
150: Unified Content Database
200: mediator unit
210: second controller
220: local motion database
300: agent unit
310: second image processing module
320: local content database

Claims (11)

Based on the communication structure between the central control unit, the mediator unit, and the agent unit, in the process of moving the moving object in a predefined target space, it is generated based on the surrounding image information of the moving object acquired by the image recognition sensor provided in the moving object A device for controlling a mixed reality experience device that provides a mixed reality image to a user located in a space separated from the target space,
The central control unit generates the mixed reality image by matching the surrounding image information of the moving object with pre-stored mixed reality content, and transmits the generated mixed reality image together with the motion information of the moving object to the mediator unit,
The mediator unit controls the motion of one or more motion simulators provided for the user's boarding based on the motion information of the moving object received from the central control unit, and transmits the mixed reality image received from the central control unit to each of the motion simulators. Synchronized with the motion and displayed through a display device possessed by a user riding on each of the motion simulators,
and the agent unit updates the mixed reality image displayed through the display device based on an interaction with the user riding in each motion simulator.
According to claim 1,
The central control unit performs a master function of integrated monitoring and integrated control for each operation of the moving object, the motion simulator, the display device, the mediator unit, and the agent unit, and is integrated as an integrated database to support the master function. A control apparatus for a mixed reality experience apparatus comprising a motion database, an integrated image database, and an integrated content database.
3. The method of claim 2,
The integrated content database stores the mixed reality content,
The central control unit extracts, from the integrated content database, the mixed reality content matching the currently acquired surrounding image information of the moving object, and matches the extracted mixed reality content with the currently acquired surrounding image information of the moving object, so that the A control device for a mixed reality experience apparatus, characterized in that it generates a mixed reality image.
3. The method of claim 2,
The image recognition sensor provided in the moving body includes one or more cameras,
and the central control unit stores, as monocular image information or binocular image information, surrounding image information of the moving object acquired by the image recognition sensor in the integrated image database, the mixed reality experience apparatus control device
3. The method of claim 2,
When an abnormal motion of the moving object is detected based on at least one of peripheral image information and motion information of the moving object, the central controller is configured to stop each operation of the moving object and the motion simulator, and to stop each operation of the moving object and the surrounding image information and motion of the moving object and storing at least one of the information in the integrated motion database.
According to claim 1,
and the mediator unit synchronizes and controls the motions of the respective motion simulators based on the motion information of the moving object received from the central control unit.
7. The method of claim 6,
The motion information of the movable body is multi-axis motion information in which translational and rotational motions of the movable body are reflected, and the one or more motion simulators include at least two heterogeneous motion simulators each having a different axis configuration,
The mediator unit selects motion information corresponding to the axis configuration of the heterogeneous motion simulator from among the multi-axis motion information, and controls the motion of the heterogeneous motion simulator using the selected motion information. controller.
8. The method of claim 7,
The mediator unit, when controlling the motion of the heterogeneous motion simulator using the selected motion information, synchronizes and controls the motion control timing of the heterogeneous motion simulator.
According to claim 1,
Including an action sensor for sensing an action of a user riding on the motion simulator,
and the agent unit includes a local content database storing mixed reality content corresponding to the user's action sensed by the action sensor.
10. The method of claim 9,
The agent unit extracts, from the local content database, mixed reality content corresponding to the user's action sensed by the action sensor, and displays the mixed reality image currently displayed through the display device in the mixed reality extracted from the local content database. A control device for a mixed reality experience apparatus, characterized in that it reflects content.
Based on the communication structure between the central control unit, the mediator unit, and the agent unit, in the process of moving the moving object in a predefined target space, it is generated based on the surrounding image information of the moving object acquired by the image recognition sensor provided in the moving object A method of controlling a mixed reality experience device that provides a mixed reality image to a user located in a space separated from the target space, the method comprising:
generating, by a central controller, the mixed reality image by matching the surrounding image information of the moving object with pre-stored mixed reality content, and transmitting the generated mixed reality image to the mediator unit together with the motion information of the moving object;
The mediator unit controls the motion of one or more motion simulators provided for the user's boarding based on the motion information of the moving object received from the central control unit, and transmits the mixed reality image received from the central control unit to each of the motion simulators. displaying through a display device possessed by a user riding in each of the motion simulators in synchronization with the motion; and
updating, by the agent, the mixed reality image displayed through the display device based on the interaction with the user riding in each of the motion simulators;
A method of controlling a mixed reality experience apparatus, comprising:

Images