RU2783486C1 - Mobile multimedia complex - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: mobile system for creation of a virtual space contains: a block of motion capture cameras, consisting of at least one motion capture camera, wherein the block of motion capture cameras is made with the possibility of reception of video data; a sensor system block consisting of at least one sensor system, receiving sensor data for creation of a sensor space; a projection system block consisting of at least two projectors projecting a virtual space on a surface; an acoustic system consisting of at least four speakers generating a volumetric sound; a control system, which is a core of the system, and which interconnects all components of the system for creation of a virtual space; a mobile dismountable structure, which is a bearing element for attachment of elements of the system for creation of a virtual space, wherein the mobile dismountable structure is made with the possibility of attachment of surfaces for projection of a virtual space.

EFFECT: increase in mobility of a system for creation of a virtual space.

3 cl, 4 dwg

Description

FIELD OF TECHNOLOGY

The present technical solution relates to the field of computer technology, in particular, to mobile systems for creating a virtual space.

BACKGROUND OF THE INVENTION

The prior art solution, selected as the closest analogue, US 2011205341 (A1), publ. 08/25/2011. This solution refers to architectures that combine several depth cameras and several projectors to cover a certain space, which allows you to develop multidimensional (for example, 3D) models of objects in space, and also makes it possible to control graphics on the same objects. The architecture combines depth data from all depth cameras as well as color information into a single multi-dimensional model in combination with calibrated projectors.

The proposed technical solution is aimed at eliminating the shortcomings of the state of the art and differs from those previously known in that the proposed solution provides a mobile system for creating a virtual space.

SUMMARY OF THE INVENTION

The technical problem to be solved by the claimed solution is the creation of a mobile system for creating a virtual space.

The technical result is to increase the mobility of the system to create a virtual space.

An additional technical result is the simplification of the system design for creating a virtual space, since when interacting with the virtual space of the system, users do not need wearable devices (for example, VR helmets, gloves, position sensors, etc.).

The claimed technical results are achieved through the implementation of a mobile system for creating a virtual space, which contains:

a motion capture camera unit 101 comprising at least one motion capture camera, the motion capture camera unit being configured to receive video data 111;

a sensor systems unit 102 consisting of at least one sensor system receiving sensor data 112 to create a sensory space;

block projection systems 103, consisting of at least two projectors projecting a virtual space on the surface 121;

speaker system 104, consisting of at least four speakers generating surround sound 122;

a control system 105, which is the core of the system and interconnects all components of the system to create a virtual space;

a mobile collapsible structure 140, which is a supporting element for attaching elements of the system for creating a virtual space, and the mobile collapsible structure is configured to attach surfaces for projecting a virtual space.

In a particular embodiment of the described system, the mobile collapsible structure is implemented as a truss structure.

In another particular embodiment of the described system, the surface for projecting virtual space is a projection canvas.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

Fig. 1 illustrates a block diagram of a mobile system.

Fig. 2 illustrates the relationship between the mobile structure and the hardware of the complex.

Fig. 3 illustrates a block diagram of the processes that occur during interaction between one or more users and system elements.

Fig. 4 illustrates an example of a general layout of a computing device.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, one skilled in the art will appreciate how the present invention can be used, both with and without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to unnecessarily obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

This technical solution is a hardware complex that creates an interactive virtual space, due to which the user can be in the virtual space and interact with the elements generated by the system. Space simulation is provided by the projection of images onto physical surfaces and the spatial sound system. Interaction with the environment is provided through technology for determining the position of the user and parts of his body in space and/or through a sensory system.

System 100 in FIG. 1 is typical hardware for implementing the processes described below. The system contains the following technical elements.

Motion capture camera block 101, consisting of one or more cameras, receiving video data 111 (can be used: RGB cameras, infrared cameras, including RGB cameras with IR illumination and thermal imagers, depth sensors (can be either separate devices or built-in devices into an existing RGB or IR camera (eg Azure Kinect))). The motion capture cameras are preferably mounted on the upper frame of the mobile structure in such a way that the unit captures the maximum possible space from different angles.

Sensor systems block 102, consisting of one or more sensor systems, receiving sensor data 112 (to create a sensory space). The following two sensor systems can be used. A touch film that is glued to the surface and detects touch due to the difference in capacitance between the layers and a system of spaced IR sensors that create a laser grid between themselves and detect not a touch, but an object falling into space (for example, Surfancy). It should be noted that, in fact, any system that can capture and transmit touches in a 2-axis coordinate system can be used. Sensor systems are attached to the wall in pairs opposite each other, either on the left and right sides of the wall, or above and below. The second option is preferable, since in most cases the width of the wall is greater than the maximum allowable distance of the planks from each other.

Projection system unit 103, consisting of two or more projectors projecting a virtual space onto surfaces 121. Short throw and ultra short throw projectors can be used, which allow the image to be “revealed” at a small distance between the projector and the surface (for example, Optoma ZH406ST). The projectors are mounted on the upper frame of the structure in such a way that:

- the projector hung against the surface;

- each projector was at the maximum distance from the projected surface for better image disclosure;

- the images from the projectors did not overlap each other.

A speaker system 104 consisting of four or more speakers generating surround sound 122. Any speaker system capable of producing 4.0 surround sound and above can be used. The acoustic system is mounted on the upper or side frame of the structure in the upper corners of the complex.

The control system 105 is the core of the system and carries out the interconnection of the elements of the complex. The control system is a media server, the technical characteristics of which must ensure the stable operation of the systems of the complex, depending on its size and filling with equipment. The control system can communicate with additional interaction elements 131 such as phone, tablet, tact buttons, additional element control systems.

Communication between the elements of the system can be carried out both wired and wireless.

- The communication system between the projection system 103 and the control system 105 must transmit a video stream in 4K resolution (3840 x 2160 pixels) at a rate of at least 60 frames per second (HDMI version 2.0 or higher).

- The data transmission system between the motion capture camera block 101 and the control system 105 must transmit a video stream at a speed of at least 4 Gbps (USB standard version 3.0 or higher).

The control system 105 receives, processes, and transmits data between system elements through one or more connections. Examples of such networks include any type of wired and wireless systems. The control system 105 includes working memory, data and software storage, microprocessors for executing instructions, graphics processors, and other components for rendering and generating graphics, images, audio, and video files.

The control system contains a set of software, the components of which may include one or more functions:

- Generation of visual content for the simulation of virtual spaces and its transfer to the output in the block of projection systems 103;

- Generation and transmission of 104 audio signals to the speaker system, including, but not limited to: sound presets, music, voice recordings, midi signals;

- Processing of incoming signals from the sensor systems block 102, determination of touch points in the selected coordinate system and launch of interaction scenarios;

- Processing of incoming signals from the block of motion capture cameras 101, determining the position in space of one or more users, including the analysis of the position of parts of the skeleton and the subsequent launch of interaction scenarios;

- Processing and sending signals to additional interaction elements 131.

The described systems are physically mobile and interconnected with the mobile structure shown in FIG. 2. Mobile structure 201 creates a physical space within which a virtual space unfolds. The mobile structure 201 can be implemented as a truss structure (an engineering solution that allows the implementation of temporary structures of various geometry). It is a collapsible and reusable (mobile) structure, including the main load-bearing element to which are attached:

- a block of motion capture cameras 202;

- block of sensory systems 203;

- a block of projection systems 204;

- acoustic system 205;

- projection panels 206.

In this solution, two main variants of projection surfaces can be implemented.

1. A projection screen that is mounted on a hollow or solid frame.

2. Projection paint. It is applied to sheets of solid material that is attached to a mobile structure.

A typical process organization scheme is described in Fig. 3.

One or more users can simultaneously interact with the interactive space using the sensor systems unit 301 and the motion capture camera unit 311.

The sensor systems block sends data to the control system 302, which is processed by the control system, and which determines the points of interaction in the coordinate system of the touch surface and synchronizes them in the coordinate system of the virtual space 303.

The motion capture camera unit sends video data to the control system 312, in which the control system recognizes users, detects their skeleton and the position of elements in the coordinate system 313.

Based on the received and processed data, the control system makes changes to the generated audio and video streams according to the previously specified scenario 304 and sends these streams to the projection system unit and / or speaker system 305.

The process of calibrating the system to create a virtual space.

Images from projectors are calibrated in 2 steps.

1. The image from each projector is adjusted and leveled to the surface, taking into account the slope and curvature of the surface, as well as the distance between the projectors and the surface.

2. The reduction of the common space is carried out. Flattening allows you to synchronize a common portion of an image so that gaps between images are removed while overlapping the edges of the image in a way that is not visible to users.

Cameras capture the user from different angles and transmit the video stream to the control system. The control system selects the nodal skeletal points and writes them down in the coordinate system. Calibration allows you to create a single coordinate system for all cameras so that the position of the conditional reference point "right elbow" is the same for all.

Sensory system. Each set of slats of the sensory system produces touch points in a coordinate system with its own reference point. Calibration combines several two-axis coordinate systems into one and allows you to transfer uniform data to the control system.

On FIG. 4, the general scheme of the computing device (media server) (400) will be presented below, providing the data processing necessary for the implementation of the claimed solution.

In general, the device (400) contains components such as: one or more processors (401), at least one memory module (402), data storage (403), input/output interfaces (404), I/O (405), networking tools (406).

The media server additionally contains a video accelerator or a set of them with sufficient power to create (including real-time rendering) and transmit a video stream.

The processor (401) of the device performs the basic computing operations necessary for the operation of the device (400) or the functionality of one or more of its components. The processor (401) executes the necessary machine-readable instructions contained in the main memory (402).

The memory (402) is typically in the form of RAM and contains the necessary software logic to provide the required functionality.

The data storage means (403) can be in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The means (403) allows long-term storage of various types of information, for example, the above-mentioned files with user data sets, a database containing records of time intervals measured for each user, user identifiers, etc.

Interfaces (404) are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces (404) depends on the specific implementation of the device (400), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, and the like.

As means of I/O data (405) in any embodiment of the system that implements the described method, the keyboard must be used. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of networking (406) are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of means (405) the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, is provided.

The components of the device (400) are coupled via a common data bus (410).

Thus, when implementing this technical solution, the following positive effects can be additionally distinguished:

- The system for creating a virtual space is completely mobile and can be used indoors and outdoors;

- The system for creating a virtual space involves the use of various ways of interacting with the space (sensors, cameras, buttons, etc.), both separately and together, thereby providing the multifunctionality of the system;

- The system for creating a virtual space allows the group use of mobile space with the possibility of simultaneous interaction of users with the space;

- The system for creating a virtual space assumes that users do not need to use wearable devices (for example, VR helmets, gloves, position sensors, etc.), which makes it easier to use, simplifies the design of the entire system and improves the user experience.

Separately, it is necessary to note the necessary functional characteristics of the software installed and used in the control system:

- Ensuring the interaction of installed software with existing systems and protocols (operating system);

- Setting up and calibrating systems, including, but not limited to: mixing projectors, calibrating capture cameras, calibrating the sensor system, configuring the audio system;

- Processing and mixing of incoming video data (for example, determining the position of the body in space and creating skeletal animation / point clouds);

- Processing and mixing incoming sensory data, and building interaction points (creating a touch map in a 2D coordinate system and transferring it to a 3D one);

- Generation of virtual space in real time;

- Playback of pre-recorded visual programs (eg 360 video);

- Generation in real time and / or transfer of ready-made audio tracks for playback;

- Reception and processing of signals from external elements (mouse / keyboard, tablet, remote control).

List of implemented software.

1. A 360 player that transmits video files in 360 to projectors and a speaker system.

2. A program that generates virtual space and objects, processes incoming sensory signals and triggers scripts and triggers (Unity, Unreal, Resolume, etc.)

3. A program for tracking people, which, based on the incoming video stream in real time, creates a skeletal animation of users, selects the nodal points of the skeleton and transfers the recorded trajectories of points with the position in the 3-axis coordinate system and time stamps to the main program (https://www1.fips .ru/fips_servl/fips_servlet?DB=EVM&DocNumber=2021660804).

4. A program for aggregating devices and I / O programs, as well as integrating them into other systems, which allows you to remotely control sound and projection equipment, as well as use tablets as controllers (No. EA-53537).

5. A program for mixing (calibrating) projectors (for example, Immersive Display PRO), which allows you to set up projectors and save the settings characteristics in the configuration.

An example of the implementation of the present invention.

1. After collecting the mobile collapsible structure and installing all the elements, they are launched and calibrated.

2. The software is launched on the control system. Possible options:

2.1 Player 360. The video content from the player 360 enters the projection system, while at the same time, the audio stream from the video content is output to the audio system. Motion capture cameras and sensor systems are not active.

2.2 Virtual world. The game educational program (IOB) is being launched.

2.2.1 Initialization.

IOB on the server creates a virtual world. During initialization in the virtual world, the following processes occur:

- Appears the world with the environment, passive and interactive objects;

- Virtual cameras appear that are not visible to users, but the image from which is subsequently transmitted to projectors;

- A virtual microphone appears, which transmits the sound inside the virtual world to the audio system;

- The coordinate system for interactive systems is initialized (block of motion capture cameras 101, block of sensor systems 102) to transfer user actions to the virtual space.

2.2.2 Work.

The IOB brings the environment and audio from the virtual world to the projectors. A user or a group of users move in a complex, and data on their movement and position in space are read by the block of motion capture cameras 101, processed and transmitted to the coordinate system of the selected world. When users interact with the touch system, the touch points are also transferred to the virtual world according to the previously initialized coordinate system. In the OBI, a system of triggers can be pre-set, leading to changes in the virtual world.

In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested legal protection and are obvious to specialists in the relevant field of technology.

Claims (9)

1. Mobile system for creating a virtual space, containing: motion capture camera unit (101) consisting of at least one motion capture camera, wherein the motion capture camera unit is configured to receive video data (111), wherein the motion capture camera unit sends video data to the control system (312), in which the control system recognizes users, detects their skeleton and position of elements in the coordinate system (313); sensor systems unit (102) consisting of at least one sensor system, receiving sensor data (112) to create a sensory space, wherein the sensor systems unit sends data to the control system (302), which are processed by the control system, which determines the points of interaction in the touch surface coordinate system and synchronizes them in the virtual space coordinate system (303); block projection systems (103), consisting of at least two projectors projecting virtual space on the surface (121); an acoustic system (104) consisting of at least four speakers generating surround sound (122); control system (105), which is the core of the system and interconnects all components of the system to create a virtual space, moreover, based on the received and processed data from the block of motion capture cameras (101) and the block of sensor systems (102), the control system makes changes to the generated audio and video streams according to the previously set scenario (304) and sends these streams to the block of projection systems and/or speaker system; a mobile collapsible structure (140), which is a carrier element for attaching elements of the system to create a virtual space, and the mobile collapsible structure is configured to attach surfaces for projecting a virtual space. 2. The system according to claim 1, in which the mobile collapsible structure is implemented as a truss structure. 3. The system of claim 1, wherein the virtual space projection surface is a projection canvas.

Images