WO2022047768A1

WO2022047768A1 - Virtual experience system and method combining hololens and cave

Info

Publication number: WO2022047768A1
Application number: PCT/CN2020/113691
Authority: WO
Inventors: 汤勇刚; 林慧洁; 莫明建; 苏会卫
Original assignee: 桂林旅游学院
Priority date: 2020-09-07
Filing date: 2020-09-07
Publication date: 2022-03-10

Abstract

Disclosed are a virtual experience system and method combining Hololens and CAVE. The system comprises a CAVE system and a Hololens device. The CAVE system generates a three-dimensional virtual environment; and the Hololens device generates, in the three-dimensional virtual environment generated by the CAVE system and according to a three-dimensional model in the Hololens device, a three-dimensional holographic image of a virtual object, wherein the three-dimensional holographic image of the virtual object and the three-dimensional virtual environment are superposed to form a virtual three-dimensional scene. According to the system and the method of the present invention, by means of CAVE, the system overcomes the defect of it being difficult for a Hololen device to provide a large-view-field virtual scene due to the limitation of a field of view; moreover, by means of the characteristics of a Hololen device, i.e. being light and flexible and being capable of mixing and superposing a holographic image of a virtual object and a real scene, the defects of relatively poor virtual scene flexibility and the interactive experience feeling of the CAVE system are overcome.

Description

A virtual experience system and method combining Hololens and CAVE

technical field

The invention relates to the technical field of virtual reality, in particular to a virtual experience system and method combining Hololens and CAVE.

Background technique

The CAVE immersive virtual reality display system is based on computer graphics, and integrates high-resolution stereo projection display technology, multi-channel visual synchronization technology, audio technology, sensor technology, etc. of fully immersive virtual environments for multiplayer use. With the help of corresponding virtual reality interactive devices (such as data gloves, position trackers, etc.), the experiencer can obtain an immersive high-resolution three-dimensional audio-visual image and six-degree-of-freedom interactive experience. Since the projection surface can cover almost all the user's field of vision, the CAVE system can provide users with a shocking immersive experience. HoloLens is a mixed reality product. It is the first holographic computer device that is not limited by cables. It can mix holographic images of virtual objects with real scenes to provide a highly immersive mixed reality experience, allowing users to interact with digital content. And interact with holograms in the real world around you. Both CAVE and Hololens are widely used in virtual simulation applications with immersion requirements, including virtual design and manufacturing, virtual assembly, simulation training, virtual demonstration, virtual biomedical engineering, geology, minerals, petroleum, aerospace, scientific visualization, Military simulation command, virtual battlefield, electronic countermeasures, topography, geographic information system (GIS), architectural vision and urban planning, earthquake and fire drill simulation, etc.

The technical characteristics of both CAVE and Hololens determine their respective advantages and disadvantages. The existing CAVE virtual reality system can provide a large field of view and panoramic images, and can produce a strong sense of immersion, but it cannot provide holographic images of virtual objects, and needs to be specially equipped with VR recognition equipment, VR interaction equipment and position tracking system to achieve and The interaction between the experiencers is not flexible and natural, and the interactive experience is weak. Hololens can provide holographic images of virtual objects, superimpose computer-generated virtual objects on top of the real world, and support position tracking, hand tracking and voice recognition of the experiencer. The interaction method is flexible and natural, but the field of view is small. Even the latest Hololens product has a field of view of only 52 degrees diagonally (28.5 degrees vertically and 43 degrees horizontally), making the virtual large scene less effective.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a virtual experience system and method combining Hololens and CAVE, combining CAVE and Hololens, so that the virtual environment provided by CAVE and the virtual objects provided by Hololens are superimposed and mixed, which can provide a stronger sense of immersion, more Natural interaction and a more realistic experience.

For achieving the above object, the present invention provides the following scheme:

A virtual experience system combining Hololens and CAVE, including:

CAVE system and Hololens equipment;

The CAVE system is used to generate a three-dimensional virtual environment; the Hololens device is used to generate a three-dimensional holographic image of a virtual object according to the three-dimensional model in the Hololens device in the three-dimensional virtual environment generated by the CAVE system; The three-dimensional holographic image and the three-dimensional virtual environment are superimposed to form a virtual three-dimensional scene.

Optionally, the virtual experience system combining Hololens and CAVE further includes:

Network equipment and cloud computing platforms;

The network device is respectively connected with the Hololens device, the CAVE system and the cloud computing platform;

The Hololens device is used to scan the environment, and send the environment scan result to the cloud computing platform through the network device;

The Hololens device is configured to receive the spatial positioning point determined by the cloud computing platform according to the environmental scanning result, and transmit the spatial positioning point to the CAVE system through the network device;

The Hololens device is used to locate the experiencer to obtain the experiencer's location data, and transmit the experiencer's location data to the CAVE system through the network device; the Hololens device is also used to identify the experiencer's actions and voice, and transmit the user's action instructions and voice instructions to the CAVE system through the network device;

The CAVE system is used to perform environmental grid space mapping according to the spatial positioning point and the experiencer position data, so that the three-dimensional virtual environment corresponds to the spatial relationship of the three-dimensional holographic image of the virtual object; the CAVE system It is also used to determine the operation of the experiencer according to the action instruction and the voice instruction, and generate a three-dimensional virtual environment corresponding to the operation of the experiencer, so that the three-dimensional virtual environment and the experiencer can interact.

Optionally, the CAVE system specifically includes:

a graphics workstation, a projector and a projection space; a plurality of projection surfaces are arranged in the projection space;

The graphics workstation is respectively connected with the projector and the network device; the graphics workstation is used for transmitting the three-dimensional rendered video image to the projector, and the projector projects the video image to the The projection surface forms a three-dimensional virtual environment.

Optionally, the network device specifically includes:

switches and wireless routers;

The wireless router is respectively connected with the Hololens device, the switch and the cloud computing platform.

Audio equipment;

The audio equipment is connected with the graphics workstation; the audio equipment is used for simulating environmental sound effects.

optional,

The graphics workstation is connected with the projector through a video cable;

The graphics workstation is connected with the audio equipment through an audio cable.

The present invention also provides a virtual experience method combining Hololens and CAVE, including:

The CAVE system generates a three-dimensional virtual environment;

Place the Hololens device in the 3D virtual environment generated by the CAVE system;

The Hololens device generates a three-dimensional holographic image of the virtual object according to the three-dimensional model in the Hololens device;

The three-dimensional holographic image of the virtual object and the three-dimensional virtual environment are superimposed to form a virtual three-dimensional scene.

Optionally, the method further includes:

The Hololens device scans the environment, and sends the environment scan result to the cloud computing platform through the network device;

The Hololens device receives the spatial positioning point determined by the cloud computing platform according to the environmental scanning result, and transmits the spatial positioning point to the CAVE system through the network device;

The Hololens device locates the experiencer to obtain the experiencer's location data, and transmits the experiencer's location data to the CAVE system through the network device;

The Hololens device recognizes the action and voice of the experiencer, and transmits the action instruction and voice command of the experiencer to the CAVE system through the network device;

The CAVE system performs environmental grid space mapping according to the spatial positioning point and the experiencer position data, so that the three-dimensional virtual environment corresponds to the spatial relationship of the three-dimensional holographic image of the virtual object;

The CAVE system determines the operation of the experiencer according to the action instruction and the voice instruction, and generates a three-dimensional virtual environment corresponding to the operation of the experiencer, so that the three-dimensional virtual environment and the experiencer can interact.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The invention provides a virtual experience system and method combining Hololens and CAVE. The system includes a CAVE system and a Hololens device; the CAVE system generates a three-dimensional virtual environment; The model generates a three-dimensional holographic image of the virtual object; the three-dimensional holographic image of the virtual object and the three-dimensional virtual environment are superimposed to form a virtual three-dimensional scene. The system uses the CAVE system to make up for the shortcomings of the Hololen device, which is difficult to provide a large-view virtual scene due to the limitation of the field of view. It also uses the Hololens device, which is light and flexible, and can mix and superimpose the holographic image of the virtual object with the real scene. Features, make up for the CAVE The disadvantage of the system's virtual scene flexibility and weak interactive experience.

In addition, the Hololens device scans the environment, and sends the environmental scan results to the cloud computing platform through the network device, and simultaneously receives the spatial positioning points determined by the cloud computing platform according to the environmental scanning results, and transmits the spatial positioning points to the CAVE system through the network device. ; Hololens equipment locates the experiencer to obtain the experiencer's position data, and transmits the experiencer's position data to the CAVE system through the network device; Corresponding to the spatial relationship of the three-dimensional holographic image of the virtual object, it can provide a stronger sense of immersion, a more natural interaction method and a more realistic experience; Touch, grasp and move the holographic image) and voice to manipulate and interact with the virtual scene, replacing the need to use virtual reality interactive devices (such as data gloves, position trackers, etc.) in the traditional CAVE system. The problem of obtaining the "six degrees of freedom interactive feeling" with the virtual scene.

Instruction drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

FIG. 1 is a structural diagram of a virtual experience system combining Hololens and CAVE in an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to make the objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

Example

FIG. 1 is a structural diagram of a virtual experience system combining Hololens and CAVE in an embodiment of the present invention. As shown in FIG. 1, a virtual experience system combining Hololens and CAVE includes: CAVE system, Hololens equipment, network equipment, and cloud computing platform and audio equipment. The network equipment is connected with Hololens equipment, CAVE system and cloud computing platform respectively. The network equipment includes: switches and wireless routers; the wireless routers are respectively connected with Hololens equipment, switches and cloud computing platforms.

The CAVE system is used to generate a 3D virtual environment; the Hololens device is used to generate a 3D holographic image of the virtual object in the 3D virtual environment generated by the CAVE system according to the 3D model in the Hololens device; the 3D holographic image of the virtual object and the 3D virtual environment are superimposed to form a virtual 3D scene.

The Hololens device is used to scan the environment and send the environmental scan results to the cloud computing platform through the network device; the Hololens device is used to receive the spatial positioning points determined by the cloud computing platform according to the environmental scanning results, and transmit the spatial positioning points through the network device. To the CAVE system; Hololens equipment is used to locate the experiencer to obtain the experiencer's location data, and transmit the experiencer's location data to the CAVE system through the network device; The grid space mapping makes the three-dimensional virtual environment correspond to the spatial relationship of the three-dimensional holographic image of the virtual object. Hololens supports manual tracking, can fully articulate models with both hands, and can touch, grasp and move holographic images in a realistic and natural way; can execute voice commands through smart microphones and natural language speech processing; can perform worldwide location tracking and tracking Real-time environment grid space mapping.

The Hololens device is also used to recognize the user's actions and voice, and transmit the user's action and voice commands to the CAVE system through the network device; the CAVE system is also used to map the environment grid space according to the action and voice commands, so that The three-dimensional virtual environment corresponds to the spatial relationship of the three-dimensional holographic image of the virtual object, and feels the operation of the experiencer, and implements corresponding changes (including menu operations, system settings, selecting virtual objects, moving virtual objects, scene switching, field of view rotation, zooming and panning) to create interaction between the three-dimensional virtual environment and the experiencer. The actions of the experiencer include touching, grasping and moving the three-dimensional holographic image by the experiencer.

The CAVE system specifically includes: a graphics workstation, a projector and a projection space; there are 3 to 5 projection surfaces in the projection space, including: left, front, right, upper, and lower; multiple projection surfaces surround the experiencer to form a three-dimensional experience Space, each side can use rear projection or front projection, and use mirrors to save space. The graphics workstation is respectively connected with the projector and the network device; the graphics workstation is used to transmit the three-dimensionally rendered video image to the projector, and the projector projects the video image to the projection surface to form a three-dimensional virtual environment.

The audio equipment is connected with the graphics workstation, the audio equipment is used to simulate environmental sound effects, the graphics workstation is connected with the projector through a video cable, and the graphics workstation is connected with the audio device through an audio cable.

The control software runs on the graphics workstation of the CAVE system, performs graphics rendering according to the three-dimensional model of the CAVE system, and transmits the images to each projector respectively. Fusion together to form a three-dimensional virtual environment of CAVE. The experiencer wears the Hololens helmet to enter the CAVE system, and can see the virtual environment generated by the CAVE system through the Hololens glasses, creating a sense of immersion in the virtual environment. At the same time, the Hololens device generates three-dimensional holographic images of virtual objects, including people, animals and plants, and various non-living objects, according to the three-dimensional model. Through the glasses on the helmet, the experiencer observes that virtual objects are superimposed into the three-dimensional virtual environment provided by the CAVE system. The virtual environment provided by the CAVE system and the virtual objects provided by the Hololens device are mixed and superimposed to form a realistic and immersive virtual 3D scene. Hololens devices connect to the Internet through a wireless router, perform high-quality remote 3D rendering through the Microsoft Azure cloud computing platform, and create spatial anchors using Azure's spatial anchor service. The Hololens equipment shares the coordinate system of the spatial positioning point to the graphics workstation through the network, and the control software on the graphics workstation performs real-time environment grid space mapping according to the shared spatial positioning point coordinate system, so that the virtual environment of the CAVE and the holographic image have a spatial relationship. remain consistent. The Hololens device locates and tracks the experiencer, and recognizes the experiencer's movements and voices. The experiencer realizes the touch, grasp and movement of the holographic image through the movements of the hands, and controls and interacts with the virtual scene through hand movements and voice.

A virtual experience method combining Hololens and CAVE provided by the present invention includes:

The CAVE system generates a 3D virtual environment; the Hololens device is placed in the 3D virtual environment generated by the CAVE system; the Hololens device generates a 3D holographic image of the virtual object according to the 3D model in the Hololens device; the 3D holographic image of the virtual object and the 3D virtual environment are superimposed to form Virtual 3D scene.

The Hololens device scans the environment, and sends the environmental scan results to the cloud computing platform through the network device; the Hololens device receives the spatial positioning points determined by the cloud computing platform according to the environmental scanning results, and transmits the spatial positioning points to the CAVE system through the network device; The Hololens device locates the experiencer to obtain the experiencer's location data, and transmits the experiencer's location data to the CAVE system through the network device; the Hololens device recognizes the experiencer's movements and voice, and sends the experiencer's action and voice commands through the network device. Transmission to the CAVE system; the CAVE system maps the environment grid space according to the spatial positioning point and the experiencer's position data, so that the 3D virtual environment corresponds to the spatial relationship of the 3D holographic image of the virtual object; the CAVE system senses the experiencer according to the action instructions and voice instructions operations, and implement corresponding changes (including menu operations, system settings, selecting virtual objects, moving virtual objects, scene switching, rotation, zooming and panning of the field of view), so that the three-dimensional virtual environment interacts with the experiencer .

While providing a large-view virtual scene, the present invention can mix and superimpose the holographic image of the virtual object with the real scene, which makes up for the shortcomings of the CAVE virtual scene's weak flexibility and interactive experience and the short HoloLens' visual field. Greater immersion, more natural interactions, and a more realistic experience. Since the existing CAVE virtual reality system can provide a large field of view and panoramic images, it can produce a strong sense of immersion, but it cannot provide holographic images of virtual objects, and needs to be specially equipped with VR recognition equipment, VR interactive equipment and position tracking system to achieve The interaction with the experiencer is less flexible and interactive. Hololens can provide holographic images of virtual objects, and can superimpose computer-generated virtual objects on top of the real world, and support the position tracking, hand tracking and voice recognition of the experiencer, but the field of view of the latest Hololens second generation products Smaller, the diagonal is only 52 degrees (28.5 degrees vertically, 43 degrees horizontally), and the effect of virtual large scenes is weak. The invention utilizes the complementary characteristics of each other, combines CAVE and Hololens, realizes the mixing of the virtual object holographic image of Hololens and the virtual environment of CAVE, uses Hololens to locate and track the experiencer, recognizes the action and voice of the experiencer, and solves the problem of solving the problem. The interaction between the experiencer and the virtual scene makes up for the shortcomings of the CAVE virtual scene's weak flexibility and interactive experience and the shortcomings of the HoloLens' small field of view, which can provide a stronger sense of immersion, a more natural interaction method and more Realistic experience.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

The above embodiments are provided for the purpose of describing the present invention only, and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalent replacements and modifications made without departing from the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

A virtual experience system combining Hololens and CAVE, characterized in that it includes:

CAVE system and Hololens equipment;

The CAVE system is used to generate a three-dimensional virtual environment; the Hololens device is used to generate a three-dimensional holographic image of a virtual object according to the three-dimensional model in the Hololens device in the three-dimensional virtual environment generated by the CAVE system; The three-dimensional holographic image and the three-dimensional virtual environment are superimposed to form a virtual three-dimensional scene.
The virtual experience system combining Hololens and CAVE according to claim 1, wherein the virtual experience system combining Hololens and CAVE further comprises:

Network equipment and cloud computing platforms;

The network device is respectively connected with the Hololens device, the CAVE system and the cloud computing platform;

The Hololens device is used to scan the environment, and send the environment scan result to the cloud computing platform through the network device;

The Hololens device is configured to receive the spatial positioning point determined by the cloud computing platform according to the environmental scanning result, and transmit the spatial positioning point to the CAVE system through the network device;

The Hololens device is used to locate the experiencer to obtain the experiencer's location data, and transmit the experiencer's location data to the CAVE system through the network device; the Hololens device is also used to identify the experiencer's actions and voice, and transmit the user's action instructions and voice instructions to the CAVE system through the network device;

The CAVE system is used to perform environmental grid space mapping according to the spatial positioning point and the experiencer position data, so that the three-dimensional virtual environment corresponds to the spatial relationship of the three-dimensional holographic image of the virtual object; the CAVE system It is also used to determine the operation of the experiencer according to the action instruction and the voice instruction, and generate a three-dimensional virtual environment corresponding to the operation of the experiencer, so that the three-dimensional virtual environment and the experiencer can interact.
The virtual experience system combining Hololens and CAVE according to claim 2, wherein the CAVE system specifically includes:

a graphics workstation, a projector and a projection space; a plurality of projection surfaces are arranged in the projection space;

The graphics workstation is respectively connected with the projector and the network device; the graphics workstation is used for transmitting the three-dimensional rendered video image to the projector, and the projector projects the video image to the The projection surface forms a three-dimensional virtual environment.
The virtual experience system combining Hololens and CAVE according to claim 3, wherein the network device specifically includes:

switches and wireless routers;

The wireless router is respectively connected with the Hololens device, the switch and the cloud computing platform.
The virtual experience system combining Hololens and CAVE according to claim 4, wherein the virtual experience system combining Hololens and CAVE further comprises:

Audio equipment;

The audio equipment is connected with the graphics workstation; the audio equipment is used for simulating environmental sound effects.
The virtual experience system combining Hololens and CAVE according to claim 5, is characterized in that,

The graphics workstation is connected with the projector through a video cable;

The graphics workstation is connected with the audio equipment through an audio cable.
A virtual experience method combining Hololens and CAVE, characterized by comprising:

The CAVE system generates a three-dimensional virtual environment;

Place the Hololens device in the 3D virtual environment generated by the CAVE system;

The Hololens device generates a three-dimensional holographic image of the virtual object according to the three-dimensional model in the Hololens device;

The three-dimensional holographic image of the virtual object and the three-dimensional virtual environment are superimposed to form a virtual three-dimensional scene.
The virtual experience method combining Hololens and CAVE according to claim 7, wherein the method further comprises:

The Hololens device scans the environment, and sends the environment scan result to the cloud computing platform through the network device;

The Hololens device receives the spatial positioning point determined by the cloud computing platform according to the environmental scanning result, and transmits the spatial positioning point to the CAVE system through the network device;

The Hololens device locates the experiencer to obtain the experiencer's location data, and transmits the experiencer's location data to the CAVE system through the network device;

The Hololens device recognizes the action and voice of the experiencer, and transmits the action instruction and voice command of the experiencer to the CAVE system through the network device;

The CAVE system performs environmental grid space mapping according to the spatial positioning point and the experiencer position data, so that the three-dimensional virtual environment corresponds to the spatial relationship of the three-dimensional holographic image of the virtual object;

The CAVE system determines the operation of the experiencer according to the action instruction and the voice instruction, and generates a three-dimensional virtual environment corresponding to the operation of the experiencer, so that the three-dimensional virtual environment and the experiencer can interact.