WO2018058601A1

WO2018058601A1 - Method and system for fusing virtuality and reality, and virtual reality device

Info

Publication number: WO2018058601A1
Application number: PCT/CN2016/101255
Authority: WO
Inventors: 骆磊
Original assignee: 深圳达闼科技控股有限公司
Priority date: 2016-09-30
Filing date: 2016-09-30
Publication date: 2018-04-05
Also published as: CN107077755A; CN107077755B

Abstract

Disclosed are a method and system for fusing virtuality and reality, and a virtual reality device. The method comprises: acquiring a realistic three-dimensional panoramic image of a reality environment around a user; rendering the realistic three-dimensional panoramic image according to a rendering virtual degree selected by the user; and outputting a rendered result to a virtual reality display module. By means of the method and system and the virtual reality device in the present application, realistic elements of a surrounding environment of a user are superimposed and fused on the virtual content in a virtual degree adjustable manner, so that the user can also walk and complete certain actions while wearing a helmet, thereby providing a new VR experience for the user.

Description

Virtual and reality fusion methods, systems and virtual reality devices

Technical field

Embodiments of the present application relate to the field of virtual reality (VR), and in particular, to a method and system for superimposing rendered content related to a real element in a virtual reality display environment.

Background technique

In the current cutting-edge technology research, virtual reality (VR) and augmented reality (AR) are highly concerned technical research directions. VR is a completely virtual content for users. Users bring a head-mounted VR device, immersed in a completely virtual world, can watch and listen, but can not be sensed by smell, taste, touch, completely isolated from reality. . Augmented Reality (AR) adopts a semi-lens method, allowing users to see the real world, while displaying virtual objects or virtual objects on the semi-lens, so that the virtual fusion into the real environment, the user sees the real A small amount of virtual content appears in the world.

For virtual reality devices, existing head mounted VR devices typically include a head mounted display and a VR content generating device.

The head mounted display can be worn on the user's head and provide the user with an immersive field of view of the virtual scene. In addition to this, the head mounted display also includes sensors for head positioning.

The VR content generation device includes a calculation module, a storage module, and a head positioning module. The head positioning module obtains data from the head positioning sensor in the head mounted display in real time, and is processed by the sensor fusion related algorithm, and the head positioning module can obtain the current user's head posture.

The VR content generating device obtains the current head pose from the head positioning module, obtains the material required for rendering the virtual scene from the storage module, and finally processes the virtual scene with the current user's head posture as a perspective. And displayed to the user through the head-mounted display. Head-mounted displays and VR content-generating devices can be embedded integrated (such as VR mobile all-in-ones) or connected together via display data lines (such as HDMI) (such as HTC Vive).

Applicants of the present application, however, have found that existing head-mounted VR devices can only present virtual scenes to a user in a particular mode. The degree of rendering cannot be adjusted according to the user's wishes. Therefore, The prior art has yet to be improved and developed.

Application content

The technical problem to be solved by the embodiments of the present application is to provide a new virtual reality experience method and system for adjusting the degree of rendering according to the user's wishes. The method and system realize the superposition of the virtual elements on the virtual content of the virtual reality system to superimpose the realistic elements of the user's surrounding environment, and bring the user a new VR experience.

In order to solve the above technical problem, one technical solution adopted by the embodiment of the present application is: providing

A fusion method of virtual and reality, including:

Obtain a realistic three-dimensional panoramic image of the real environment around the user;

Rendering the realistic three-dimensional panoramic image according to the rendering virtual degree selected by the user;

The rendered result is output to the virtual reality display module.

In order to solve the above technical problem, another technical solution adopted by the embodiment of the present application is to provide a virtual reality integration system, including a virtual reality system, for displaying a virtual three-dimensional scene for a user, and a reality fusion module, wherein The reality fusion module is configured to acquire a realistic three-dimensional panoramic image of a real environment around the user; the virtual reality system is configured to render the realistic three-dimensional panoramic image according to a rendering virtual degree selected by the user; the virtual reality system is further used for rendering The obtained result is output to the virtual reality display module.

In order to solve the above technical problem, another technical solution adopted by the embodiment of the present application is to provide an electronic device, including: a processor module, an interaction module connected by the processor module, and a virtual reality display module;

The processor module is configured to acquire a realistic three-dimensional panoramic image according to an image of a surrounding environment of the user, and render the realistic three-dimensional panoramic image according to a rendering virtual degree selected by the user in the interaction module;

The processor module is further configured to output the rendered result to the virtual reality display module.

Preferably, the electronic device is a virtual reality wearing device, and further comprises a helmet and a reality capturing module.

The reality capture module is configured to capture images of a user's surroundings from multiple angles. In this way, when the user brings the virtual reality wearing device, the user can well immerse the virtual display ring. In the environment, improve the user's embodiment.

Preferably, the interaction module includes a quantity adjustment interaction device, wherein the processor module is further configured to identify and extract a realistic element in the realistic three-dimensional panoramic image; and acquire a first rendering virtual degree selected by the user from the quantity adjustment interaction device. The first rendering virtual level is used to indicate a proportion of real-world elements that the user wishes to retain; and to process the real-world elements in the real-world three-dimensional panoramic image according to the acquired first rendering virtual degree.

In order to ensure user security, the real-life element includes a security reality element, wherein the processor module is further configured to reserve the real-world three-dimensional when the first rendering virtual degree indicates that the proportion of the real-life element that the user wishes to retain is non-zero. At least a portion of the security reality elements in the panoramic image.

Preferably, the interaction module further includes a rendering virtuality interaction device quantity adjustment interaction device, wherein the processor module is further configured to identify and extract a realistic element in the realistic three-dimensional panoramic image; and obtain a user selection from the rendering virtuality interaction device. The second rendering virtual degree is used to indicate the rendering degree of each real element, and render each real element in the real three-dimensional panoramic image according to the acquired second rendering virtual degree.

In order to provide an immersive VR experience, the processor module is configured to determine a sound rendering corresponding to the virtual three-dimensional scene according to the security reality element and the other real-life elements.

The reality capture module includes a plurality of cameras disposed in different orientations of the helmet.

In an embodiment, the interaction module includes a knob and an analog-to-digital conversion device; the analog-to-digital conversion device is coupled to the knob and the processor module, and configured to generate a corresponding digital signal according to a rotation angle of the knob and output the same to the processor Module.

The beneficial effect of the embodiment of the present application is that the image in the surrounding real environment can be rendered according to the rendering virtual degree selected by the user. Allows users to choose the rendering virtual level to help improve the user experience.

DRAWINGS

1 is a block diagram of a virtual reality integration system according to an embodiment of the present application;

2 is a perspective view of a virtual reality integration system according to an embodiment of the present application;

3 is a perspective rear view of a virtual reality integration system according to an embodiment of the present application;

4 is a flowchart of an embodiment of a virtual and reality fusion method according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of hardware of a virtual reality integration system according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a hardware structure of an electronic device in a virtual and reality fusion method according to an embodiment of the present application.

detailed description

In order to make the objectives, technical solutions, and advantages of the embodiments of the present application more clearly, the embodiments of the present application are further described in detail below with reference to the accompanying drawings. The illustrative embodiments of the present application and the description thereof are for explaining the present application, but are not intended to limit the application.

FIG. 1 is a schematic diagram of an implementation environment, which may be applied to a processor module 100, a virtual reality display module 200, a real-life capture module capable of collecting a surrounding real environment, and capable of interacting with a user to determine a user. In the electronic device of the selected virtual degree interaction module 300, each component included in the electronic device can be connected through a communication interface (compared to an I/O interface).

The reality capture module can be a three-dimensional image recognition sensor that can be modeled in the forward direction or in multiple directions for real-time modeling (similar devices are already available on the market). If only one face-oriented three-dimensional image recognition sensor is used, the real-time requirements are extremely high, and the head rotation requires substantially time-delayed modeling and rendering, and the viewing angle is extremely demanding; and if it is a plurality of three-dimensional The image recognition sensor splicing the realistic three-dimensional panoramic image is equivalent to constructing a spatial model, and the head rotation is only another part of the display space.

Referring to FIG. 5 together, the processor module in this embodiment may include a central processing unit module (including a central processing unit CPU 520 and a storage medium 530 connected to the central processing unit), and may further include an image processing module (image The processing module can include a graphics processor GPU 510 and a storage medium 530 coupled to GPU 510). The latter case is shown in FIG. The virtual and reality fusion system of this embodiment may be a software system stored in a storage medium of the processor module described above. Referring to the figure, the system can include a reality fusion module and a virtual reality system.

The reality fusion module acquires a realistic three-dimensional panoramic image of a real environment around the user. For example, in some embodiments, the reality fusion module can pre-acquire images of real environments around the user captured by multiple cameras.

The plurality of cameras are photographed based on the same stereo coordinate system to splicing out the three-dimensional panoramic image.

The reality fusion module acquires a realistic three-dimensional panoramic image of a real environment around the user;

The virtual reality system renders the realistic three-dimensional panoramic image according to the rendering virtual degree selected by the user; the virtual reality system outputs the rendered result to the virtual reality display module.

The virtual reality system identifies and extracts realistic elements in the realistic three-dimensional panoramic image.

The virtual reality system provided by the embodiment of the present application can be applied to a device such as a helmet. When used in a device such as a helmet, the user can walk with a helmet, and the virtual reality system can be in front of the user according to the helmet worn by the user in real time. The image is rendered. In this case, in order to ensure the security of the user, in the specific implementation: it can be implemented as follows:

The virtual reality system retains at least a portion of the security reality elements in the realistic three-dimensional panoramic image when the first rendering virtual level indicates that the proportion of the real-world elements that the user wishes to retain is non-zero. In order to make a selection number for the rendered object, the first rendering virtual level is used to indicate the proportion of real-world elements that the user wishes to retain.

The real elements in the realistic three-dimensional panoramic image are processed according to the acquired first rendering virtual degree.

In addition to rendering according to the first rendering virtual degree, the rendering degree of each (retained) real element may be according to the second virtual rendering degree (the second rendering virtual degree is used to indicate the rendering degree of each real element) Rendering. At this point: the corresponding rendering process can include:

Identifying and extracting realistic elements in the realistic three-dimensional panoramic image;

Obtaining a second rendering virtual degree selected by the user; the second rendering virtual degree is used to indicate a rendering degree of each real element;

And rendering each real element in the realistic three-dimensional panoramic image according to the acquired second rendering virtual degree.

This enables the user to autonomously select the degree of blurring of each real element. Not difficult to understand The solution is that, in the specific implementation, each real element can also be rendered according to the second virtual rendering degree.

In a specific implementation, the processor module acquires a second rendering virtual degree selected by the user, where the second rendering virtual level is used to indicate the rendering degree of each real element. The image processing module renders each real element in the real three-dimensional panoramic image according to the acquired second rendering virtual degree.

For those of ordinary skill in the art, the extraction and rendering of real-world elements can be done separately by the central processing module, either separately by the image processing module or by distributed processing by the central processing module and the image processing module.

The traditional virtual reality system includes a central processing module, a rendering module, a sensor, an interaction module, and a virtual reality display module. One of the embodiments of the virtual reality display module is VR imaging glasses. The interaction module aggregates data of all sensors, and the user's behavior and interaction instructions are sent to the interaction module and the central processing module through the sensor. The central processing module displays the virtual content through the rendering module and displays the virtual content on the VR imaging glasses three-dimensionally according to the indication of the user at the interaction module.

VR Imaging Glasses: Two discrete display devices, the same as traditional VR imaging glasses.

Sensors: In addition to traditional motion sensors, various other types of sensors can be included, depending on the needs of the content.

The virtual reality system in the embodiment of the present application further provides a real-life capture module for acquiring a realistic three-dimensional panoramic image of the user's surrounding environment, an interaction module for implementing interaction with the user, and a unique image processing module.

The reality fusion module includes a plurality of realistic capture modules for acquiring realistic three-dimensional panoramic images of the real environment around the user, an interaction module for realizing virtual quantity and virtuality adjustment, and a unique image processing module.

The plurality of real-time capture modules are simultaneously captured by a plurality of three-dimensional modeling modules that are covered by the viewing angles and are spliced to form a realistic three-dimensional panoramic image. In this embodiment, the real-time capture module is a three-dimensional image recognition sensor 540. In order to realize the fusion of the real elements, the image processing module comprises a real element extraction module, a fusion matching module, an image processing module for merging images, and a sound processing module for merging sounds.

The interaction module is for the user to select a virtual reality level of the realistic three-dimensional panoramic image or the real element on the virtual reality wearing device.

Referring to FIG. 2 and FIG. 3 together, the virtual reality wearing device includes a helmet 1, a central processing module 2, an earphone holder 3, a data line 4, a

power port

5, and 5 realistic capture modules 6-10, a microphone. 20. The virtuality interaction device 30 and the real element number interaction device 40 are rendered.

The central processing module: receives all the real capture modules 6-10, the sensors 540, the peripheral devices, the rendering virtuality interaction device 30 and the real element number interaction device 40 and the content source signals, and transmits the processed signals to the VR imaging glasses. 550, output devices such as headphones. For example, image stitching processing of each reality capture module, display of image changes after the head is rotated, and the like.

The reality capture module 6-10 acquires a realistic three-dimensional panoramic image of the real environment around the user; the image processing module renders the realistic three-dimensional panoramic image according to the rendering virtual degree selected by the user; wherein the central processing module renders the obtained result Output to the virtual reality display module.

The reality capture module 6-10 identifies and extracts the real elements in the realistic three-dimensional panoramic image, and the central processing module can also perform the recognition and extraction of the real elements.

In the embodiment of the present application, in order to implement the selection of the number of rendered objects, the interaction module includes a quantity adjustment interaction device, and the user implements setting of the first rendering virtual degree by using the quantity adjustment interaction device. The image processing module retains at least a portion of the security real-life elements in the realistic three-dimensional panoramic image when the first rendering virtual level indicates that the proportion of the real-world elements that the user wishes to retain is non-zero.

The interaction module further includes a rendering virtuality interaction device by which the user completes the setting of the second rendering virtual degree. The central processing module acquires a second rendering virtual degree selected by the user from the quantity adjustment interaction device, where the second rendering virtual level is used to indicate the rendering degree of each real element.

In view of the immersive experience, the sound processing module is further configured to determine a sound rendering corresponding to the virtual three-dimensional scene according to the security reality element and the other real-life elements.

Preferably, the first rendering virtual degree or the second rendering virtual degree is determined by a ratio selected by the user on the interactive interface.

The reality capture module 6-10 of the embodiment can measure the depth of field and construct a three-dimensional model in real time, and the actual number of applications depends on the perspective and processing capability of a single reality capture module. Real capture mode The embodiment of the group takes a plurality of real-time modeling three-dimensional image recognition sensors as an example, and distributes the top of the helmet 1 and the five front, rear, left, and right directions according to the coverage requirements of the viewing angle.

Referring to FIG. 1 together, as a specific implementation manner of the present application, the reality capture module is connected to the image recognition module. The image recognition module identifies all of the realistic elements of the stitched realistic three-dimensional panorama. The solid coordinates of the real elements are consistent with their coordinates in the three-dimensional panorama.

The virtual reality system can display a virtual three-dimensional scene for the user. The reality fusion module of the reality fusion module acquires a realistic three-dimensional panoramic image of the real environment around the user, and the image recognition module identifies all the realistic elements in the realistic three-dimensional panoramic image.

The Reality Element Extraction module extracts security reality elements from all real-world elements.

The realistic elements include security reality elements and other realistic elements selected by the user. The image processing module of the reality fusion module combines the security reality elements corresponding to the image rendering and the selected other realistic elements in the virtual three-dimensional scene of the virtual reality system according to the security reality element and other realistic elements selected by the user. . The reality fusion module also includes a sound processing module. The sound processing module is configured to determine a sound rendering corresponding to the virtual three-dimensional scene according to the security reality element and other real-life elements, and implement the fused sound.

The setting of the second rendering virtual degree is described below. The virtual rendering of the security reality elements and other selected real-world elements can be set by the user through the interaction module. The interaction module includes a rendering virtuality interaction device 30, and the reality fusion module is further configured to render a virtual degree according to a real element selected by the user in the rendering virtuality interaction device 30, and select all the realities in the virtual three-dimensional scene of the virtual reality system. The element performs image rendering corresponding to the selected degree of virtuality.

The rendering virtuality interaction device 30 is used by the user to adjust the rendering virtuality/trueness of the fused real-world elements. It can be a hardware knob or a software knob. In the software knob embodiment, the user implements the selection through the interaction module. The following is an example of a hardware knob.

The rendering virtuality interaction device 30 sets a number of realistic element number positions for the user to select, wherein, when adjusted to the most virtual state, the display of the fused real-life elements and the content are completely virtualized, such as rendering a tall building into a castle. , the sky is rendered into the universe, etc., according to the content source of the current virtual reality; when adjusted to the most real state, the reality 3D panoramic image or the display of the fused real element is directly superimposed in the virtual 3D scene of the virtual reality system. Completely true. In this case, there will be a phenomenon that the virtual content cannot be merged, but The user can know the state of the environment around him, and there are very few cases where the user actually uses it. The middle gear adjusts the realism/virtuality of the actual element display, such as a running ordinary car, which is rendered as a running monster in the most virtual situation, and may gradually become able to rotate as it moves to the real side. Locomotives, tanks, armored vehicles, etc. that incorporate virtual content styles. Like a river, in the most virtual situation, the bank is rendered as a cliff, and as it rotates to the side of reality, it may gradually become a waterfall, sea, lake, etc. that can be integrated into the virtual content style. Note that in the rendering of realistic elements Full consideration of safety factors.

The setting of the first rendering virtual degree is described below. The real element number interaction device 40 is used by the user to adjust the number of other realistic elements of the fusion. In this embodiment, the security reality element must be fused and displayed in the virtual three-dimensional scene, thereby ensuring the security of the user's activity and preventing collision and fall. The real element number interaction device 40 can be a hardware knob or a software knob. In the software knob embodiment, the user implements selection through an interaction module. The following is an example of a hardware knob.

The set 0 position means that the real elements are not displayed and rendered, and no objects in the surrounding real environment will be seen. When adjusting to the 0 position, it is the same as the current VR device, and will not see any surrounding reality. Object; the set 100 gear position indicates that all the realistic elements extracted by the real environment around the user are superimposed and displayed in the virtual three-dimensional scene; the middle gear position of 0-100 adjusts the superimposed display quantity of the real element in the virtual three-dimensional scene according to the proportion of the gear position. The middle gear adjusts how much the real element is displayed in the virtual 3D scene. For example, it is possible to display the chair and sofa around the user, but not the TV, air conditioner and other equipment. The knob only controls how much, and the actual display does not display which real element is determined by the central processing module according to the matching degree of the actual element and the virtual three-dimensional content.

In order to realize the fusion of the virtual content and the superimposed content of the real element, the matching fusion module of the real fusion module sorts the matching fusion degree of the real elements other than the security real element, and matches and merges the virtual three-dimensional scene according to the real element. Degrees determine whether the corresponding real elements are superimposed and displayed in the virtual three-dimensional scene.

The equipment workflow is as follows:

The central processing module reads the virtual content or game to be played to form a three-dimensional model of the content and surface rendering information. If the real element number interaction device 40 is 0 or renders true If the degree knob is completely true, the content can be played or run directly without any processing;

A plurality of stereo modeling modules 6-10 work simultaneously. At this time, the setting of the real element number interaction device 40 is not 0, and a real three-dimensional panoramic image in various directions around the user is obtained. The central processing module splices the three-dimensional data of each of the plurality of stereo modeling modules to obtain a realistic three-dimensional panoramic image and model of the real environment around the user. The image recognition module extracts all realistic elements from the realistic three-dimensional panoramic image and model.

The real element extraction module extracts security realities and determines other real elements selected by the user.

The image processing module determines the fusion rendering information of the aforementioned security reality element and other real elements selected by the user.

The image processing module sorts the matching degree of all the real elements according to the position of the interactive element 40 and the three-dimensional model of the current virtual content according to the actual number of elements, and determines and selects which of the other realistic elements selected by the user. Elements can be more integrated into the current virtual content from the shape structure model. For example, the VR virtual world can adjust the three-dimensional model of the real element, such as modifying the model of a building into a castle, but try to ensure the outer dimensions of the castle and the building. Basically consistent. The matching fusion module specifically performs the sortability integration of the three-dimensional models of the real-life elements, and renders according to the three-dimensional model of the plurality of suitable realistic elements before the location of the interactive element 40. The selection must take into account the user's security issues, such as steps, walls, rivers, etc. All elements that may pose a threat to personal safety must be displayed.

According to the position of the rendering virtuality interaction device 30 and the three-dimensional stereo model and rendering data of the current virtual content, the selected respective real elements are subjected to surface rendering capable of being integrated into the VR virtual three-dimensional scene. For example, if the 3D model of the realistic element of a building is modified into a 3D model of the castle with the same outer diameter at the bottom, in this step, the 3D model of the castle is integrated into the surface rendering of the virtual content, such as style, color, surface material, old and new, and sun reflection. Etc. Similarly, users should be fully considered for security issues such as steps, walls, rivers, etc. Elements that may pose a threat to personal safety must be rendered realistically and must be realistically rendered. Steps in the following building may be rendered as mossy steps according to the scene in the virtual, but may not be rendered as a downhill or a flat road.

The sound processing module of the fused sound is controlled by the central processing module to perform different sound renderings in accordance with different three-dimensional virtual scenes. For example, if the user opens a door, the user in the VR virtual three-dimensional world is in a prison scene, and the opened door is rendered into an iron fence door. At the same time, when the user opens the door, the sound should also be correspondingly rendered into the sound of opening the iron fence door.

During user motion or head rotation, based on the data of sensor 540, the central processing module processes the display and rendering of the synchronized virtual content and the surrounding real-world elements in real time.

The technical solution of the present application is based on existing VR devices, providing a completely different experience. The VR helmet that allows the user to wear a completely obscured view can still walk and move. It can be said that the experience is better than the VR virtual experience, and the fusion of virtual and reality. While seeing the virtual world, the user can independently adjust the amount of reality elements and virtualness displayed in the virtual three-dimensional world in the surrounding real environment. From the visual to the auditory, the virtual reality is added to the pure virtual experience, such as sitting On the long bench rendered by the real high step, virtual and reality are difficult to distinguish, and it is easy for users to immerse themselves in this new world of virtual reality and reality. In addition, the embodiment also breaks the limitation that the traditional VR is only used in the pure virtual environment, and at the same time, the user can ensure the security of the user in the technical way, so that the user can walk around, or even walk to the street, and can be worn at any time, this embodiment The world of virtual and real-world hybrids is no longer limited to pure virtual environments, it is a fly-by for VR device applications.

The embodiment further provides a virtual and reality fusion method, which can be performed by the processor module of the above electronic device, and the method may include

Obtaining a realistic three-dimensional panoramic image of a real environment around the user, and identifying and extracting a realistic element in the realistic three-dimensional panoramic image;

Merging the realistic three-dimensional panoramic image or the realistic element in a virtual three-dimensional scene of the virtual reality system;

The realistic three-dimensional panoramic image or the real element is rendered according to the virtual degree selected by the user, and the realistic three-dimensional panoramic image or the image-selected realistic element is rendered in the virtual three-dimensional scene of the virtual reality system.

The reality element includes a security reality element; according to the security reality element and other realistic elements selected by the user, the security reality element corresponding to the image rendering and other selected real elements are merged in the virtual three-dimensional scene of the virtual reality system. According to the security reality The element and the other real element determine a sound rendering corresponding to the virtual three-dimensional scene.

Referring to FIG. 4, a flow of one embodiment of the virtual and reality fusion method of the present embodiment is shown. Wherein, when the user starts the virtual reality system, the three-dimensional modeling module 6-10 and the fused rendering are not necessarily started synchronously. When the user selects the number of realistic element fusions and renders the virtual degree, the reality fusion module starts working.

Step 401: The central processing module presents the virtual three-dimensional scene to the user through the VR imaging glasses 550.

Step 402: Determine whether the user starts the real fusion module, and if not, continue to present the virtual three-dimensional content and the scene in the virtual reality system working mode; if activated, the real-life capturing module and the merge rendering start working.

Step 403: Several reality capture modules 6-10 simultaneously capture stereoscopic images and splicing to form a realistic three-dimensional panoramic image or model.

Step 404: The image recognition module identifies all the realistic elements according to the realistic three-dimensional panoramic image.

Step 406: The real element extraction module extracts a security reality element with a fusion priority.

Step 407: determining whether the user selects the number of actual element fusions, and if not selected, waiting for the user to select in the virtual reality system working mode; wherein, when the user selects the number of real elements, the number of the actual elements is selected, wherein The set 0 position indicates that the real element is not displayed and rendered, and no objects existing in the surrounding real environment are seen; the set 100 position indicates that all the realistic elements extracted by the real environment around the user are superimposed in the virtual three-dimensional scene; 0-100 The middle gear adjusts the number of superimposed displays of the real elements in the virtual three-dimensional scene according to the proportion of the gears.

Step 408: The fusion matching module fuses the security reality element and other selected real elements in the virtual three-dimensional scene.

Step 409: Determine whether the user selects the rendering virtual degree, if not selected, wait for the user to select in the virtual reality system working mode; if selected, render the virtual degree according to the realistic element selected by the user, in the virtual reality system Image rendering corresponding to the selected virtual degree is performed on all selected real elements in the virtual three-dimensional scene.

Step 410: The image and sound processing module renders the security reality element and other real elements that are merged in the virtual three-dimensional scene according to the rendering virtual degree.

The virtual and reality fusion method and system of the embodiment realize real-time three-dimensional modeling module and fusion image processing module, and realize real-time elements integrated into the user in the VR virtual three-dimensional world or image rendering based on the real-life elements. Providing a unique experience that is illusory and beyond reality, and the real elements are real, presented in a virtual three-dimensional world that allows users to wear helmets and perceive the real world around them. By setting the proportion, the true degree, and the rendering scene of the real-life elements in the virtual three-dimensional world, the user can make the fusion video seen by the user through the VR imaging glasses 550 the same or different. Although AR is also a mixture of virtual and reality, the main scene is still true, and the virtual part is just a small amount of additional information. In the technical solution of the embodiment, the virtual and the reality are perfectly integrated, the user can set the ratio of the virtual reality to the reality, and the VR imaging glasses 550 can actually feel the existence of the real element, thereby providing the user with a A new virtual and realistic experience.

In summary, the present application provides a method and system for seamlessly integrating virtual reality with reality, allowing users to completely immerse themselves between virtual reality and reality. Moreover, the method and system can superimpose and superimpose the realistic elements of the surrounding environment of the user on the virtual content of the virtual reality system, so that the user can walk the activity and complete certain actions while wearing the virtual reality helmet. Give users a new VR experience. While seeing the virtual world, the user can adjust the amount of the elements and the virtual degree displayed in the virtual world from the visual to the auditory, and the virtual reality is added to the pure virtual experience, so that the user is immersed in the virtual reality. A new world of virtual and reality integration. The method and system of the embodiment of the present application break the boundary between the traditional VR and the reality, and the security fusion element is preferentially integrated in the video fusion, which can fully ensure the security of the user using the helmet to walk and move.

In the technical solution: 1. The present application integrates the security reality element in the virtual three-dimensional scene, so that the user can sense the external real world by wearing the VR helmet, and realize safe walking and activities; 2. The present application sets the realistic element rendering. Set the interactive device, the user selects the actual element to render the virtual/real degree, realizes the rendering according to the user's will, and provides the best experience; 3. This application realizes the capture and modeling of the realistic 3D panoramic image through several realistic capture modules. Several realistic capture modules covering each other simultaneously capture realistic three-dimensional images and stitch together to form a reality three Dimensional panoramic image; 4, the application also sets the interactive device of the real element fusion setting, by setting a number of realistic element number of gears for the user to select, to achieve the best experience according to the user's willingness; 5, in addition to preferential extraction of security in this application The realistic element also sorts the matching degree of other realistic elements to solve the fusion matching problem between the real element and the virtual three-dimensional scene.

FIG. 6 is a schematic diagram of the hardware structure of the electronic device 600 of the virtual and reality fusion method provided by the embodiment of the present application. As shown in FIG. 6, the electronic device 600 includes:

One or more processors 610 and memory 620, one processor 610 is taken as an example in FIG.

The processor 610 and the memory 620 may be connected by a bus or other means, as exemplified by a bus connection in FIG.

The memory 620 is used as a non-volatile computer readable storage medium, and can be used for storing a non-volatile software program, a non-volatile computer executable program, and a module, as in the virtual and reality fusion method in the embodiment of the present application. Program instructions/modules (eg, the fusion matching module, the realistic element extraction module, and the image recognition module shown in FIG. 1). The processor 610 executes various functional applications and data processing of the terminal or the server by executing non-volatile software programs, instructions, and modules stored in the memory 620, that is, implementing the virtual and reality fusion method in the foregoing method embodiments.

The memory 620 can include a storage program area and an storage data area, wherein the storage program area can store an operating system, an application required for at least one function; the storage data area can store data created according to the use of the virtual and reality fusion system, and the like. Moreover, memory 620 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 620 can optionally include memory remotely located relative to processor 610, which can be connected to the virtual reality fusion system via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, perform a virtual and reality fusion method in any of the above method embodiments, for example, performing the above described diagram Method 403 to step 410 in 4, implementing FIG. 1 The functions of the fusion matching module, the realistic element extraction module, and the image recognition module.

The above products can perform the methods provided by the embodiments of the present application, and have the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.

The electronic device of the embodiment of the present application exists in various forms, such as other electronic devices having data interaction functions.

The embodiment of the present application provides a non-transitory computer readable storage medium storing computer-executable instructions that are executed by one or more processors, such as in FIG. The processor 610 is configured to enable the one or more processors to perform the virtual and reality fusion method in any of the foregoing method embodiments, for example, to perform the method steps 403 to 410 in FIG. 4 described above, to implement FIG. The functions of the fusion matching module, the realistic element extraction module, and the image recognition module.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, ie may be located A place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Through the description of the above embodiments, those skilled in the art can clearly understand that the various embodiments can be implemented by means of software plus a general hardware platform, and of course, by hardware. A person skilled in the art can understand that all or part of the process of implementing the above embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; in the idea of the present application, the technical features in the above embodiments or different embodiments may also be combined. The steps can be implemented in any order and there is the present application as described above Many other variations of the various aspects are not provided in the details for the sake of brevity; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that The technical solutions are modified or equivalently replaced with some of the technical features; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present application.

Claims

A virtual and reality fusion method, which comprises:

Obtain a realistic three-dimensional panoramic image of the real environment around the user;

Rendering the realistic three-dimensional panoramic image according to a rendering virtual degree selected by the user;

The rendered result is output to the virtual reality display module.
The method according to claim 1, wherein the rendering of the realistic three-dimensional panoramic image according to a rendering virtual degree selected by a user comprises:

Identifying and extracting realistic elements in the realistic three-dimensional panoramic image;

Obtaining a first rendering virtual degree selected by the user; the first rendering virtual degree is used to indicate a proportion of real elements that the user wishes to retain;

The real elements in the realistic three-dimensional panoramic image are processed according to the acquired first rendering virtual degree.
The method according to claim 2, wherein the real-life element comprises a security reality element; the processing the real-world element in the real-world three-dimensional panoramic image according to the acquired first rendering virtual degree, comprising:

And retaining at least a part of the security reality elements in the realistic three-dimensional panoramic image when the first rendering virtual degree indicates that the proportion of the real elements that the user wishes to retain is non-zero.
The method according to claim 1, wherein the rendering of the realistic three-dimensional panoramic image according to a rendering virtual degree selected by a user comprises:

Identifying and extracting realistic elements in the realistic three-dimensional panoramic image;

Obtaining a second rendering virtual degree selected by the user; the second rendering virtual degree is used to indicate a rendering degree of each real element;

And rendering respective real elements in the realistic three-dimensional panoramic image according to the acquired second rendering virtual degree.
The method according to any one of claims 2-4, characterized in that the sound rendering corresponding to the virtual three-dimensional scene is determined according to the security real element and the other real elements.
The method according to claim 5, wherein the acquiring a realistic three-dimensional panoramic image of a real environment around the user comprises:

Obtain images taken by different image acquisition devices from different angles, and use the three-dimensional modeling module to perform three-dimensional modeling on the acquired images.
The method of claim 6, wherein the first rendering virtual level or the second rendering virtual level is determined by a ratio selected by a user on the interactive interface.
A virtual reality integration system, including a virtual reality system, is used for presenting a virtual three-dimensional scene for a user, and is characterized in that it further comprises a reality fusion module, wherein

The reality fusion module is configured to acquire a realistic three-dimensional panoramic image of a real environment around the user;

The virtual reality system is configured to render the realistic three-dimensional panoramic image according to a rendering virtual degree selected by a user;

The virtual reality system is further configured to output the rendered result to the virtual reality display module.
The system according to claim 8, wherein the virtual reality system is further configured to identify and extract a realistic element in the realistic three-dimensional panoramic image; and to view the realistic three-dimensional panoramic image according to a rendering virtual degree selected by a user. And a first rendering virtual degree used by the user to obtain a user-selected first rendering virtual degree, the first rendering virtual level is used to indicate a proportion of a real-life element that the user wishes to retain; and is configured to use the acquired first rendering virtual degree to The realistic elements in the realistic three-dimensional panoramic image are processed.
The system according to claim 9, wherein the real element comprises a security reality element; the virtual reality system is configured to indicate, at the first rendering virtual level, that a proportion of real elements that the user wishes to retain is non-zero. At least a portion of the security reality elements in the realistic three-dimensional panoramic image are retained.
The system according to claim 8, wherein the virtual reality system is further configured to identify and extract a realistic element in the realistic three-dimensional panoramic image; and to view the realistic three-dimensional panoramic image according to a rendering virtual degree selected by a user. When the image is rendered, the second rendering virtual degree selected by the user is obtained, where the second rendering virtual level is used to indicate the rendering degree of each real element; and the acquired second rendering virtual degree is used in the real three-dimensional panoramic image. The various realistic elements are rendered.
An electronic device, comprising: a processor module, an interaction module connected by the processor module, and a virtual reality display module;

The processor module is configured to receive an image of a user's surrounding environment captured from multiple angles and acquire a realistic three-dimensional panoramic image according to the image, and then render the realistic three-dimensional panoramic image according to a rendering virtual degree selected by the user in the interaction module. ;

The processor module is further configured to output the rendered result to the virtual reality display module.
The electronic device according to claim 12, wherein the electronic device is a virtual reality wearing device, and further comprises a helmet and a reality capturing module.

The reality capture module is configured to capture images of a user's surroundings from multiple angles.
The electronic device according to claim 13, wherein the interaction module comprises a quantity adjustment interaction device, wherein the processor module is further configured to identify and extract a real element in the realistic three-dimensional panoramic image; Obtaining, by the quantity adjustment interaction device, a first rendering virtual degree selected by the user, the first rendering virtual degree is used to indicate a proportion of a real element that the user wishes to retain; and is configured to target the reality according to the acquired first rendering virtual degree The realistic elements in the 3D panoramic image are processed.
The electronic device according to claim 14, wherein the real element comprises a security reality element, wherein the processor module is further configured to indicate, at the first rendering virtual level, a realistic element that the user wishes to retain When the ratio is non-zero, at least a part of the security reality elements in the realistic three-dimensional panoramic image are retained.
The electronic device according to claim 13, wherein the interaction module further comprises a rendering virtuality interaction device, wherein the processor module is further configured to identify and extract a realistic element in the realistic three-dimensional panoramic image Obtaining, by the rendering virtuality interaction device, a second rendering virtual degree selected by the user, the second rendering virtual degree is used to indicate a rendering degree of each real element, and the real three-dimensionality is obtained according to the acquired second rendering virtual degree Each realistic element in the panoramic image is rendered.
The electronic device according to any one of claims 14-16, wherein the processor module is configured to determine a sound rendering corresponding to the virtual three-dimensional scene according to the security real element and the other real elements.
The electronic device according to any one of claims 14-16, wherein the reality capture module comprises a plurality of cameras, and the plurality of cameras are disposed in different ones of the helmets Orientation.
The electronic device according to any one of claims 14-16, wherein the interaction module comprises a knob and an analog to digital conversion device; the analog to digital conversion device is connected to the knob and the processor module, It is used to generate a corresponding digital signal according to the rotation angle of the knob and output it to the processor module.
An electronic device, comprising:

At least one processor; and,

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any of claims 1-7 method.
A non-readable readable storage medium, characterized in that the readable storage medium stores executable instructions for causing a processor to perform the method of any one of claims 1-7 .
A program product, comprising: a program stored on a non-volatile readable storage medium, the program comprising program instructions, when the program instructions are executed by a processor module, The method of any of claims 1-7 is performed when the processor module is described.