CN113450439A

CN113450439A - Virtual-real fusion method, device and system

Info

Publication number: CN113450439A
Application number: CN202010225385.7A
Authority: CN
Inventors: 李江伟; 罗巍; 何峰
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-03-26
Filing date: 2020-03-26
Publication date: 2021-09-28

Abstract

The embodiment of the invention discloses a virtual-real fusion method, a device and a system, which comprise an information acquisition module, a first space positioning module, a scene understanding module, a position determining module, an electronic device of a rendering module and a display module and a server with a second space positioning module, wherein the information acquisition module acquires environment information; the first space positioning module sends environment information; the second space positioning module determines absolute position information and a corresponding 3D map according to the environment information; the scene understanding module identifies the environment of the electronic equipment according to the environment information and the 3D map; the position determining module determines a first position of the digital content on the 3D map and a second position of the electronic equipment on the 3D map according to the absolute position information and the identification result; the rendering module renders the digital content according to the first position, the identification result and the second position; the display module displays the rendered digital content. The embodiment of the invention can realize seamless fusion of the digital world and the physical world.

Description

Virtual-real fusion method, device and system

Technical Field

The invention relates to the technical field of Augmented Reality (AR), in particular to a virtual-real fusion method, equipment and system.

Background

The AR technology is a technology for skillfully fusing virtual information and a real world, and a plurality of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and the like are widely applied, and virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer is applied to the real world after analog simulation, and the two kinds of information complement each other, so that the 'enhancement' of the real world is realized. With the continuous development of computer vision technology, the application range of the AR technology is wider and wider, but the existing AR technology cannot realize the seamless fusion of the digital world and the physical world.

Disclosure of Invention

The embodiment of the invention discloses a method, equipment and a system for fusing virtuality and reality, which are used for realizing seamless fusion of a digital world and a physical world.

The first aspect discloses a virtual-real fusion system, which may include an electronic device and a server, wherein the electronic device may include an information acquisition module, a first spatial localization module, a scene understanding module, a position determination module, a rendering module, and a display module, and the server may include a second spatial localization module, the information acquisition module is configured to acquire environmental information; the first space positioning module is used for sending the environment information to the server; the second spatial positioning module is used for determining the absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and sending the absolute position information and the 3D map corresponding to the absolute position information to the electronic equipment; the scene understanding module is used for identifying the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information; the position determining module is used for determining the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result; the rendering module is used for rendering the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information; and the display module is used for displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the information of the current scene can be obtained through the scene understanding module in the electronic equipment, and the reasonable placement position of the digital content in the whole 3D map can be calculated, so that the 3D digital content is rendered at a proper position, and the seamless fusion of the digital world and the physical world can be realized. In addition, the server is used for positioning, power consumption required by the electronic equipment can be reduced, and meanwhile, the server can be used for positioning all the electronic equipment, 3D map construction and 3D map storage are not required to be respectively carried out on each piece of electronic equipment, so that the consumption of electronic equipment resources can be reduced, and the resources of the electronic equipment can be saved.

As a possible implementation manner, the first spatial location module is further configured to determine an environmental status of the electronic device according to the collection device of the environmental information, and send the environmental status to the server, where the environmental status includes an outdoor environment or an indoor environment; the second spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and the second spatial positioning module comprises: and determining the absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the environment of the electronic equipment can be determined to be an outdoor environment or an indoor environment through the acquisition device of the environment information, so that different 3D map search ranges are adopted for positioning, and the positioning precision and accuracy can be improved.

As a possible implementation manner, the determining, by the first spatial location module, the environmental state of the electronic device according to the collection device of the environmental information includes: determining that the environmental state of the electronic equipment is an indoor environment under the condition that the acquisition device of the environmental information comprises a first preset device; and under the condition that the acquisition device of the environmental information comprises a second preset device, determining that the environmental state of the electronic equipment is an outdoor environment.

As a possible implementation, the environment information may include an image, and the identifying, by the scene understanding module, the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute location information includes one or more of: identifying the scene of the electronic equipment according to the 3D map corresponding to the image and the absolute position information; performing target detection on the 3D map corresponding to the image and the absolute position information; performing text detection on the 3D map corresponding to the image and the absolute position information; performing sign detection on the 3D map corresponding to the image and the absolute position information; performing material identification according to the environment information; carrying out illumination identification according to the environment information; calculating scene depth according to the image and the 3D map corresponding to the absolute position information; carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information; carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information; performing plane estimation on the 3D map corresponding to the image and the absolute position information; carrying out face recognition on the image; performing emotion recognition on the image; and tracking the target according to the image and the 3D map corresponding to the absolute position information.

According to the virtual-real fusion system disclosed by the embodiment of the invention, information such as a target, a foreground, a background, a spatial structure and the like contained in the current scene can be obtained through the scene understanding module, and the reasonable placement position and the mutual shielding relation of digital content in the real world can be calculated, so that the seamless fusion of the digital world and the physical world is realized. Meanwhile, high-fidelity 3D rendering based on the global illumination technology can be achieved by means of a 3D graphic rendering engine, illumination estimation, material estimation and a digital content geometric structure. The foreground and background analysis technology for scene understanding can avoid the digital content from being superimposed on the foreground by mistake, and the motion information for scene understanding can also ensure the continuity of rendering effect.

As a possible implementation, the server may further include a user generated content UGC module, and the electronic device may further include a content acquisition module; a UGC module for generating the digital content; and the content acquisition module is used for acquiring the digital content from the server.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the digital content can be shared in the server, and in addition, the privacy of the digital content can also be realized through privacy setting.

As a possible implementation manner, the electronic device may further include a first codec module, and the server may further include a second codec module; the first coding and decoding module is used for coding the environment information; the first spatial location module sending the environmental information to the server comprises: sending the encoded environment information to the server; and the second coding and decoding module is used for decoding the coded environment information to obtain the environment information.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the transmitted environment information is the information acquired by various sensors after being coded, so that the reliability of information transmission can be improved.

As a possible implementation manner, the second encoding and decoding module is further configured to encode a 3D map corresponding to the absolute position information; the second spatial positioning module sending the absolute position information and the 3D map corresponding to the absolute position information to the electronic device includes: sending the absolute position information and a 3D map corresponding to the encoded absolute position information to the electronic equipment; and the first coding and decoding module is further configured to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, so that the reliability of information transmission can be improved.

As a possible implementation, the second codec module is further configured to encode the digital content; the content obtaining module obtaining the digital content from the server includes: acquiring encoded digital content from the server; the first coding and decoding module is further configured to decode the coded digital content to obtain the digital content.

In the virtual-real fusion system disclosed by the embodiment of the invention, the transmitted digital content is the coded digital content, so that the reliability of information transmission can be improved.

As a possible implementation, the server may further include a training module; the training module is used for training a model according to training data and sending the trained model to the electronic equipment; the scene understanding module identifies the environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, and the scene understanding module includes: and identifying the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

The virtual-real fusion system disclosed by the embodiment of the invention can realize the sharing of the model by training the model used by the electronic equipment by the server, and can also reduce the power consumption and resources required by the training of the model of the electronic equipment.

As a possible implementation, the server further comprises a modeling module; the modeling module is used for acquiring modeling data and constructing a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

According to the virtual-real fusion system disclosed by the embodiment of the invention, the 3D map is constructed by the server, the sharing of the 3D map can be realized, and in addition, the power consumption and resources required by the 3D map of the electronic equipment can be reduced.

A second aspect discloses an electronic device, comprising an information acquisition module, a spatial localization module, a scene understanding module, a position determination module, a rendering module, and a display module, wherein: the information acquisition module is used for acquiring environmental information; the spatial positioning module is configured to send the environment information to the server, receive absolute position information of the electronic device from the server and a 3D map corresponding to the absolute position information, where the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map; the scene understanding module is used for identifying the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information; the position determining module is used for determining the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result; the rendering module is configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result, and the position of the electronic device in the 3D map corresponding to the absolute position information; and the display module is used for displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

According to the electronic equipment disclosed by the embodiment of the invention, the information of the current scene can be obtained through the scene understanding module in the electronic equipment, and the reasonable placement position of the digital content in the whole 3D map can be calculated, so that the 3D digital content can be rendered at a proper position, and the seamless fusion of the digital world and the physical world can be realized. In addition, the server is used for positioning, power consumption required by the electronic equipment can be reduced, and meanwhile, the server can be used for positioning all the electronic equipment, 3D map construction and 3D map storage are not required to be respectively carried out on each piece of electronic equipment, so that the consumption of electronic equipment resources can be reduced, and the resources of the electronic equipment can be saved.

As a possible implementation manner, the spatial location module is further configured to determine an environmental status of the electronic apparatus according to the collection device of the environmental information, and send the environmental status to the server, where the environmental status includes an outdoor environment or an indoor environment; the determining, by the server, the absolute position information and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes: and the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information, the environment state and the stored 3D map.

According to the electronic equipment disclosed by the embodiment of the invention, the environment of the electronic equipment can be determined to be an outdoor environment or an indoor environment through the acquisition device of the environment information, so that different 3D map search ranges are adopted for positioning, and the positioning precision and accuracy can be improved.

As a possible implementation manner, the determining, by the spatial location module according to the environmental information collecting device, the environmental state of the electronic device includes: determining that the environmental state of the electronic equipment is an indoor environment under the condition that the acquisition device of the environmental information comprises a first preset device; and under the condition that the acquisition device of the environmental information comprises a second preset device, determining that the environmental state of the electronic equipment is an outdoor environment.

As a possible implementation manner, the environment information includes an image, and the identifying, by the scene understanding module, the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of: identifying the scene of the electronic equipment according to the 3D map corresponding to the image and the absolute position information; performing target detection on the 3D map corresponding to the image and the absolute position information; performing text detection on the 3D map corresponding to the image and the absolute position information; performing sign detection on the 3D map corresponding to the image and the absolute position information; performing material identification according to the environment information; carrying out illumination identification according to the environment information; calculating scene depth according to the image and the 3D map corresponding to the absolute position information; carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information; carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information; performing plane estimation on the 3D map corresponding to the image and the absolute position information; carrying out face recognition on the image; performing emotion recognition on the image; and tracking the target according to the image and the 3D map corresponding to the absolute position information.

According to the electronic equipment disclosed by the embodiment of the invention, the information of the target, the foreground, the background, the spatial structure and the like contained in the current scene can be obtained through the scene understanding module, and the reasonable placement position and the mutual shielding relation of the digital content in the real world can be calculated, so that the seamless fusion of the digital world and the physical world is realized. Meanwhile, high-fidelity 3D rendering based on the global illumination technology can be achieved by means of a 3D graphic rendering engine, illumination estimation, material estimation and a digital content geometric structure. The foreground and background analysis technology for scene understanding can avoid the digital content from being superimposed on the foreground by mistake, and the motion information for scene understanding can also ensure the continuity of rendering effect.

As a possible implementation, the electronic device further includes a content obtaining module, wherein: the content obtaining module is used for obtaining the digital content from the server.

According to the electronic equipment disclosed by the embodiment of the invention, the digital content can be shared in the server, and in addition, the privacy of the digital content can also be realized through privacy setting.

As a possible implementation, the electronic device further includes a coding and decoding module, wherein: the coding and decoding module is used for coding the environment information; the sending, by the spatial location module, the environmental information to the server includes: and sending the coded environment information to the server.

According to the electronic equipment disclosed by the embodiment of the invention, the transmitted environment information is the information acquired by various sensors after being coded, so that the reliability of information transmission can be improved.

As a possible implementation, the receiving, by the spatial location module, the absolute location information of the electronic device from the server and the 3D map corresponding to the absolute location information includes: receiving absolute position information of the electronic equipment from the server and a 3D map corresponding to the encoded absolute position information; and the coding and decoding module is further configured to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.

According to the electronic device disclosed by the embodiment of the invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, so that the reliability of information transmission can be improved.

As a possible implementation, the content obtaining module obtaining the digital content from the server includes: acquiring encoded digital content from the server; the coding and decoding module is further configured to decode the coded digital content to obtain the digital content.

According to the electronic equipment disclosed by the embodiment of the invention, the transmitted digital content is the coded digital content, so that the reliability of information transmission can be improved.

A third aspect discloses a server comprising a spatial localization module, wherein: the spatial positioning module is configured to receive environment information from an electronic device, determine, according to the environment information and a stored 3D map, absolute position information of the electronic device and a 3D map corresponding to the absolute position information, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, where the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify an environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, determine, according to the absolute position information and an identification result, a position of digital content in the 3D map corresponding to the absolute position information, and a position of the electronic device in the 3D map corresponding to the absolute position information, and determine, according to the position of the digital content in the 3D map corresponding to the absolute position information, Rendering the digital content according to the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information, and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

According to the server disclosed by the embodiment of the invention, the electronic equipment is positioned through the environment information of the electronic equipment, the information of the current scene can be obtained through the scene understanding module in the electronic equipment, and the reasonable placement position of the digital content in the whole 3D map can be calculated so as to render the 3D digital content at the proper position, so that the seamless fusion of the digital world and the physical world can be realized. In addition, the server is used for positioning, power consumption required by the electronic equipment can be reduced, and meanwhile, the server can be used for positioning all the electronic equipment, 3D map construction and 3D map storage are not required to be respectively carried out on each piece of electronic equipment, so that the consumption of electronic equipment resources can be reduced, and the resources of the electronic equipment can be saved.

As a possible implementation, the spatial location module is further configured to receive an environmental status from the electronic device, where the environmental status includes an outdoor environment or an indoor environment; the spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and the spatial positioning module comprises: and determining the absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

The server disclosed by the embodiment of the invention can determine whether the environment of the electronic equipment is an outdoor environment or an indoor environment through the environment state, so that different 3D map search ranges are adopted for positioning, and the positioning precision and accuracy can be improved.

As a possible implementation manner, the environment information includes an image, and the identifying, by the electronic device, the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of: identifying the scene of the electronic equipment according to the 3D map corresponding to the image and the absolute position information; performing target detection on the 3D map corresponding to the image and the absolute position information; performing text detection on the 3D map corresponding to the image and the absolute position information; performing sign detection on the 3D map corresponding to the image and the absolute position information; performing material identification according to the environment information; carrying out illumination identification according to the environment information; calculating scene depth according to the image and the 3D map corresponding to the absolute position information; carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information; carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information; performing plane estimation on the 3D map corresponding to the image and the absolute position information; carrying out face recognition on the image; performing emotion recognition on the image; and tracking the target according to the image and the 3D map corresponding to the absolute position information.

According to the server disclosed by the embodiment of the invention, the electronic equipment can obtain information such as a target, a foreground, a background, a space structure and the like contained in the current scene through the scene understanding module, and can calculate the reasonable placement position and the mutual shielding relation of the digital content in the real world, so that the seamless fusion of the digital world and the physical world is realized. Meanwhile, high-fidelity 3D rendering based on the global illumination technology can be achieved by means of a 3D graphic rendering engine, illumination estimation, material estimation and a digital content geometric structure. The foreground and background analysis technology for scene understanding can avoid the digital content from being superimposed on the foreground by mistake, and the motion information for scene understanding can also ensure the continuity of rendering effect.

As a possible implementation, the server further comprises a user generated content UGC module, wherein: the UGC module is used for generating the digital content and sending the digital content to the electronic equipment.

According to the server disclosed by the embodiment of the invention, the digital content can be shared in the server, and in addition, the privacy of the digital content can also be realized through privacy setting.

As a possible implementation, the server further includes a coding and decoding module, wherein: the spatial location module receiving environmental information from an electronic device includes: receiving environment information coded by the electronic equipment; and the coding and decoding module is used for decoding the coded environment information to obtain the environment information.

According to the server disclosed by the embodiment of the invention, the transmitted environment information is the information acquired by various sensors after being coded, so that the reliability of information transmission can be improved.

As a possible implementation manner, the encoding/decoding module is further configured to encode a 3D map corresponding to the absolute location information; the sending, by the spatial location module, the absolute location information and the 3D map corresponding to the absolute location information to the electronic device includes: and sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic equipment.

According to the server disclosed by the embodiment of the invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, so that the reliability of information transmission can be improved.

As a possible implementation, the encoding/decoding module is further configured to encode the digital content; the UGC module transmitting the digital content to the electronic device comprises: and transmitting the coded digital content to the electronic equipment.

According to the server disclosed by the embodiment of the invention, the transmitted digital content is the coded digital content, so that the reliability of information transmission can be improved.

As a possible implementation, the server further comprises a training module, wherein: the training module is used for training a model according to training data and sending the trained model to the electronic equipment; the identifying, by the electronic device, the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information includes: and identifying the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

According to the server disclosed by the embodiment of the invention, the model used by the electronic equipment is trained by the server, so that the sharing of the model can be realized, and in addition, the power consumption and resources required by the electronic equipment for training the model can be reduced.

As a possible implementation, the server further comprises a modeling module, wherein: the modeling module is used for acquiring modeling data and constructing a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

According to the server disclosed by the embodiment of the invention, the server constructs the 3D map, so that the sharing of the 3D map can be realized, and in addition, the power consumption and resources required by the 3D map of the electronic equipment can be reduced.

The fourth aspect discloses a virtual-real fusion method, wherein the electronic equipment collects environment information and sends the environment information to a server; the server determines absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic equipment; the electronic equipment identifies the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information, determines the position of the digital content in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result, renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information, and displays the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the information of the current scene can be obtained by identifying the environment of the electronic equipment, and the reasonable placement position of the digital content in the whole 3D map can be calculated, so that the 3D digital content can be rendered at a proper position, and the seamless fusion of the digital world and the physical world can be realized. In addition, the server is used for positioning, power consumption required by the electronic equipment can be reduced, and meanwhile, the server can be used for positioning all the electronic equipment, 3D map construction and 3D map storage are not required to be respectively carried out on each piece of electronic equipment, so that the consumption of electronic equipment resources can be reduced, and the resources of the electronic equipment can be saved.

As a possible implementation, the method further comprises: the electronic equipment determines the environmental state of the electronic equipment according to the acquisition device of the environmental information and sends the environmental state to the server, wherein the environmental state comprises an outdoor environment or an indoor environment; the server determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes: and the server determines the absolute position information of the electronic equipment and the 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the environment of the electronic equipment can be determined to be an outdoor environment or an indoor environment through the acquisition device of the environment information, so that different 3D map search ranges are adopted for positioning, and the positioning precision and accuracy can be improved.

As a possible implementation manner, the determining, by the electronic device according to the environmental information collecting device, the environmental state of the electronic device includes: under the condition that the acquisition device of the environmental information comprises a first preset device, the electronic equipment determines that the environmental state where the electronic equipment is located is an indoor environment; and under the condition that the acquisition device of the environmental information comprises a second preset device, the electronic equipment determines that the environmental state where the electronic equipment is located is an outdoor environment.

As a possible implementation manner, the environment information includes an image, and the identifying, by the electronic device, the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of: the electronic equipment identifies the scene where the electronic equipment is located according to the image and the 3D map corresponding to the absolute position information; the electronic equipment performs target detection on the 3D map corresponding to the image and the absolute position information; the electronic equipment performs text detection on the 3D map corresponding to the image and the absolute position information; the electronic equipment carries out mark detection on the 3D map corresponding to the image and the absolute position information; the electronic equipment identifies the material quality according to the environment information; the electronic equipment performs illumination identification according to the environment information; the electronic equipment carries out scene depth calculation according to the image and the 3D map corresponding to the absolute position information; the electronic equipment carries out shielding detection on the 3D map corresponding to the image and the absolute position information; the electronic equipment carries out motion foreground detection on the 3D map corresponding to the image and the absolute position information; the electronic equipment carries out plane estimation on the 3D map corresponding to the image and the absolute position information; the electronic equipment carries out face recognition on the image; the electronic equipment carries out emotion recognition on the image; and the electronic equipment tracks the target according to the image and the 3D map corresponding to the absolute position information.

The virtual-real fusion method disclosed by the embodiment of the invention can obtain the information of the target, the foreground, the background, the spatial structure and the like contained in the current scene, and can calculate the reasonable placement position and the mutual shielding relation of the digital content in the real world, thereby realizing the seamless fusion of the digital world and the physical world. Meanwhile, high-fidelity 3D rendering based on the global illumination technology can be achieved by means of a 3D graphic rendering engine, illumination estimation, material estimation and a digital content geometric structure. The foreground and background analysis technology for scene understanding can avoid the digital content from being superimposed on the foreground by mistake, and the motion information for scene understanding can also ensure the continuity of rendering effect.

As a possible implementation, the method further comprises: the server generates the digital content; the electronic device obtains the digital content from the server.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the digital content can be shared in the server, and in addition, the privacy of the digital content can also be realized through privacy setting.

As a possible implementation, the method further comprises: the electronic equipment encodes the environment information; the electronic device sending the environmental information to the server includes: the electronic equipment sends the coded environment information to the server; the method further comprises the following steps: and the server decodes the coded environment information to obtain the environment information.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the transmitted environment information is the information acquired by various sensors after being coded, so that the reliability of information transmission can be improved.

As a possible implementation, the method further comprises: the server encodes the 3D map corresponding to the absolute position information; the step of sending, by the server, the absolute position information and the 3D map corresponding to the absolute position information to the electronic device includes: the server sends the absolute position information and a 3D map corresponding to the encoded absolute position information to the electronic equipment; the method further comprises the following steps: and the electronic equipment decodes the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, so that the reliability of information transmission can be improved.

As a possible implementation, the method further comprises: the server encodes the digital content; the electronic device obtaining the digital content from the server comprises: the electronic equipment acquires the coded digital content from the server; the method further comprises the following steps: and the electronic equipment decodes the coded digital content to obtain the digital content.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the transmitted digital content is the coded digital content, so that the reliability of information transmission can be improved.

As a possible implementation, the method further comprises: the server trains a model according to the training data and sends the trained model to the electronic equipment; the identifying, by the electronic device, the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information includes: and the electronic equipment identifies the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the model used by the electronic equipment is trained by the server, so that the sharing of the model can be realized, and in addition, the power consumption and resources required by the electronic equipment for training the model can be reduced.

As a possible implementation, the method further comprises: the server acquires modeling data, and constructs a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

According to the virtual-real fusion method disclosed by the embodiment of the invention, the server constructs the 3D map, so that the sharing of the 3D map can be realized, and in addition, the power consumption and resources required by the 3D map of the electronic equipment can be reduced.

The fifth aspect discloses a virtual-real fusion method, which comprises the following steps: collecting environmental information; sending the environment information to the server, receiving absolute position information of the electronic equipment from the server and a 3D map corresponding to the absolute position information, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map; identifying the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information; determining the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result; rendering the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information; and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

As a possible implementation, the method further comprises: determining the environment state of the electronic device according to the acquisition device of the environment information, and sending the environment state to the server, wherein the environment state comprises an outdoor environment or an indoor environment; the determining, by the server, the absolute position information and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes: and the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information, the environment state and the stored 3D map.

As a possible implementation manner, the determining, by the acquisition device according to the environment information, the environment state where the electronic device is located includes: determining that the environmental state of the electronic equipment is an indoor environment under the condition that the acquisition device of the environmental information comprises a first preset device; and under the condition that the acquisition device of the environmental information comprises a second preset device, determining that the environmental state of the electronic equipment is an outdoor environment.

As a possible implementation, the environment information includes an image, and the identifying the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of: identifying the scene of the electronic equipment according to the 3D map corresponding to the image and the absolute position information; performing target detection on the 3D map corresponding to the image and the absolute position information; performing text detection on the 3D map corresponding to the image and the absolute position information; performing sign detection on the 3D map corresponding to the image and the absolute position information; performing material identification according to the environment information; carrying out illumination identification according to the environment information; calculating scene depth according to the image and the 3D map corresponding to the absolute position information; carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information; carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information; performing plane estimation on the 3D map corresponding to the image and the absolute position information; carrying out face recognition on the image; performing emotion recognition on the image; and tracking the target according to the image and the 3D map corresponding to the absolute position information.

As a possible implementation, the method further comprises: the digital content is obtained from the server.

As a possible implementation, the method further comprises: encoding the environment information; the sending the environmental information to the server includes: and sending the coded environment information to the server.

As a possible implementation manner, the receiving the absolute position information of the electronic device from the server and the 3D map corresponding to the absolute position information includes: receiving absolute position information of the electronic equipment from the server and a 3D map corresponding to the encoded absolute position information; the method further comprises the following steps: and decoding the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.

As a possible implementation, the obtaining the digital content from the server includes: acquiring encoded digital content from the server; the method further comprises the following steps: and decoding the coded digital content to obtain the digital content.

A sixth aspect discloses a virtual-real fusion method, including: receiving environmental information from the electronic device; determining absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information and the stored 3D map; sending the absolute position information and a 3D map corresponding to the absolute position information to the electronic device, where the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to recognize the environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, determine the position of the digital content in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result, determine the position of the electronic device in the 3D map corresponding to the absolute position information, render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information The digital content of (1).

According to the virtual-real fusion method disclosed by the embodiment of the invention, the electronic equipment is positioned through the environment information of the electronic equipment, the information of the current scene can be obtained through identifying the environment where the electronic equipment is located, and the reasonable placement position of the digital content in the whole 3D map can be calculated so as to render the 3D digital content at the proper position, so that the seamless fusion of the digital world and the physical world can be realized. In addition, the server is used for positioning, power consumption required by the electronic equipment can be reduced, and meanwhile, the server can be used for positioning all the electronic equipment, 3D map construction and 3D map storage are not required to be respectively carried out on each piece of electronic equipment, so that the consumption of electronic equipment resources can be reduced, and the resources of the electronic equipment can be saved.

As a possible implementation, the method further comprises: receiving an environmental status from the electronic device, the environmental status comprising an outdoor environment or an indoor environment; the determining of the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes: and determining the absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

According to the virtual-real fusion method disclosed by the embodiment of the invention, whether the environment of the electronic equipment is an outdoor environment or an indoor environment can be determined through the environment state, so that different 3D map search ranges are adopted for positioning, and the positioning precision and accuracy can be improved.

As a possible implementation, the method further comprises: generating the digital content; transmitting the digital content to the electronic device.

As a possible implementation, the receiving the environment information from the electronic device includes: receiving environment information coded by the electronic equipment; the method further comprises the following steps: and decoding the coded environment information to obtain the environment information.

As a possible implementation, the method further comprises: encoding the 3D map corresponding to the absolute position information; the sending the absolute position information and the 3D map corresponding to the absolute position information to the electronic device includes: and sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic equipment.

As a possible implementation, the method further comprises: encoding the digital content; the sending the digital content to the electronic device comprises: and transmitting the coded digital content to the electronic equipment.

As a possible implementation, the method further comprises: training the model according to the training data; sending the trained model to the electronic equipment; the identifying the environment of the electronic device according to the 3D map corresponding to the environment information and the absolute position information includes: and identifying the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

As a possible implementation, the method further comprises: acquiring modeling data, wherein the modeling data comprises images and/or videos; and constructing a 3D map according to the modeling data.

Drawings

Fig. 1 is a schematic structural diagram of a virtual-real fusion system disclosed in the embodiment of the present invention;

FIG. 2 is a schematic structural diagram of another virtual-real fusion system disclosed in the embodiment of the present invention;

FIG. 3 is a schematic diagram of a virtual-real fusion disclosed in the embodiments of the present invention;

FIG. 4 is a schematic diagram of a virtual-real fusion 3D navigation system according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure;

FIG. 6 is a schematic structural diagram of another electronic device disclosed in the embodiments of the present invention;

FIG. 7 is a schematic structural diagram of a server according to an embodiment of the present invention;

fig. 8 is a schematic flow chart of a virtual-real fusion method disclosed in the embodiment of the present invention.

Detailed Description

The embodiment of the invention discloses a method, equipment and a system for fusing virtuality and reality, which are used for realizing seamless fusion of a digital world and a physical world. The following are detailed below.

In order to better understand the virtual-real fusion method, device and system disclosed in the embodiments of the present invention, an application scenario of the embodiments of the present invention is described below. The AR technology is a technology for skillfully fusing virtual information and a real world, and a plurality of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and the like are widely applied, and virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer is applied to the real world after analog simulation, and the two kinds of information complement each other, so that the 'enhancement' of the real world is realized. With the continuous development of electronic technology and computer vision technology, some mature AR applications, such as the ARKit platform introduced by apple inc, the arcre platform introduced by google (google), etc., are gradually appearing on electronic devices. These AR applications may perform estimation of electronic device position, estimation of electronic device pose, planar estimation, illumination estimation, and the like. The above-described AR applications have limited ability to understand the 3D physical world, and a full understanding of the physical world is an important basis for implementing AR technology with a high degree of realism. Seamless nesting of AR content can only be achieved by identifying the structure, semantics, 3-dimensional (3D) information, lighting, and material of the physical world.

At present, a digital grid model of a city can be constructed according to a high-resolution two-dimensional (2D) satellite image of ground data to generate map features. When electronic device capture is enabled, the system matches the visual information with the map features. It is therefore able to determine the absolute position at which the electronic device is located. However, the above situation only supports outdoor positioning of a large scene, and does not have indoor positioning capability. In addition, the technical scheme is limited to satellite images, the outdoor positioning precision is insufficient, the error reaches more than 3m, and the high-reality fusion of the physical world and the digital world cannot be realized.

Therefore, the AR technology can only provide limited AR functions, and cannot realize seamless fusion of the digital world and the physical world.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a virtual-real fusion system according to an embodiment of the present invention. As shown in fig. 1, the virtual-real fusion system may include an electronic device 11 and a server 12, the electronic device 11 may include an information acquisition module 111, a first spatial localization module 112, a scene understanding module 113, a position determination module 114, a rendering module 115, and a display module 116, and the server 12 may include a second spatial localization module 121, where:

an information acquisition module 111 for acquiring environmental information;

a first spatial location module 112 for sending environmental information to the server 12;

the second spatial positioning module 121 is configured to determine, according to the environment information and the stored 3D map, absolute position information of the electronic device 11 and the 3D map corresponding to the absolute position information, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 11;

the scene understanding module 113 is configured to identify an environment where the electronic device 11 is located according to the environment information and the 3D map corresponding to the absolute position information;

a position determining module 114, configured to determine, according to the absolute position information and the recognition result, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device 11 in the 3D map corresponding to the absolute position information;

a rendering module 115, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 11 in the 3D map corresponding to the absolute position information;

and the display module 116 is configured to display the rendered digital content according to a position of the digital content in the 3D map corresponding to the absolute position information.

When the user uses the virtual-real fusion system, the electronic device 11 may collect the environmental information through the information collection module 111. Specifically, the electronic device 11 may collect the environmental information through one or more sensors of a camera (camera), a wireless fidelity (WiFi), a bluetooth, a Global Positioning System (GPS), a barometer, a geomagnetic sensor, an Inertial Measurement Unit (IMU), a 5G, a light sensor, and the like. The camera is used for gathering image and/or video, and the wiFi sensor is used for gathering WiFi fingerprint signal, and the bluetooth sensor is used for gathering bluetooth signal, and GPS is used for gathering geographical position information, and the barometer is used for measuring the atmospheric pressure, and geomagnetic sensor is used for detecting earth's magnetism, and IMU is used for measuring electronic equipment 11's triaxial attitude angle and acceleration, and illumination sensor is used for gathering illumination, and 5G is used for gathering 5G basic station signal. The environmental information may include image, lighting, multispectral, depth, material, geographic location information, and the like. Multispectral, depth and material can be acquired and calculated by a multispectral sensor.

The electronic device 11 may then transmit the collected environmental information to the server 12 via the first spatial location module 112. Wireless communication or wired communication may be used between the electronic device 11 and the server 12. The server 12 may be a cloud server, an edge server, or other types of servers. After the server 12 receives the environment information from the electronic device 11, the absolute position information of the electronic device 11 and the 3D map corresponding to the absolute position information may be determined by the second spatial positioning module 121 according to the environment information and the stored 3D map. The absolute position information of the electronic device 11 may be determined by using a pre-trained positioning model, the environment information and the stored 3D map may be input into the positioning model, the output of the positioning model is the absolute position information of the electronic device 11, and then the 3D map corresponding to the absolute position information may be determined according to the absolute position information and the stored 3D map, that is, the 3D map corresponding to the absolute geographical position information is obtained from the 3D map stored in the server 12, that is, the 3D map where the current position of the electronic device 11 is located is obtained from the stored 3D map. The server 12 may then send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 11. The 3D map stored by the server 12 is a 3D map previously constructed by the server 12. Absolute position information is position information in a terrestrial coordinate system.

After the electronic device 11 receives the absolute position information and the 3D map corresponding to the absolute position information from the server 12, the scene understanding module 113 may identify the environment where the electronic device 11 is located according to the environment information and the 3D map corresponding to the absolute position information. The recognition result may include information of the target, foreground, background, spatial structure, etc. contained in the current scene. The scene where the electronic device 11 is located can be identified, the scene can be identified through a pre-trained scene identification model, the scene can be identified according to the image included in the environment information and the 3D map corresponding to the absolute position information, and the identification result can be a specific geographic position, a name of the scene, or a certain same scene in the database. The object detection (i.e., target detection) may also be performed on the image included in the environment information in combination with the 3D map corresponding to the absolute position information, that is, the object in the 3D map corresponding to the image included in the environment information and the absolute position information may be detected by using a trained object detection model. The text in the environment where the electronic device 11 is located may also be detected, and the text appearing in the 3D map corresponding to the image and the absolute position information included in the environment information may be detected by a pre-trained text detection model. For text detection, Optical Character Recognition (OCR) may be used, or other text recognition models may be used. The logo (logo) in the environment where the electronic device 11 is located can be detected, and a 3D map corresponding to the image and the absolute position information included in the environment information can be detected through a pre-trained logo detection model. It is also possible to identify the material of the object in the environment where the electronic device 11 is located, i.e. identify the material of the object in the environment information. The illumination of the environment where the electronic device 11 is located may also be identified, that is, the illumination in the environment information is identified, and the illumination may include illumination intensity, a lighting degree of light, and the like. The scene depth, i.e. the distance from the content seen by the camera to the camera, can also be calculated, and can be measured or calculated by using a time of flight (TOF), structured light or software algorithm. The occlusion in the environment where the electronic device 11 is located can also be detected, and a 3D map corresponding to the image and absolute position information included in the environment information can be detected through a pre-trained occlusion detection model. The motion foreground in the environment where the electronic device 11 is located can also be detected, and a pre-trained motion foreground detection model can be used to detect a 3D map corresponding to the image combined absolute position information included in the environment information. It is also possible to perform a plane estimation of the environment in which the electronic device 11 is located. The face in the environment where the electronic device 11 is located can also be recognized, and the face in the image included in the environment information can be recognized and labeled through a pre-trained face recognition model. The emotion of the face in the environment where the electronic device 11 is located can be identified, and the emotion of the face in the image included in the environment information can be identified and labeled through a pre-trained emotion identification model. Some information in the environment where the electronic device 11 is located can also be tracked, and some information can be tracked and labeled through a pre-trained tracking model according to images included in the environment information, and the information can be scenes, logos, faces and the like. The image included in the environment information may be an image acquired by a camera, and may also be an image included in a video acquired by the camera.

The position determining module 114 may then determine, according to the absolute position information and the recognition result of the scene understanding module 113, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device 11 in the 3D map corresponding to the absolute position information, the rendering module 115 may render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result of the scene understanding module 113 and the position of the electronic device 11 in the 3D map corresponding to the absolute position information, and the display module 116 may display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information. The scene understanding module 113 can obtain information of a target, a foreground, a background, a spatial structure and the like contained in the current scene, and can calculate a reasonable placement position and a mutual shielding relation of the digital content in the real world, so that seamless splicing of the digital world and the physical world is realized. Meanwhile, high-fidelity 3D rendering based on the global illumination technology can be achieved by means of a 3D graphic rendering engine, illumination estimation, material estimation and a digital content geometric structure. The foreground and background analysis technology for scene understanding can avoid the digital content from being superimposed on the foreground by mistake, and the motion information for scene understanding can also ensure the continuity of rendering effect.

Referring to fig. 2, fig. 2 is a schematic structural diagram of another virtual-real fusion system disclosed in the embodiment of the present invention. Wherein fig. 2 is optimized from fig. 1. As shown in fig. 2, the virtual-real fusion system may include an electronic device 21 and a server 22, the electronic device 21 may include an information acquisition module 211, a first spatial localization module 212, a scene understanding module 213, a position determination module 214, a rendering module 215, and a display module 216, and the server 22 may include a second spatial localization module 221, where:

an information collecting module 211, configured to collect environment information;

a first spatial location module 212 for sending environmental information to the server 22;

the second spatial positioning module 221 is configured to determine, according to the environment information and the stored 3D map, the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 21;

the scene understanding module 213 is configured to identify an environment in which the electronic device 21 is located according to the 3D map corresponding to the environment information and the absolute position information;

the position determining module 214 is configured to determine, according to the absolute position information and the recognition result, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device 21 in the 3D map corresponding to the absolute position information;

a rendering module 215, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 21 in the 3D map corresponding to the absolute position information;

and the display module 216 is configured to display the rendered digital content according to a position of the digital content in the 3D map corresponding to the absolute position information.

For the detailed description, reference may be made to the description corresponding to fig. 1, which is not repeated herein.

In one embodiment, the information collection module 211 collecting the environmental information may include:

environmental information is collected by one or more of a camera, WiFi, bluetooth, GPS, barometer, IMU, 5G and geomagnetic sensor.

The detailed description may refer to the related description corresponding to fig. 1, and is not repeated herein.

In another embodiment, the first spatial positioning module 212 is further configured to determine an environmental status of the electronic device 21 according to the environmental information collecting means, and send the environmental status to the server 22, where the environmental status may include an outdoor environment or an indoor environment;

the second spatial positioning module 221, determining the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, includes:

and determining the absolute position information of the electronic equipment 21 and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

In yet another embodiment, the first spatial location module 212 determining the environmental status of the electronic device 21 according to the environmental information collection means includes:

determining that the environmental state of the electronic device 21 is an indoor environment under the condition that the acquisition device of the environmental information comprises a first preset device;

in the case where the collecting device of the environmental information includes the second preset device, it is determined that the environmental state where the electronic device 21 is located is the outdoor environment.

The environmental state of the electronic device 21 may be an indoor environment or an outdoor environment. The 3D maps stored by the server 22 may include indoor 3D maps and outdoor 3D maps. Therefore, when the electronic device 21 is located, the environmental state of the electronic device 21 can be determined. The electronic device 21 may determine the environmental state of the electronic device 21 through the first spatial positioning module 212 according to the collecting device of the environmental information, and in a case that the collecting device of the environmental information includes a first preset device, the environmental state of the electronic device 21 may be determined to be an indoor environment. The first preset device may include WiFi, bluetooth, etc., and the first preset device may further include 5G, a camera, etc. In the case where the collecting means of the environmental information includes the second preset means, it may be determined that the environmental state in which the electronic device 21 is located is an outdoor environment. The second preset device may include a GPS, etc., and the second preset device may further include a 5G camera, etc. The electronic device 21 may then transmit the environmental status to the server 22 via the first spatial location module 212. After receiving the environment state from the electronic device 21, the server 22 may determine the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map, that is, determine the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the 3D map corresponding to the environment information and the environment state. The second spatial location module 221 in the server 22 may comprise two parts, one part for outdoor location and the other part for indoor location. Therefore, after receiving the environmental information and the environmental status from the electronic device 21, the server 22 may determine whether to select outdoor positioning or indoor positioning according to the environmental status, and then perform positioning according to the indoor or outdoor map and the environmental information. As can be seen, the positioning may include outdoor positioning and indoor positioning, and the positioning accuracy of the outdoor positioning and the indoor positioning may be different. Generally, since the indoor space is small, a small positioning error, that is, a high positioning accuracy is required. And the outdoor positioning space is larger, the positioning error can be slightly larger, namely the positioning precision is smaller. The absolute position information of the electronic device 21 is changed along with the movement of the electronic device 21, and during the movement of the electronic device 21, the absolute position information of the electronic device 21 can be determined in real time through a position tracking algorithm, such as a Visual Inertial Odometer (VIO).

In yet another embodiment, the environment information may include an image, and the scene understanding module 213 identifies the environment in which the electronic device 21 is located according to the 3D map corresponding to the environment information and the absolute location information includes one or more of:

identifying the scene of the electronic device 21 according to the image included in the environment information and the 3D map corresponding to the absolute position information;

performing target detection on a 3D map corresponding to an image and absolute position information included in environment information;

performing text detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

detecting a logo (logo) of a 3D map corresponding to an image and absolute position information included in environment information;

performing material identification according to the environment information;

carrying out illumination identification according to the environment information;

calculating scene depth according to the image included in the environment information and the 3D map corresponding to the absolute position information;

carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

performing plane estimation on a 3D map corresponding to an image and absolute position information included in environment information;

performing face recognition on an image included in the environment information;

performing emotion recognition on an image included in the environment information;

and tracking the target according to the image included in the environment information and the 3D map corresponding to the absolute position information.

The above detailed description may refer to the above description of the scene understanding module 113, which is not repeated herein.

After the rendering module 216 renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 21 in the 3D map corresponding to the absolute position information, the display module 116 may display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, and may display the rendered digital content at the position of the digital content in the 3D map corresponding to the absolute position information.

In yet another embodiment, the server 22 may further include a User Generated Content (UGC) module 222, and the electronic device 21 may further include a content obtaining module 217, wherein:

a UGC module 222 for generating digital content;

a content obtaining module 217 for obtaining the digital content from the server 22.

The digital content is generated by the server, and then the electronic device 21 may obtain the digital content from the server 22 through the content obtaining module 217, and may obtain the digital content together with the absolute position information, or may obtain the digital content at other times, which is not limited herein. The content acquisition module may send an acquisition request to the server 22 for acquiring the digital content. The server 22, upon receiving the acquisition request from the electronic device 21, can transmit the digital content generated by the UGC module 222 to the electronic device 21. In addition, the acquisition request may carry an identifier, and the server may transmit the digital content identified by the identifier to the electronic device. The digital content can be shared in the server, and in addition, the privacy of the digital content can also be realized through privacy setting.

The user can customize the digital content as personalized content of the digital world. Human interface and content creation tools may be provided to facilitate the user in creating his or her digital content and placing it in the UGC module 222. By setting a user privacy mechanism, a user can publish personalized content throughout the digital world, and can also set to be visible to a small extent as rendered content input.

In yet another embodiment, the electronic device 21 may further include a first codec module 218, and the server 22 may further include a second codec module 223, wherein:

a first codec module 218 for encoding environment information;

the first spatial location module 212 sending the environmental information to the server 22 includes:

sending the encoded environmental information to the server 22;

the second encoding and decoding module 223 is configured to decode the encoded environment information to obtain the environment information.

The first codec module 218 encodes the environment information, which may be all environment information or part of the environment information, such as an image, a video, etc.

In another embodiment, the second codec module 223 is further configured to encode a 3D map corresponding to absolute position information;

the sending, by the second spatial positioning module 221, the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 21 includes:

sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device 21;

the first encoding/decoding module 218 is further configured to decode the 3D map corresponding to the encoded absolute position information to obtain a 3D map corresponding to the absolute position information.

In yet another embodiment, the second codec module 223 is further configured to encode the digital content;

the content obtaining module 217 obtains the digital content from the server 22 by:

obtaining encoded digital content from the server 22;

the first codec module 218 is further configured to decode the encoded digital content to obtain the digital content.

In yet another embodiment, the server 22 may further include a training module 224, wherein:

a training module 224, configured to train a model according to the training data, and send the trained model to the electronic device 21;

the identifying, by the scene understanding module 213, the environment where the electronic device 21 is located according to the 3D map corresponding to the environment information and the absolute position information includes:

and identifying the environment of the electronic equipment 21 according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

The scene understanding module 213 of the electronic device 21 uses a plurality of trained models that can be trained by the server 22 via the training module 224 from training data. The training data may be data obtained by the server 22 from other electronic devices. The training data may be images, video, etc.

In yet another embodiment, the server 22 may further include a modeling module 225, wherein:

a modeling module 225 for obtaining modeling data, which may include images and/or video, from which to construct a 3D map.

The server 22 may acquire modeling data through the modeling module 225, and may construct a 3D map from the acquired modeling data. The indoor 3D map and the outdoor 3D map are mainly constructed. In addition, the modeling module 225 may perform updates of 3D maps using images and video acquired in real time. According to different characteristics of the indoor 3D map and the outdoor 3D map, different acquisition means can be utilized to acquire modeling data for constructing the map. For example, images and/or video used to construct outdoor 3D maps may be captured by means of drones, satellites, aerial photographs, panoramas, etc., while images and/or video maps used to construct indoor 3D maps may be captured by means of laser scanning, panoramic cameras, etc. The construction of the 3D map may be performed offline by the server 22, which may ensure its accuracy and effectiveness. In addition, since real world 3D scenes change frequently over time, the 3D map should be continuously updated. The map update can be automatically updated by means of images and videos collected by users and corresponding spatial position information and video tracking information, and can also be updated by periodically supplementing data by a publisher of the 3D map.

The 3D map is placed on the server, all users can call the map, and meanwhile storage resources used by the electronic equipment can be reduced. The UGC module stores user personalized digital content, is arranged on the server, can be shared conveniently, and adopts proper life cycle and privacy setting. The scene understanding module is arranged on the electronic equipment, so that the image can be prevented from being uploaded to a server, and the privacy of a user can be protected. The rendering module is arranged on the electronic equipment, so that the condition that the current scene is fused with the digital content and uploaded to a server can be avoided, and the privacy of a user is revealed. In addition, the rendering module is arranged on the electronic equipment, so that the virtual-real fusion efficiency can be improved, and the user experience can be improved.

For example, the electronic device collects video and geographical location information, and sends the video and geographical location information to the server. The video is collected by a camera, and the geographic location information may be collected by a Visual Positioning System (VPS), a GPS (global positioning system), a Location Based Service (LBS), and the like. The server determines the absolute position information of the electronic equipment according to the video, the geographical position information and the 3D map, and sends the absolute position information of the electronic equipment to the electronic equipment. The electronic device continuously obtains the relative pose of the camera by instant positioning in a mapping (SLAM), and couples with the VPS to continuously obtain the absolute pose of the camera in the world coordinate system (i.e. the absolute position information of the electronic device). The electronic equipment analyzes scene understanding information of the real world, such as illumination information, material information and the like, and performs realistic rendering on digital content superposed to the physical world by combining the relative pose of the camera, so that the effects of virtual and real shielding, navigation, holographic signage and the like are realized. And displaying the rendered digital content, and presenting the virtual and real fusion effect to a user. Referring to fig. 3, fig. 3 is a schematic diagram of virtual-real fusion according to an embodiment of the present invention. As shown in fig. 3, digital contents such as a convenience store logo, a virtual large screen, etc. are displayed in fusion with the physical world. Referring to fig. 4, fig. 4 is a schematic diagram of a virtual-real fusion 3D navigation according to an embodiment of the present invention. As shown in fig. 4, the route is indicated by virtual digital content.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure. As shown in fig. 5, the electronic device may include an information acquisition module 51, a spatial localization module 52, a scene understanding module 53, a location determination module 54, a rendering module 55, and a display module 56, wherein:

an information acquisition module 51 for acquiring environmental information;

the spatial positioning module 52 is configured to send environment information to the server, receive absolute position information from the server and a 3D map corresponding to the absolute position information, where the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map;

the scene understanding module 53 is configured to identify an environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information;

a position determining module 54, configured to determine, according to the absolute position information and the identification result, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device in the 3D map corresponding to the absolute position information;

a rendering module 55, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information;

and the display module 56 is configured to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

Referring to fig. 6, fig. 6 is a schematic structural diagram of another electronic device according to an embodiment of the disclosure. As shown in fig. 6, the electronic device may include an information acquisition module 61, a spatial localization module 62, a scene understanding module 63, a position determination module 64, a rendering module 65, and a display module 66, wherein:

the information acquisition module 61 is used for acquiring environmental information;

the spatial positioning module 62 is configured to send environment information to the server, receive absolute position information from the server, and a 3D map corresponding to the absolute position information, where the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map;

the scene understanding module 63 is configured to identify an environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information;

a position determining module 64, configured to determine, according to the absolute position information and the identification result, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device in the 3D map corresponding to the absolute position information;

a rendering module 65, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information;

and the display module 66 is configured to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

In another embodiment, the collecting of the environmental information by the information collecting module 61 may include:

In another embodiment, the spatial location module 62 is further configured to determine an environmental status of the electronic device according to the environmental information collecting device, and send the environmental status to the server, where the environmental status may include an outdoor environment or an indoor environment;

the determining, by the server, the absolute position information and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes:

the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information, the environment state and the stored 3D map.

In yet another embodiment, the determining, by the spatial location module 62, the environmental status of the electronic device according to the collection means of the environmental information includes:

under the condition that the acquisition device of the environmental information comprises a first preset device, determining that the environmental state of the electronic equipment is an indoor environment;

and under the condition that the acquisition device of the environmental information comprises a second preset device, determining that the environmental state of the electronic equipment is the outdoor environment.

For the specific description, reference may be made to the above corresponding description, which is not repeated herein.

In yet another embodiment, the environment information may include an image, and the scene understanding module 63 identifies the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of the following:

identifying a scene where the electronic equipment is located according to the image included in the environment information and the 3D map corresponding to the absolute position information;

performing material identification according to the environment information;

In yet another embodiment, the electronic device may further comprise a content acquisition module 67, wherein:

a content obtaining module 67 for obtaining digital content from the server.

For the detailed description of the content obtaining module, reference may be made to the above corresponding description, which is not repeated herein.

In yet another embodiment, the electronic device may further include a codec module 68, wherein:

a coding/decoding module 68 for coding the environment information;

the spatial location module 62 sending the environmental information to the server includes:

and sending the coded environment information to a server.

In another embodiment, the spatial location module 62 receiving the absolute position information of the electronic device from the server and the 3D map corresponding to the absolute position information includes:

receiving absolute position information of the electronic equipment from a server and a 3D map corresponding to the encoded absolute position information;

the encoding and decoding module 68 is further configured to decode the 3D map corresponding to the encoded absolute position information to obtain a 3D map corresponding to the absolute position information.

In yet another embodiment, the content obtaining module 67 obtaining the digital content from the server includes:

acquiring the encoded digital content from the server;

the codec module 68 is further configured to decode the encoded digital content to obtain the digital content.

The above detailed description of each module of the electronic device may refer to the above related description, and is not repeated herein.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention. As shown in fig. 7, the server may include a spatial location module 71, wherein:

the space positioning module 71 is configured to receive environment information from the electronic device, determine, according to the environment information and the stored 3D map, an absolute position information of the electronic device and a 3D map corresponding to the absolute position information, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, where the 3D map corresponding to the absolute position information and the absolute position information is used to instruct the electronic device to identify an environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, determine, according to the absolute position information and an identification result, a position of the digital content in the 3D map corresponding to the absolute position information and a position of the electronic device in the 3D map corresponding to the absolute position information, render the digital content according to a position of the digital content in the 3D map corresponding to the absolute position information, the identification result, and a position of the electronic device in the 3D map corresponding to the absolute position information, and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

In another embodiment, the spatial location module 71 is further configured to receive an environmental status from the electronic device, where the environmental status may include an outdoor environment or an indoor environment;

the spatial location module 71 determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and includes:

and determining absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

In yet another embodiment, the environment information may include an image, and the electronic device identifies the environment in which the electronic device is located according to a 3D map corresponding to the environment information and the absolute location information includes one or more of:

identifying a scene where the electronic equipment is located according to the 3D map corresponding to the image and the absolute position information;

carrying out target detection on the 3D map corresponding to the image and the absolute position information;

performing text detection on the 3D map corresponding to the image and the absolute position information;

performing mark detection on the 3D map corresponding to the image and the absolute position information;

performing material identification according to the environment information;

calculating scene depth according to the image and the 3D map corresponding to the absolute position information;

carrying out occlusion detection on the 3D map corresponding to the image and the absolute position information;

carrying out motion foreground detection on the 3D map corresponding to the image and the absolute position information;

carrying out plane estimation on the 3D map corresponding to the image and the absolute position information;

carrying out face recognition on the image;

performing emotion recognition on the image;

and tracking the target according to the 3D map corresponding to the image and the absolute position information.

In yet another embodiment, the server may further include a UGC module 72, wherein:

the UGC module 72 is configured to generate digital content and transmit the digital content to the electronic device.

In yet another embodiment, the server may further comprise a codec module 73, wherein:

the spatial location module 71 receiving the environment information from the electronic device includes:

receiving environment information coded by the electronic equipment;

and the coding and decoding module 73 is configured to decode the coded environment information to obtain the environment information.

In another embodiment, the encoding/decoding module 73 is further configured to encode a 3D map corresponding to absolute position information;

the sending of the absolute position information and the 3D map corresponding to the absolute position information to the electronic device by the spatial positioning module 71 includes:

and sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic equipment.

In another embodiment, the codec module 73 is further configured to encode digital content;

the UGC module 72 transmitting the digital content to the child device includes:

and transmitting the coded digital content to the electronic equipment.

In yet another embodiment, the server may further comprise a training module 74, wherein:

a training module 74, configured to train the model according to the training data, and send the trained model to the electronic device;

the electronic equipment identifies the environment of the electronic equipment according to the 3D map corresponding to the environment information and the absolute position information, and the identification comprises the following steps:

and identifying the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

In yet another embodiment, the server may further comprise a modeling module 75, wherein:

the modeling module 75 is configured to obtain modeling data, and construct a 3D map according to the modeling data, where the modeling data includes images and/or videos.

The detailed description of each module above the server can refer to the above related description, and is not repeated herein.

Referring to fig. 8, fig. 8 is a schematic flow chart of a virtual-real fusion method according to an embodiment of the present invention. As shown in fig. 8, the virtual-real fusion method may include the following steps.

801. The electronic device collects environmental information.

The electronic device may collect environmental information through one or more of a camera, WiFi, bluetooth, GPS, barometer, IMU, 5G, and a geomagnetic sensor.

802. The electronic device sends the environment information to the server.

Accordingly, the server receives the environment information from the electronic device.

803. And the server determines the absolute position information of the electronic equipment and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map.

804. And the server sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic equipment.

805. And the electronic equipment identifies the environment of the electronic equipment according to the 3D map corresponding to the environment information and the absolute position information.

806. And the electronic equipment determines the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result.

807. And the electronic equipment renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information.

808. And the electronic equipment displays the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

Optionally, the method may further include:

the electronic equipment determines the environmental state of the electronic equipment according to the acquisition device of the environmental information and sends the environmental state to the server, wherein the environmental state can comprise an outdoor environment or an indoor environment;

step 805 specifically comprises:

Optionally, the determining, by the electronic device according to the collection device of the environmental information, the environmental state of the electronic device includes:

Optionally, the environment information may include an image, and the identifying, by the electronic device, the environment in which the electronic device is located according to the 3D map corresponding to the environment information and the absolute position information includes one or more of the following:

the electronic equipment identifies the scene where the electronic equipment is located according to the image included in the environment information and the 3D map corresponding to the absolute position information;

the electronic equipment detects the target of the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment performs text detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment carries out mark detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment identifies the material quality according to the environment information;

the electronic equipment performs illumination identification according to the environment information;

the electronic equipment carries out scene depth calculation according to the image included by the environment information and the 3D map corresponding to the absolute position information;

the electronic equipment carries out shielding detection on the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment detects the movement foreground of the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment carries out plane estimation on the 3D map corresponding to the image and the absolute position information included in the environment information;

the electronic equipment carries out face recognition on the image included by the environment information;

the electronic equipment identifies the emotion of the image included in the environment information;

and the electronic equipment tracks the target according to the image contained in the environment information and the 3D map corresponding to the absolute position information.

Optionally, the method may further include:

the server generates digital content;

the electronic device obtains digital content from a server.

Optionally, the method may further include:

the electronic equipment encodes the environment information;

the electronic device sending the environment information to the server includes:

the electronic equipment sends the coded environment information to a server;

the above method may further comprise:

and the server decodes the coded environment information to obtain the environment information.

Optionally, the method may further include:

the server encodes the 3D map corresponding to the absolute position information;

the server sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic equipment, and the method comprises the following steps:

the server sends the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic equipment;

the above method may further comprise:

and the electronic equipment decodes the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.

Optionally, the method may further include:

the server encodes the digital content;

the electronic device obtaining the digital content from the server comprises:

the electronic equipment acquires the coded digital content from the server;

the above method may further comprise:

the electronic equipment decodes the encoded digital content to obtain the digital content.

Optionally, the method may further include:

the server trains the model according to the training data and sends the trained model to the electronic equipment;

and the electronic equipment identifies the environment of the electronic equipment according to the trained model, the environment information and the 3D map corresponding to the absolute position information.

Optionally, the method may further include:

the server obtains modeling data, and constructs a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

The details of each step above can refer to the above related description, and are not repeated herein.

The contents of the above several embodiments can be referred to each other, and the contents of each embodiment are not limited to the embodiment, and can also be applied to the corresponding contents in other embodiments.

The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only illustrative of the present invention and are not intended to limit the scope of the present invention, and any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present invention should be included in the scope of the present invention.

Claims

1. The utility model provides a virtual reality fuses system, its characterized in that includes electronic equipment and server, electronic equipment includes information acquisition module, first spatial localization module, scene understanding module, position determination module, renders up module and display module, the server includes second spatial localization module, wherein:

the information acquisition module is used for acquiring environmental information;

the first space positioning module is used for sending the environment information to the server;

the second spatial positioning module is configured to determine, according to the environment information and a stored 3D map, absolute position information of the electronic device and the 3D map corresponding to the absolute position information, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device;

the scene understanding module is used for identifying the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information;

the position determining module is used for determining the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result;

the rendering module is configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result, and the position of the electronic device in the 3D map corresponding to the absolute position information;

and the display module is used for displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

2. The system according to claim 1, wherein the first spatial positioning module is further configured to determine an environmental status of the electronic device according to the collecting device of the environmental information, and send the environmental status to the server, where the environmental status includes an outdoor environment or an indoor environment;

the second spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and the second spatial positioning module comprises:

and determining the absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

3. The system of claim 2, wherein the first spatial location module determining the environmental state of the electronic device according to the environmental information collection device comprises:

determining that the environmental state of the electronic equipment is an indoor environment under the condition that the acquisition device of the environmental information comprises a first preset device;

and under the condition that the acquisition device of the environmental information comprises a second preset device, determining that the environmental state of the electronic equipment is an outdoor environment.

4. The system according to any one of claims 1-3, wherein the environment information comprises an image, and the scene understanding module identifies the environment in which the electronic device is located according to a 3D map corresponding to the environment information and the absolute position information comprises one or more of:

identifying the scene of the electronic equipment according to the 3D map corresponding to the image and the absolute position information;

performing target detection on the 3D map corresponding to the image and the absolute position information;

performing sign detection on the 3D map corresponding to the image and the absolute position information;

performing material identification according to the environment information;

performing plane estimation on the 3D map corresponding to the image and the absolute position information;

carrying out face recognition on the image;

performing emotion recognition on the image;

and tracking the target according to the image and the 3D map corresponding to the absolute position information.

5. The system according to any of claims 1-4, wherein the server further comprises a User Generated Content (UGC) module, the electronic device further comprising a content acquisition module, wherein:

the UGC module is used for generating the digital content;

the content obtaining module is used for obtaining the digital content from the server.

6. The system of any one of claims 1-5, wherein the server further comprises a training module, wherein:

the training module is used for training a model according to training data and sending the trained model to the electronic equipment;

the scene understanding module identifies the environment of the electronic device according to the environment information and the 3D map corresponding to the absolute position information, and the scene understanding module comprises:

7. The system of any one of claims 1-6, wherein the server further comprises a modeling module, wherein:

the modeling module is used for acquiring modeling data and constructing a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

8. An electronic device comprising an information acquisition module, a spatial localization module, a scene understanding module, a position determination module, a rendering module, and a display module, wherein:

the spatial positioning module is configured to send the environment information to the server, receive absolute position information of the electronic device from the server and a 3D map corresponding to the absolute position information, where the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map;

9. The electronic device according to claim 8, wherein the spatial positioning module is further configured to determine an environmental status of the electronic apparatus according to the collecting device of the environmental information, and send the environmental status to the server, where the environmental status includes an outdoor environment or an indoor environment;

and the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information, the environment state and the stored 3D map.

10. The electronic device of claim 9, wherein the determining, by the spatial location module, the environmental status of the electronic device according to the environmental information collecting means comprises:

11. The electronic device according to any one of claims 8 to 10, wherein the environment information includes an image, and the scene understanding module identifies the environment in which the electronic device is located according to a 3D map corresponding to the environment information and the absolute position information includes one or more of:

performing material identification according to the environment information;

carrying out face recognition on the image;

performing emotion recognition on the image;

12. The electronic device of any of claims 8-11, further comprising a content acquisition module, wherein:

13. A server, comprising a spatial location module, wherein:

the spatial positioning module is configured to receive environment information from an electronic device, determine, according to the environment information and a stored 3D map, absolute position information of the electronic device and a 3D map corresponding to the absolute position information, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, where the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify an environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, determine, according to the absolute position information and an identification result, a position of digital content in the 3D map corresponding to the absolute position information, and a position of the electronic device in the 3D map corresponding to the absolute position information, and determine, according to the position of the digital content in the 3D map corresponding to the absolute position information, Rendering the digital content according to the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information, and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

14. The server according to claim 13, wherein the spatial location module is further configured to receive an environmental status from the electronic device, the environmental status comprising an outdoor environment or an indoor environment;

the spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and the spatial positioning module comprises:

15. The server according to claim 13 or 14, characterized in that the server further comprises a user generated content, UGC, module, wherein:

the UGC module is used for generating the digital content and sending the digital content to the electronic equipment.

16. The server according to any one of claims 13-15, wherein the server further comprises a training module, wherein:

the identifying, by the electronic device, the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information includes:

17. The server according to any of claims 13-16, wherein the server further comprises a modeling module, wherein:

18. A method for fusing virtuality and reality is characterized by comprising the following steps:

the electronic equipment collects environmental information and sends the environmental information to a server;

the server determines absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information and the stored 3D map, and sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic equipment;

the electronic equipment identifies the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information, determines the position of the digital content in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result, renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information, and displays the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

19. The method of claim 18, further comprising:

the electronic equipment determines the environmental state of the electronic equipment according to the acquisition device of the environmental information and sends the environmental state to the server, wherein the environmental state comprises an outdoor environment or an indoor environment;

the server determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes:

and the server determines the absolute position information of the electronic equipment and the 3D map corresponding to the absolute position information according to the environment information, the environment state and the stored 3D map.

20. The method of claim 19, wherein determining, by the electronic device, the environmental state of the electronic device according to the environmental information collection device comprises:

under the condition that the acquisition device of the environmental information comprises a first preset device, the electronic equipment determines that the environmental state where the electronic equipment is located is an indoor environment;

and under the condition that the acquisition device of the environmental information comprises a second preset device, the electronic equipment determines that the environmental state where the electronic equipment is located is an outdoor environment.

21. The method according to any one of claims 18 to 20, wherein the environment information includes an image, and the electronic device identifies the environment in which the electronic device is located according to a 3D map corresponding to the environment information and the absolute position information includes one or more of:

the electronic equipment identifies the scene where the electronic equipment is located according to the image and the 3D map corresponding to the absolute position information;

the electronic equipment performs target detection on the 3D map corresponding to the image and the absolute position information;

the electronic equipment performs text detection on the 3D map corresponding to the image and the absolute position information;

the electronic equipment carries out mark detection on the 3D map corresponding to the image and the absolute position information;

the electronic equipment carries out scene depth calculation according to the image and the 3D map corresponding to the absolute position information;

the electronic equipment carries out shielding detection on the 3D map corresponding to the image and the absolute position information;

the electronic equipment carries out motion foreground detection on the 3D map corresponding to the image and the absolute position information;

the electronic equipment carries out plane estimation on the 3D map corresponding to the image and the absolute position information;

the electronic equipment carries out face recognition on the image;

the electronic equipment carries out emotion recognition on the image;

and the electronic equipment tracks the target according to the image and the 3D map corresponding to the absolute position information.

22. The method according to any one of claims 18-21, further comprising:

the server generates the digital content;

the electronic device obtains the digital content from the server.

23. The method according to any one of claims 18-22, further comprising:

the server trains a model according to the training data and sends the trained model to the electronic equipment;

24. The method according to any one of claims 18-23, further comprising:

the server acquires modeling data, and constructs a 3D map according to the modeling data, wherein the modeling data comprises images and/or videos.

25. A method for fusing virtuality and reality is characterized by comprising the following steps:

collecting environmental information;

sending the environment information to the server, receiving absolute position information of the electronic equipment from the server and a 3D map corresponding to the absolute position information, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map;

identifying the environment of the electronic equipment according to the environment information and the 3D map corresponding to the absolute position information;

determining the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic equipment in the 3D map corresponding to the absolute position information according to the absolute position information and the identification result;

rendering the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the identification result and the position of the electronic equipment in the 3D map corresponding to the absolute position information;

and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.

26. The method of claim 25, further comprising:

determining the environment state of the electronic device according to the acquisition device of the environment information, and sending the environment state to the server, wherein the environment state comprises an outdoor environment or an indoor environment;

the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map to include

27. The method according to claim 25 or 26, wherein the environment information comprises an image, and the identifying the environment of the electronic device according to the 3D map corresponding to the environment information and the absolute position information comprises one or more of the following:

performing material identification according to the environment information;

carrying out face recognition on the image;

performing emotion recognition on the image;

28. A method for fusing virtuality and reality is characterized by comprising the following steps:

receiving environmental information from the electronic device;

determining absolute position information of the electronic equipment and a 3D map corresponding to the absolute position information according to the environment information and the stored 3D map;

sending the absolute position information and a 3D map corresponding to the absolute position information to the electronic device, where the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to recognize the environment where the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, determine the position of the digital content in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result, determine the position of the electronic device in the 3D map corresponding to the absolute position information, render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information The digital content of (1).

29. The method of claim 28, further comprising:

receiving an environmental status from the electronic device, the environmental status comprising an outdoor environment or an indoor environment;

the determining of the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environment information and the stored 3D map includes:

30. The method of claim 28 or 29, further comprising:

generating the digital content;

transmitting the digital content to the electronic device.