CN116860113A

CN116860113A - XR combined scene experience generation method, system and storage medium

Info

Publication number: CN116860113A
Application number: CN202311028835.3A
Authority: CN
Inventors: 蔡铁峰
Original assignee: Shenzhen Polytechnic
Current assignee: Shenzhen Polytechnic
Priority date: 2023-08-16
Filing date: 2023-08-16
Publication date: 2023-10-10
Anticipated expiration: 2043-08-16
Also published as: CN116860113B

Abstract

The invention discloses an XR combined scene experience generation method, an XR combined scene experience generation system and a storage medium, wherein the XR combined scene experience generation method comprises the following steps: step S10, generating experience pictures of all sub-scenes of the combined scene; and S20, carrying out shielding calculation on all the sub-scene experience pictures to synthesize a combined scene experience picture of the user p, and displaying the combined scene experience picture to the user p for watching. According to the technical scheme, under the condition that the combined scene does not meet the condition that separation surfaces exist among all sub-scenes, under the real-time pose of a user, whether each sub-scene is unidirectional blocked or not blocked with other sub-scenes is judged, and for the sub-scenes which are unidirectional blocked or not blocked with other sub-scenes, when a user experience picture is rendered, a depth image corresponding to the experience picture does not need to be generated and transmitted, and the depth image can be correctly synthesized with the experience picture images of other sub-scenes according to the unidirectional blocking relation, so that the bandwidth required for transmitting the depth image is remarkably saved, the calculated amount of image synthesis is reduced, and the economic benefit is clear.

Description

XR combined scene experience generation method, system and storage medium

Technical Field

The invention relates to the technical field of virtual reality, in particular to an extended reality (XR) combined scene experience generation method, an extended reality (XR) combined scene experience generation system and a storage medium.

Background

Technologies such as Virtual Reality (VR), augmented Reality (AR), and Mixed Reality (MR) are compatible, and are collectively referred to as an augmented reality technology (XR). The XR combined scene is a special scene formed by combining more than 2 sub-scenes, and a user can view the contents of all the sub-scenes in the combined scene and interact, wherein the sub-scenes can be virtual scenes, digital twin scenes or real scenes. The combined scene can break the colorful experience activity types of the fence construction among the scenes, for example, in a classroom, a plurality of single teaching activity scenes or a plurality of collaborative teaching activity scenes can be combined to generate a teaching combined scene containing a plurality of scenes, and a plurality of combined scenes can be constructed by changing the types and the quantity of the contained scenes, wherein the method mainly comprises the following steps: "in-study" scenes, "demonstration-practical training" scenes, "cognition-practical training" scenes, practical training help scenes, assessment competition scenes, multi-group practical operation demonstration and the like.

The sub-scenes of the combined scene can be used for both single user experience and multi-user collaborative experience. In a virtual-real fusion combined scene formed by combining a real scene and a virtual sub-scene, a real scene experience picture is generated in real time by a real camera, a virtual scene experience picture is rendered by a computing system, and a complete experience picture of the combined scene is obtained by synthesizing the real scene experience picture and the virtual sub-scene experience picture. When a combined scene contains multiple virtual sub-scenes, a single computing device is likely not able to load the rendering of this combined scene. Under the energization of high-performance wireless network technologies such as a 5G network and wifi 6, services such as storage, calculation, rendering and the like required by an XR mobile terminal (5G mobile phone, head display and the like) can be put on the cloud. Therefore, based on cloud services such as cloud storage, cloud computing and cloud rendering, the computing, storage and rendering capabilities of a single XR terminal can be unlimited, each virtual sub-scene of a combined scene is deployed and operated on a plurality of servers on a server cluster or on a plurality of display cards of a large server, each virtual sub-scene is respectively and independently rendered on different servers or different display cards to generate experience pictures, all sub-scene experience pictures are summarized together after being transmitted, and the experience pictures of the combined scene are obtained through image synthesis, so that based on the cloud services, XR combined scene immersive experience containing a large number of sub-scenes can be generated for users.

When sub-scenes in the combined scene are respectively and independently rendered, the user experience picture of each sub-scene needs to be synthesized by shielding calculation to the experience picture of the user on the XR combined scene. In the patent application of a new dimension space construction method, system and platform based on XR technology (application number CN 202210022608.9), corresponding depth images are generated while rendering and generating user experience pictures of all sub-scenes, and shielding calculation is carried out on the user experience pictures of all the sub-scenes according to the depth images. In order to subtract the generation and transmission of the depth image, in the patent application of the ubiquitous training campus construction method, system and storage medium based on the XR technology (application number: CN 202210908795.0), the separation surfaces must exist between all sub-scenes in advance, and the occlusion relation between the experience pictures of all virtual scenes can be calculated without the depth image by the position of the separation surface where the scene is located. However, many XR combined scenes in practical application requirements cannot meet the requirement that each sub-scene has a separation surface, for example: the combined scene composed of a real campus scene and a plurality of virtual training scenes is deployed in the scenes of squares, building halls and the like of the real campus, the virtual training scenes are often surrounded by buildings in the real campus scene, and a separation surface capable of separating the virtual training scenes from the real campus scene does not exist between the virtual training scenes and the real campus scene, so that the construction method of the ubiquitous training campus construction method, the ubiquitous training campus construction system and the construction method of a storage medium (application number: CN 202210908795.0) based on the XR technology cannot be applied to.

Disclosure of Invention

The inventors found that: even if there is no separation surface between two sub-scenes of the combined scene, there may be a certain user position interval, where the two sub-scenes are blocked unidirectionally, so that when the user is in the position interval, the blocking relationship of the experience picture between the two sub-scenes can be determined without depth information.

Based on the above findings, a main object of the present invention is to provide a method, a system and a storage medium for generating an XR combined scene experience. For the combined scene which does not meet the requirement that a separation surface exists between all the sub-scenes, the invention respectively judges whether each sub-scene is one-way blocked or not blocked with other sub-scenes under the real-time pose of the user, and for the sub-scene which is one-way blocked or not blocked with other sub-scenes, the invention does not need to generate and transmit a depth image corresponding to the experience picture when the user experience picture is rendered, and can accurately synthesize the experience picture image of other sub-scenes according to the one-way blocking relation.

In order to achieve the above purpose, the present invention provides an XR combined scene experience generation method, which comprises the following steps:

step S10, generating experience pictures of all sub-scenes of the combined scene, wherein all arbitrary virtual sub-scenes S _i Acquiring the pose of the user p in the coordinate system in real time, and according to the real-time pose and s _i Is the imaging interval of (a) and the user p is s _i Pupil distance in scene space, generating user p versus s _i Is an experience picture of the computer;

step S20, each sub-scene experience picture is subjected to shielding calculation to be synthesized into a combined scene experience picture of the user p, and the combined scene experience picture is displayed to the user p for viewing, wherein any sub-scene S which is one-way shielding or non-shielding with all other sub-scenes is selected _i When experiencing picture composition, when s _i Experience picture image arbitrary pixel and other arbitrary sub-scene s _j When the shielding relation exists among the picture image pixels, the method for calculating the shielding relation among the pixels is as follows: according to the real-time position of the user p in the combined scene, s is obtained _i And s _j The one-way occlusion relation between the pixels is s _i And s _j Is a one-way occlusion relationship;

if the combined scene comprises a real scene, the sub-scenes of the combined scene comprise a real scene and a virtual sub-scene, and in addition, each sub-scene is a convex section or a non-convex section in a display section of the combined scene.

As a further improvement of the invention, the step S10 is preceded by a step S00 of setting a combined scene experience generation parameter, including setting imaging intervals of all virtual sub-scenes, pupil distance of a user p in each virtual sub-scene, rotation translation transformation relation between the combined scene coordinate system and each virtual sub-scene coordinate system, display intervals of all virtual sub-scenes in the combined scene, calculating user position intervals corresponding to different shielding relations among all sub-scenes, and calculating a position of the user corresponding to any sub-scene S _i And s _j ，s _i And s _j The user position intervals corresponding to the different shielding relations comprise: s is(s) _i Non-shielding shelter _j User location intervals, s _j Does not shade s _i User location intervals, s _i And s _j And the user position interval is blocked in two directions.

As a further improvement of the present invention, step S30 is also provided after step S20, wherein the combined scene receives the user p interaction input, judges whether the interaction input is interaction of the sub-scene, and if the interaction input is judged to be the user p to any sub-scene S _i Is then converted into a scene s _i Interactive input in coordinate system s _i Responding to the converted interactive input.

As a further development of the invention, step S00 also comprises: for arbitrary virtual sub-scene s _i If it is in a combined sceneDisplay interval omega 'of (2)' _i For non-convex region, several sub-convex regions omega 'are set' _i,0 、Ω′ _i,1 、...、/>Completely cover omega' _i I.e. +.>At the same time require->As small as possible, and all sub-convex sections do not intersect with the display sections and sub-convex sections of other sub-scenes.

As a further improvement of the present invention, for any sub-scene pair s _i And s _j Step S00, calculating the user position interval corresponding to different shielding relations comprises the following steps:

s001, judging S _i And s _j In a combined sceneWhether the display interval of (2) is constantly and unidirectionally blocked, if so, calculating s _i Does not shade s _j User location interval and s of (2) _j Non-shielding shelter _i Ending the calculation, if not, entering step S002;

s002, if S _i In a combined sceneDisplay interval omega 'of (2)' _i For convex regions, calculate Ω' _i And s _j The user position interval corresponding to each shielding relation of each sub-convex interval is further calculated to obtain s _i And s _j User position intervals corresponding to each shielding relation, wherein s is satisfied _i Does not shade s _j The user position interval of all the sub-convex intervals is s _i Does not shade s _j Interval satisfies s _j All sub-convex regions do not shade s _i The user location interval of (2) is s _j Does not shade s _i Section, if s _i For non-convex regions, traverse s _i All sub-convex regions are respectively calculated and s _j Corresponding to each shielding relation of the system, thereby further calculating s _i And s _j User position intervals corresponding to each shielding relation, wherein s is satisfied _i All sub-convex regions do not shade s _j The user location interval of (2) is s _i Does not shade s _j Interval satisfies s _j Does not shade s _i The user position interval of all the sub-convex intervals is s _j Does not shade s _i Interval.

As a further improvement of the present invention, the step S10 further judges the scene S in the current real-time pose of the user p _i If not, generating a depth image corresponding to the experience picture is also needed, and if so, generating no depth image is needed.

As a further improvement of the present invention, the step S10 judges the scene S in the current real-time pose of the user p _i The way of whether all other sub-scenes are unidirectional occlusion or non-occlusion is as follows: the real-time position of the user p in the combined scene is W _j According to the user position intervals corresponding to different shielding relations among all the sub-scenes calculated in the step S00, respectively searching out W _j S where is located _i User position interval corresponding to different shielding relations of all other sub-scenes, when W _j S where is located _i If the user position interval corresponding to the different shielding relations of all other sub-scenes is not the bidirectional shielding user position interval, under the current real-time pose of the user p, the scene s _i Is one-way occlusion or non-occlusion with all other sub-scenes.

To achieve the above object, the present invention also provides an XR combined scene experience generation system, which is characterized in that the system comprises a memory, a processor, and an XR combined scene experience generation method program stored on the processor, wherein the XR combined scene experience generation method program is executed by the processor to perform the steps of the method as described above.

To achieve the above object, the present invention also proposes a computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when invoked by a processor, performs the steps of the XR combined scene experience generation method as described above.

The invention provides an XR combined scene experience generation method, system and storage medium, which have the beneficial effects that: according to the technical scheme, under the condition that the combined scene does not meet the condition that separation surfaces exist among all sub-scenes, under the real-time pose of a user, whether each sub-scene is unidirectional blocked or not blocked with other sub-scenes is judged, and for the sub-scenes which are unidirectional blocked or not blocked with other sub-scenes, when a user experience picture is rendered, a depth image corresponding to the experience picture does not need to be generated and transmitted, and the depth image can be correctly synthesized with the experience picture images of other sub-scenes according to the unidirectional blocking relation, so that the bandwidth required for transmitting the depth image is remarkably saved, the calculated amount of image synthesis is reduced, and the economic benefit is clear.

Drawings

Fig. 1 is a flow chart of an XR combined scene experience generation method of the present invention.

Fig. 2 is a diagram of an example combined scenario.

Fig. 3 is a schematic diagram of unidirectional occlusion between sub-scenes of a combined scene.

FIG. 4 is a schematic diagram of bi-directional occlusion between sub-scenes of a combined scene.

FIG. 5 is an exemplary diagram of user location intervals corresponding to different occlusion relationships between convex intervals.

FIG. 6 is an exemplary diagram of user location intervals corresponding to different occlusion relationships between concave and convex intervals.

Fig. 7 is a system configuration diagram of a first embodiment of the present invention.

Fig. 8 is a system configuration diagram of a second embodiment of the present invention.

Fig. 9 is a system configuration diagram of a third embodiment of the present invention.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Referring to fig. 1, the invention provides an XR combined scene experience generation method, which comprises the following steps:

Further, a step S00 is provided before the step S10, wherein the step S00 is provided for setting the generating parameters of the combined scene experience, including setting the imaging interval of each virtual sub-scene, the pupil distance of the user p in each virtual sub-scene, the rotation translation transformation relation between the combined scene coordinate system and each virtual sub-scene coordinate system, the display interval of each virtual sub-scene in the combined scene, calculating the user position interval corresponding to different shielding relation among all sub-scenes, and for any sub-scene S _i And s _j ，s _i And s _j The user position intervals corresponding to the different shielding relations comprise: s is(s) _i Non-shielding shelter _j User location intervals, s _j Does not shade s _i User location intervals, s _i And s _j And the user position interval is blocked in two directions.

Step S30 is also performed after the step S20, wherein the combined scene receives the interactive input of the user p, judges whether the interactive input is the interaction of the sub-scene, and if the interactive input is judged to be the interaction of the user p on any sub-scene S _i Is then converted into a scene s _i Interactive input in coordinate system s _i Responding to the converted interactive input.

In addition, step S00 further comprises: for arbitrary virtual sub-scene s _i If it is in a combined sceneDisplay interval omega 'of (2)' _i For non-convex region, several sub-convex regions omega 'are set' _i,0 、Ω′ _i,1 、...、/>Completely cover omega' _i I.e. +.>At the same time require->As small as possible, and all sub-convex regions are notIntersecting the display intervals of other sub-scenes and sub-convex intervals.

Wherein for any sub-scene pair s _i And s _j Step S00, calculating the user position interval corresponding to different shielding relations comprises the following steps:

Further, the step S10 further determines that the scene S is in the current real-time pose of the user p _i If not, generating a depth image corresponding to the experience picture is also needed, and if so, generating no depth image is needed.

Wherein, the step S10 judges the scene S under the current real-time pose of the user p _i The way of whether all other sub-scenes are unidirectional occlusion or non-occlusion is as follows: the real-time position of the user p in the combined scene is W _j According to the user position intervals corresponding to different shielding relations among all the sub-scenes calculated in the step S00, respectively searching out W _j S where is located _i User position interval corresponding to different shielding relations of all other sub-scenes, when W _j S where is located _i If the user position interval corresponding to the different shielding relations of all other sub-scenes is not the bidirectional shielding user position interval, under the current real-time pose of the user p, the scene s _i Is one-way occlusion or non-occlusion with all other sub-scenes.

In addition, step S10 further determines whether the person in the real scene enters the display section of the virtual sub-scene in the combined scene in real time, if it is determined that the person in the real scene enters any virtual sub-scene S _i After combining the display intervals of the scene, if s _i If the scene is one-way occlusion or non-occlusion with all other sub-scenes, setting s _i It is necessary to generate corresponding depth images while generating the user p stereoscopic experience picture when the person in the real scene leaves s _i Resetting s after the display interval of the combined scene _i The corresponding depth image is not generated while the user p stereoscopic experience picture is generated.

An XR composite scene experience generation method according to the present invention will be described in detail with reference to fig. 2 to 9, and the first to fourth embodiments.

The combined scene of the invention does not meet the requirement that the separation surface exists among all the sub-scenes, taking fig. 2 as an example, the separation surface does not exist among the sub-scene a and the sub-scenes b, c and d, but at the user position shown in fig. 2, the separation surface is one-way shielding or non-shielding between the sub-scene a and the sub-scenes b, c and d and between other sub-scenes.

The following describes important technical terms related to the present invention.

Convex section

Order theIs a three-dimensional convex section, and has the following properties: let->Wherein the point a and the point b are two points, and the coordinates of the point a areThe coordinates of the point b are +.>The point c is on the connection line of the points a and b and between the points a and b, and the coordinate of the point c can be expressed as +.>Wherein 1 > lambda > 0, then the c-point necessarily also belongs to +.>

Scene and scene instance

A scene defines objects contained in a three-dimensional space, object states, object itself running logic, and logic for interactions between objects; the scene example is a program process or thread which is operated in real time by computing resources such as a computer processor, a memory, a display card and the like according to scene definition, and the program process or thread calculates the states of all objects in the scene in real time, renders pictures and responds to the interaction of users. When a single scene has multiple user experiences at the same time, if the computing resource which can be obtained by the single scene instance can not generate experience pictures for all users in real time, multiple scene instances are needed to be generated for the scene and distributed to all users, object states in the scene are synchronized by establishing communication connection among the scene instances, and the experience pictures are respectively generated for the corresponding users in real time by all the scene instances, so that all the users share the experience scene.

XR scene imaging interval

User p in a combined scenarioPerforming immersive interactive experience s _i To form a combined scene->Is one of the sub-scenes of (a). In generating a combined scene->S is as experience of _i It is possible that only part of the scene content needs to be in the combined scene +.>Is presented internally to the user p-view in order to define s _i Need to be in the combined scene->Scene content presented to user p to see is set to a three-dimensional interval omega _i Scene s _i Only the content of the three-dimensional interval needs to be imaged when the experience picture of the user p is generated, and the set three-dimensional interval is an XR scene imaging interval. The imaging interval can be infinite, or can be any three-dimensional shape interval such as a cube, a sphere and the like.

XR scene in display interval of combined scene

Combined sceneComprising a plurality of sub-scenes, in order to make the sub-scenes combined in the scene->Without or with reduced overlap, for any sub-scene s _i In the combined scene->A three-dimensional interval omega 'is internally set' _i For presenting s _i Defining s _i Only at omega' _i Internal presentation, Ω' _i Just is in the combined scene->When the combined scene is set>Coordinate system and sub-scene s _i In the case of a rotation-translation scaling relationship between coordinate systems, a three-dimensional display section Ω 'is displayed according to the rotation-translation scaling relationship' _i Mapping to sub-scene s _i The three-dimensional interval omega _i Can be used as scene s _i In turn, scene s according to this rotation-translation scaling relationship _i Is of the imaging interval omega _i Mapping to combined scene->The three-dimensional interval omega 'is obtained' _i Can be used as scene s _i In a combined scene->Is displayed in the display section of (a).

Unidirectional shielding and bidirectional shielding

Scene s _i Sum s _j For combined scenesAt any time t _k If on all the lines of view of user p, if s _i And s _j Shielding occurs, only s _i S of scene content occlusion _j Scene content, absence of s _j Scene content occlusion s _i In the case of scene content, at time t _k For user p, scene s _i Sum s _j Unidirectional shielding, and the shielding relation is s _i Unidirectional shielding s _j Taking fig. 3 as an example, in the user position of fig. 3, scene a unidirectionally obscures scene b; conversely, ifOn all view lines of user p, if s _i And s _j The shielding is s _j S of scene content occlusion of (2) _i Scene content, at time t _k For user p, scene s _i Sum s _j Unidirectional shielding, and the shielding relation is s _j Unidirectional shielding s _i The method comprises the steps of carrying out a first treatment on the surface of the In contrast, if in the line of sight of user p, if s _i And s _j Shielding occurs if at the same time s may be present _i S of scene content occlusion of (2) _j Scene content, s _j S of scene content occlusion of (2) _i Scene content, at time t _k For user p, scene s _i Sum s _j Taking fig. 4 as an example, in the user position of fig. 4, the point D of the scene b is blocked by the point C of the scene a, and the point D also blocks the point E of the scene a, so that the scene a and the scene b are blocked in both directions. If no matter the user p is in the combined scene +.>Where in it there are scenes s _i Sum s _j Non-bidirectional occlusion, then we call: in a combined scene->In, scene s _i Sum s _j Is constantly one-way occlusion. Two disjoint convex sections are necessarily one-way blocked no matter where the user is, and see the specification 115-116 of patent application (application number: CN 202210908795.0) of a ubiquitous training campus building method, system and storage medium based on XR technology for details.

The core principle of the present invention is explained below.

1) User position intervals corresponding to different shielding relations between convex intervals

As shown in fig. 5, in the combined scene, two sub-scenes a and B, in which the display section is a convex section and does not overlap, have 3 occlusion relations on the line of sight of the user: scene a occludes B, scene B occludes a, and does not occlude each other.

Between two non-overlapping lands there must be a separation surface such that two lands are on either side of the separation surface, and such separation surface is present in 1 or more. Only in special cases are there only 1 separating surface, for example: the display sections of both scenes are spherical and the 2 spherical display sections are contiguous, in this particular case the only separating plane is the plane tangential to the 2 spherical sections at the same time. When there are multiple separation surfaces between two lands, there are more than 2 double-adjoining separation surfaces, where double-adjoining separation surfaces refer to separation surfaces that adjoin both lands at the same time (adjoining refers to separation surfaces that are in communication with lands). Taking fig. 5 as an example, as shown in fig. 5 a, the combined scene includes a sub-scene A, B, and both sub-scenes are convex in the display section of the combined scene. In fig. 5B, the double-contiguous separation plane 1 separates the display interval of scene A, B on both sides, where when the user is on the upper left, if there is an occlusion between scenes a and B on the user's line of sight, it is necessarily scene a occlusion B; when the user is on the lower right, if there is an occlusion between scene a and scene B on the user's view, it is necessary that scene B occlude scene a. Note that when the user is on both sides of the separation plane, it may occur that, in all directions of the line of sight of the user, the scene a and the scene B do not occur at the same time, that is, no occlusion occurs between the scenes a and B in all directions of the line of sight of the user. In fig. 5 c, the separating surface 2 is another double adjoining separating surface. As shown in fig. 5 d, which is available from the 2 double-contiguous separation planes, in the combined scene space, when the user intersects the scene a block B interval in fig. 5B with the scene a block B interval in fig. 5 c and subtracts the interval portion sandwiched between scene a and scene B, it is still the scene a block scene B interval; similarly, when the user intersects the scene B block a section in the B diagram of fig. 5 with the scene B block a section in the c diagram of fig. 5 and subtracts the section part between the scene a and the scene B, the section part still is the scene B block scene a section; when the user is in other intervals of the combined scene, no occlusion between scenes a and B occurs. Thereby obtaining three user position intervals corresponding to different occlusion relations of the A, B interval of the scene: the scene A unidirectionally shields the user position interval of the scene B, the scene B unidirectionally shields the user position interval of the scene A, and the user position interval which is not shielded between the scene B and the scene A.

2) User position interval corresponding to different shielding relations between non-convex interval and convex interval

As shown in fig. 6, the display section of the scene a in the combined scene is a non-convex section, the display section of the scene B in the combined scene is a convex section, and for the user in the combined scene, the occlusion relationship between the scene a and the scene B includes the following 4 cases: the scene A and the scene B are blocked in two directions, the scene A is blocked in one direction, the scene B is blocked in one direction, the scene A is blocked in one direction, and the scene B and the scene A are not blocked.

As shown in fig. 6 a, the non-convex section a may be divided into a plurality of convex sections. As shown in B and c of fig. 6, according to the user position interval corresponding to the convex interval shielding relationship, the user position interval corresponding to the shielding relationship between each sub-convex interval of the non-convex interval a and the convex interval B can be obtained. As shown in d diagram of fig. 6, when at a certain user position in the combined space, the convex section B is blocked unidirectionally by a certain sub-convex section of the non-convex section a, and another sub-convex section of the non-convex section a is blocked unidirectionally, the convex section B is blocked bidirectionally by the non-convex section a; when all the sub-convex regions of the convex region B and the non-convex region A are not shielded, the scene A and the scene B are not shielded at the user position; when any sub-convex section of the convex section B and the non-convex section A is the B unidirectional block A, and all other sub-convex sections of the convex section B and the non-convex section A cannot generate the A unidirectional block B, the user position is the scene B unidirectional block A; when any sub-convex section of the convex section B and the non-convex section A is the A unidirectional block B, and all other sub-convex sections of the convex section B and the non-convex section A cannot generate the B unidirectional block A, the user position is the scene A unidirectional block B.

3) User position intervals corresponding to different shielding relations between non-convex intervals and non-convex intervals

The combined scene includes scene a and scene B, wherein the display intervals of scene A, B in the combined scene are all non-convex intervals. For a user in a combined scene, the occlusion relationship of scene a and scene B includes the following 4 cases: the scene A and the scene B are blocked in two directions, the scene A is blocked in one direction, the scene B is blocked in one direction, the scene A is blocked in one direction, and the scene B and the scene A are not blocked.

The non-convex section A and the non-convex section B can be divided into a plurality of sub-convex sections, and the user position section corresponding to the shielding relation of each sub-convex section of the non-convex section A and each sub-convex section of the non-convex section B can be obtained according to the user position section corresponding to the shielding relation of the convex sections. When a certain user position in the combined space is located, a certain sub-convex section of the non-convex section B is blocked in one direction by a certain sub-convex section of the non-convex section A, and a certain sub-convex section of the non-convex section A is blocked in two directions by a certain sub-convex section of the non-convex section B, the non-convex section B and the non-convex section A exist; when all the sub-convex regions of the non-convex region B and all the sub-convex regions of the non-convex region A are not shielded, at the user position, the scene A and the scene B are not shielded; when a certain sub-convex section of the non-convex section B and any sub-convex section of the non-convex section A are B unidirectional shielding A, and all sub-convex sections of the convex section B and the non-convex section A cannot be A unidirectional shielding B, the user position is scene B unidirectional shielding A; when a certain sub-convex section of the non-convex section B and any sub-convex section of the non-convex section A are A unidirectional shielding B, and all sub-convex sections of the non-convex section A and any sub-convex section of the convex section B cannot be B unidirectional shielding A, the user position is scene A unidirectional shielding B. The method for acquiring the user position interval corresponding to different shielding relations between the non-convex interval and the convex interval can be specifically expanded, and is not described herein.

In the embodiment of the invention, the imaging view angles and the resolutions of all images are defaulted to be the same, so that the pixels with the same image coordinates among different images have a shielding relationship; if the imaging field angle and the resolution are different between different images, the imaging field angle and the resolution between the images can be converted to be the same by interpolation or sampling, which is a common means for those skilled in the art.

First embodiment:

the combined scene of the embodiment is a virtual-real mixed combined scene, and the combined scene consists of 1 real scene s ₀ Is composed of N virtual sub-scenes, wherein N is a natural number greater than 1, and the virtual sub-scenes are s respectively ₁ 、s ₂ 、...、s _N . As shown in fig. 7, in this embodiment, the combined scene experience generating system includes: user interface 10, figureThe image synthesis module 20, the real scene imaging module 30, the combined scene parameter setting module 40, the occlusion relationship user location interval calculation module 50, the sub-scene occlusion relationship user location interval lookup table 60, the example of the 1 st virtual sub-scene (70-1), the example of the 2 nd virtual sub-scene (70-2), -the third and the example of the nth virtual sub-scene (70-N). Wherein the 1 st instance (70-1) of the virtual sub-scene is the virtual sub-scene s ₁ Is the virtual sub-scene s, the 2 nd virtual sub-scene (70-2) ₂ The instance (70-N) of the nth sub-scene is the virtual sub-scene s _N Is an example of (a). For video perspective MR head display, the real scene imaging module 30 generates a binocular stereoscopic experience picture of a real scene and a corresponding depth image of the user through photoelectric devices such as a camera module on the user MR head display terminal, and for optical perspective MR head display, the real scene imaging module 30 does not need to generate a binocular stereoscopic experience picture but needs to generate a binocular stereoscopic depth image through photoelectric devices such as a camera on the user MR head display terminal (the binocular stereoscopic picture can be considered to be generated, and only the color value of each pixel in the binocular stereoscopic picture image is 0, namely a completely black image). The user sends a command to the combined scene parameter setting module 40 through the user interface 10 to set parameters such as imaging intervals of each sub-scene, display intervals of each sub-scene in the combined scene, and the system may also send a command to the combined scene parameter setting module 40 to set parameters of the combined scene, and the set parameters may be sent to the instances of each virtual sub-scene, the occlusion relationship user position interval calculating module 50, and the user interface 10. The occlusion relation user position interval calculation module 50 obtains the user position interval corresponding to the occlusion relation between the sub-scenes from the combined scene parameter setting module 40 after the display interval of each sub-scene in the combined scene is obtained, and stores the user position interval in the sub-scene occlusion relation user position interval lookup table 60. Each virtual sub-scene instance receives user pose and interaction operation information from the user interface 10 in real time, responds to the interaction operation, and each virtual sub-scene searches the occlusion relation with other sub-scenes according to the user pose in real time in the sub-scene occlusion relation user position interval lookup table 60 to judge whether the occlusion relation with other sub-scenes is present or not Is unidirectional or non-unidirectional, and thus respectively sets whether each virtual sub-scene needs to synchronously generate a depth image when generating a user experience picture, and each sub-scene generates a user binocular stereoscopic experience picture in real time according to the imaging interval, the interpupillary distance and other parameters set by the combined scene parameter setting module 40. Each virtual sub-scene sends a binocular stereoscopic user experience picture or binocular stereoscopic user experience picture and a corresponding depth image to the image composition module 20. The image synthesis module 20 synthesizes the real scene stereoscopic imaging picture or the real scene binocular stereoscopic imaging picture read from the real scene imaging module 30 in real time with the corresponding depth image or the binocular stereoscopic depth image, synthesizes the user experience picture or the user experience picture sent by each virtual sub-scene with the corresponding depth image, and reads the occlusion relation between the sub-scene occlusion relation and other sub-scenes from the sub-scene occlusion relation user position interval lookup table 60 for the sub-scene with the occlusion relation of the sub-scene which is unidirectional occlusion or non-occlusion with other sub-scenes including the real scene during image synthesis. The image composition module 20 sends the composed image to the user interface 10 for display to the user. The user interface 10 determines whether the interactive operation is the interactive operation of the sub-scene according to whether the user interactive operation position parameter is in the display section of any virtual sub-scene, if so, converts the interactive operation into an interactive operation command under the corresponding sub-scene coordinate system, and sends the interactive operation command to the corresponding sub-scene for implementation.

In the embodiment of the invention, the real scene s is used for simplicity ₀ And combining scenesAnd sharing a coordinate system. As shown in fig. 1, the combined scene experience generating method of the present embodiment includes the following steps:

step S00, setting experience generation parameters of a combined scene, including setting imaging intervals of each virtual sub-scene, pupil distance of a user p in each virtual sub-scene, rotation translation transformation relation between the combined scene coordinate system and each virtual sub-scene coordinate system, display intervals of each virtual sub-scene in the combined scene, and calculating all sub-scenesUser position intervals corresponding to different shielding relations among the sub-scenes s _i And s _j ，s _i And s _j The user position intervals corresponding to the different shielding relations comprise: s is(s) _i Non-shielding shelter _j User location intervals, s _j Does not shade s _i User location intervals, s _i And s _j Bidirectionally shielding a user position interval;

step S10, generating experience pictures of all sub-scenes of the combined scene, wherein all arbitrary virtual sub-scenes S _i Acquiring the pose of the user p in the coordinate system in real time, and according to the real-time pose and s _i Is the imaging interval of (a) and the user p is s _i Pupil distance in scene space, generating user p versus s _i The experience picture of the user p is also judged, and the scene s is judged under the current real-time pose of the user p _i If not, generating a depth image corresponding to the experience picture, and if so, generating no depth image;

step S30, the combined scene receives the interactive input of the user p, judges whether the interactive input is the interaction of the sub-scene, and judges that the interactive input is the user p to any sub-scene S if the interactive input is judged _i Is then converted into a scene s _i Interactive input in coordinate system s _i Responding to the converted interactive input.

The specific implementation of each step is described in detail below:

The step S00 is specifically implemented as follows:

1) Setting combined scene experience generation parameters

In the combined scene parameter setting module 40, each combined scene parameter is set: setting each virtual sub-scene s ₁ 、s ₂ 、...、s _N The imaging intervals of (a) are omega respectively ₁ 、Ω ₂ 、...、Ω _N The method comprises the steps of carrying out a first treatment on the surface of the Setting a three-dimensional interval omega ' in a combined scene, wherein virtual scenes cannot be deployed in the three-dimensional interval omega ' cannot be deployed in the combined scene ' ₀ For example, a virtual scene cannot be deployed in an interval occupied by a wall body, equipment and other objects in the real scene, and the interval is used for representing a display interval of the real scene when judging the shielding relation between the virtual sub-scene and the real scene, but does not actually limit real scene imaging; setting the user p in each virtual sub-scene s ₁ 、s ₂ 、...、s _N The interpupillary distances of d are respectively ₁ 、d ₂ 、...、d _N The method comprises the steps of carrying out a first treatment on the surface of the The pupil distance of the user in the combined scene isSetting each virtual sub-scene s ₁ 、s ₂ 、...、s _N In a combined scene->The display intervals of (2) are omega 'respectively' ₁ 、Ω′ ₂ 、...、Ω′ _N The method comprises the steps of carrying out a first treatment on the surface of the Setting a rotation-translation scaling relationship from each virtual sub-scene coordinate system to a combined scene coordinate system, and for any virtual sub-scene s in the rotation-translation scaling relationship _i (i is more than or equal to 1 and less than or equal to N), scene s _i The rotation, translation and scaling relationship between the coordinate system and the combined scene coordinate system is that the rotation angles around the Z, X, Y three axes are respectively theta according to the sequence of Z, X, Y ⁱ 、β ⁱ 、α ⁱ Translation of +.>Z, X, Y scaling factor lambda for three axes _i ；

The parameters are not completely independent, and are neededThe following constraints are to be satisfied: enabling the pupil distance used by the terminal of the user p to generate a real scene stereoscopic experience picture or a depth image to be d ₀ (d ₀ As much as possible the true interpupillary distance of user p, the user is in the combined scenePupil distance->Pupil distance for arbitrary virtual sub-scene s _i There is->Arbitrary virtual sub-scene s _i Its imaging interval omega _i Any point b of (2) ₀ According to s _i Coordinate system to combined scene->After coordinate transformation is carried out on the rotation translation scaling relation of the coordinate system, the obtained point is +.>Belonging to omega' _i The method comprises the steps of carrying out a first treatment on the surface of the Display interval omega' ₁ 、Ω′ ₂ 、...、Ω′ _N Are not overlapped with each other, omega' ₁ 、Ω′ ₂ 、...、Ω′ _N Are all equal to omega' ₀ And do not overlap.

Still further, for s ₁ 、s ₁ 、s ₂ 、...、s _N Any scene s _i If it is in a combined sceneDisplay interval omega 'of (2)' _i For non-convex region, several sub-convex regions omega 'are set' _i,0 、Ω′ _i,1 、...、/>Completely cover omega' _i I.e. +.>At the same time require->As small as possible, and all sub-convex sections do not intersect with the display sections and sub-convex sections of other sub-scenes.

2) Calculating a user location interval of occlusion relationship

In the embodiment of the invention, the scene is combinedFrom real scene s ₀ And virtual sub-scene composition, wherein the virtual sub-scene composition set Λ= { s ₁ s ₂ … s _N }。s ₁ 、s ₂ 、...、s _N In a combined scene->The display intervals of (2) are omega 'respectively' ₁ 、Ω′ ₂ 、...、Ω′ _N In addition, omega 'when judging the shielding relation between the virtual sub-scene and the real scene' ₀ The display interval of the virtual training scene is used to represent the display interval of the real scene. For s ₀ 、s ₁ 、s ₂ 、...、s _N Any pair of scenes s _i And s _j Calculated in a combined sceneThe user position intervals corresponding to different shielding relations among the two user position intervals comprise the following steps:

The step S001 is specifically implemented as follows:

if s is _i And s _j In a combined sceneDisplay interval omega 'of (2)' _i With omega' _j Are convex regions, due to omega' _i With omega' _j Is arranged so as not to overlap, so that there must be omega' _i With omega' _j The separation surface is present, s _i And s _j Between which is a constant unidirectional barrier. If omega' _i With omega' _j When the requirements of the convex areas are not met, judging whether the convex areas are blocked in one direction or not is as follows: if omega' _i Non-convex region and omega' _j For convex regions, first, a surrounding non-convex region Ω 'is obtained' _i Is +.>If->With omega' _j Non-overlapping, then->For omega' _j Necessarily have a separation surface, s _i And s _j Constant unidirectional shielding is arranged between the two; if omega' _j Omega 'for non-convex region' _i For convex regions, surrounding non-convex regions Ω 'are also obtained' _j Is +.>If omega' _i And->Non-overlapping, then omega' _i For->Necessarily have a separation surface, s _i And s _j Constant unidirectional shielding is arranged between the two; if omega' _i With omega' _j Are all non-convex regions, and respectively surround the non-convex regions omega' _i With omega' _j Is +.>And->If->And->Non-overlapping, then->For->Necessarily have a separation surface, s _i And s _j Between which is a constant unidirectional barrier. The method for obtaining the minimum convex section surrounding the non-convex section can be manually determined or automatically determined by a system, and the automatic determination method of the system is as follows: in omega' _i For the non-convex section as an example, traverse Ω' _i Any pair of boundary points on the boundary, and any point in all points between the boundary points if not belongs to omega' _i All incorporate omega' _i Thus obtaining

If s _i And s _j In a combined sceneDisplay interval omega 'of (2)' _i With omega' _j Satisfying the above situation, s is obtained according to the method in the patent application "ubiquitous training campus construction method, system and storage Medium based on XR technology" (application number: CN 202210908795.0) _i And s _j A separation plane Γ therebetween _i,j This partition divides the combined scene space into 2 intervals +.>And->Wherein s is _i Display interval->s _j Display interval->When the user is +.>Scene s _i Unidirectional shielding s _j Or no shielding condition exists between the two, and the case of +.>Is s _j Does not shade s _i Is a user location interval of (1); when the user is +.>Scene s _j Shielding s _i Or no shielding condition exists between the two. Thereby completing s _i And s _j User position interval calculation corresponding to different shielding relations, & lt/L >Is s _i Does not shade s _j Is a user location interval of (a). And (5) ending the calculation.

In addition to the above, s _i And s _j Non-constant unidirectional occlusion, proceed to step S002.

Step S002 is specifically implemented as follows:

s _i and s _j Non-constant unidirectional occlusion, because s is set when setting the combined scene parameters _i And s _j In a combined sceneDisplay interval omega 'of (2)' _i With omega' _j Are not overlapped, so omega' _i With omega' _j Either 1 or both are non-convex. For convenience of description, when Ω' _i When the scene is a convex section, the scene is also considered to be s when the scene parameters are set _i The sub-convex regions are set, but the sub-convex regions have only 1 omega' _i,0 And omega' _i,0 ＝Ω′ _i Also when Ω' _j When the scene is a convex section, the scene is also considered to be s when the scene parameters are set _j 1 Ω 'is set' _j,0 ＝Ω′ _j Is a sub-convex section of (a).

Thus, regardless of Ω' _i With omega' _j In the case of 1 or both being non-convex, according to the combined fieldJing Canshu set the result to scene s _i The set sub-convex interval set isFor scene s _j The set subconvex interval set is +.>Traversing lambda _i Respectively solving Λ in all sub-convex regions _i Omega 'of any subconvex interval' _i,k And lambda _j User location intervals corresponding to different shielding relations among all the sub-convex intervals comprise: s is(s) _i Unidirectional shielding s _j User location intervals, s _j Unidirectional shielding s _i User location intervals, s _i And s _j No occlusion of the user location interval occurs.

For lambda _i Omega 'of any subconvex interval' _i,k And lambda _j Omega 'of any subconvex interval' _j,l 1 double adjacent separation surface is calculated by iterative search algorithmSeparated in the region on both sides of the separating surface, Ω' _i,k One side is divided intoΩ′ _j,l One side interval is +.>The user position is +.>Omega' _i,k Omega 'of baffle' _j,l Or both are not blocked, the user position is +.>Omega' _j,l Omega 'of baffle' _i,k Or both are not shielded; if 1 new double-adjacent separation surface is calculated by iterative search algorithm>Separated in the region on both sides of the separating surface, Ω' _i,k One side is +.>Ω′ _j,k One side interval is +.>The user position is +.>Omega' _j,l Does not shade omega' _i,k The user position is +.>Omega' _i,k Does not shade omega' _j,l The method comprises the steps of carrying out a first treatment on the surface of the Further iterative searching may potentially search for more double-contiguous separation surfaces. Let co-search for n _ijkl A double adjoining partition surface, wherein omega' _j,l Does not shade omega' _i,k The user location intervals of (a) are respectivelyΩ′ _i,k Does not shade omega' _j,l The user location intervals of (a) are respectivelyLet Θ be the combined scene +.>The following are: />Is omega' _j,l Does not shade omega' _i,k User location interval,/, of (2)>Is omega' _i,k Does not shade omega' _j,l User location interval,/, of (2) >Is omega' _i,k With omega' _j,l No blocked zones occur.

Completion lambda _i All arbitrary sub-convex regions and Λ _j After the user position intervals with different shielding relations among all arbitrary sub-convex intervals are calculated, the following steps are:for scene s _j Is not covered by s _i The user location interval that is occluded is defined,for scene s _i Is not covered by s _j Blocked user position interval +.>Is s _i And s _j And a bidirectional shielding interval. Handle->A user location interval lookup table 60 is stored for sub-scene occlusion relationships.

Wherein for lambda _i Omega 'of any subconvex interval' _i,k And lambda _j Omega 'of any subconvex interval' _j,l The double-adjacent separation surface obtained by calculation of the iterative search algorithm is a multi-independent variable optimization problem: the optimization objective is to minimize the separation plane and two convex intervals Ω' _i,k With omega' _j,l Is defined by the distance between the separating surface and the two convex regions omega' _i,k 、Ω′ _j,l Disjoint and omega' _i,k With omega' _j,l On different sides of the separation plane. In omega' _i,k With omega' _j,l Arbitrary partition surface A ₀ ·x+B ₀ ·y+C ₀ ·z＝D ₀ Parameter [ A ] of (B) ₀ B ₀ C ₀ D ₀ ]By setting a plurality of search directions for initial values of the independent variables of the partition, it is possible toFinding multiple double-contiguous separation surfaces, iterative search algorithms are well known to those skilled in the art and are not described in detail herein.

The step S10 is specifically implemented as follows:

at any time t _j User p is in a combined sceneThe pose in the coordinate system is [ W ] _j Q _j ]Wherein W is _j Is the position, Q _j For each virtual sub-scene s, for the attitude angle ₁ 、s ₂ 、...、s _N Receiving real-time pose [ W ] of user p from user interface _j Q _j ]According to the rotation, translation and scaling relation between the respective coordinate system and the combined scene coordinate system, calculating to obtain the pose of the user under each virtual scene coordinate system, and for s ₁ 、s ₂ 、...、s _N Any virtual scene s _i According to s _i Coordinate system and combined scene->Rotation translation scaling relation of coordinate system, and calculating to obtain user p in s _i The pose in the coordinate system is +.>Computing the pose of the user p in the virtual sub-scene coordinate system from the rotational translational scaling relationship of the coordinate system is well known to those skilled in the art and will not be described in detail herein. For s ₁ 、s ₂ 、...、s _N Any virtual scene s _i According to the s of the user _i Pupil distance d of (2) _i Posture->Imaging interval omega thereof _i Imaging scene content in the scene to generate an immersive binocular stereoscopic experience picture, wherein s _i Left and right eye images of the stereoscopic picture are respectively used +.>And->Representation, wherein->And->To identify which pixels correspond to empty scene content, e.g. to set the value of the empty scene content pixels in the experience picture image to a specific value τ _null (e.g., blue) or may be identified by a specific array, where each element in the array corresponds to a pixel in the image, and the element can only be given a value of 0 or 1 to identify whether the corresponding pixel images the scene content empty, in which embodiment the experience picture image ∈ >And->The value of the pixel whose middle scene content is empty is set to a specific value τ _null 。

For arbitrary virtual sub-scene s _i The scene instance real-time from sub-scene shielding relation user position interval lookup table 60 according to the user position W _j Retrieving occlusion relations with all other sub-scenes for any other sub-scene s _k If (if)S is then _i Unidirectional shielding s _k Or not block each other, if->S is then _k Unidirectional shielding s _i Or do not block each other, otherwise s _i And s _k And (5) bidirectional shielding. When s is _i And combined scene->When all other subscreens included are not bi-directional occlusions, then in generating s _i When the user binocular stereoscopic vision experiences the picture, the corresponding depth image does not need to be generated, otherwise, the corresponding depth image needs to be generated.

For any scene s in a virtual sub-scene set Λ which needs to generate a corresponding depth image while rendering and generating a binocular stereoscopic experience picture _k Rendering the left and right eye images of the generated stereoscopic experience picture asAnd->It is also necessary to generate a corresponding depth image +.>And->Pixels in which the imaged scene content is empty are represented by a specific value delta for the depth value, which may be infinity. Let Λ= { s ₁ s ₂ … s _N The scene set in which the corresponding depth image of the stereoscopic experience picture does not need to be generated is +. >The scene set needed to generate the stereoscopic experience picture corresponding depth image is Λ'.

The step S20 is specifically implemented as follows:

the specific way of synthesizing the image by the image synthesizing module 20 is as follows: at any time t _j The real scene s received from the real scene imaging module 30 ₀ The stereoscopic experience picture left and right eye depth image isAnd->Corresponding depth image +.>And (3) withFor any scene s in the set Λ _k The left eye and the right eye of the received stereoscopic vision picture are +.>And->The corresponding depth image is +.>And->(s _k In a combined scene->Depth image in coordinate system is +.>And->λ _k For sub-scene s _k Coordinate system to combined scene->Scaling factor of the coordinate system), for the set +.>Is +.>The left eye and the right eye of the received stereoscopic vision picture are +.>And->Make the combined scene after synthesis ∈ ->The user p stereoscopic experience picture left eye image is +.>The right eye image is +.>The corresponding depth images are +.>Constructing a two-dimensional matrix E for recording sub-scene numbers corresponding to pixels of the synthesized image ^L And E is connected with ^R ，E ^L Row and column number of (2)>Identical, E ^R Row and column number of (2)>Same, initial settingInitial setting E ^R And E is ^L Each element is 0, all sub-scenes in the set Λ' are traversed, for any sub-scene s therein _k Traversing- >All pixels, for any of which +.>If it isThen: /> E ^L (u v) =k, traversal +.>All pixels, for any of which +.>If->Then: E ^R (u v) =k; traversing set +.>For any of which sub-scenes +.>Walk->All pixels, for any of which +.>If it isOcclusion of relational user locations from sub-scenesInterval lookup table 60 queries +.>And->Is a shading relation (wherein->Sub-scene numbered EL (uv), if->There is->Shielding->Then:E ^L (u v)＝k ₁ the method comprises the steps of carrying out a first treatment on the surface of the Walk->All pixels, for any of which +.>If->Query +.>And->Is a shading relation (wherein->Is numbered E ^R Sub-scene of (u v), if ∈>There is->Shielding->Then:E ^R (u v)＝k ₁ . After the traversal is completed, the resulting image +.>And->Namely the user p is about the combined scene>Is a binocular stereoscopic experience picture. The binocular stereoscopic experience picture is displayed to the user p through the user interface.

The step S30 is specifically implemented as follows:

user p is in a combined scenario via user interface 10Performing interactive operation to generate an interactive operation command a, wherein the position parameter of the interactive operation is +.>The attitude angle parameter is +.>User interface traversal combined scene +.>All virtual sub-scenes in the line show interval omega' ₁ 、Ω′ ₂ 、...、Ω′ _N If it is judged that the coordinate value is +.>Is at display interval omega' _k In which Ω' _k For virtual sub-scene s _k According to s _k Coordinate system and combined scene->Rotation-translation scaling of the coordinate system, handle +.>Pose parameter in coordinate system>And->Conversion to s _k Pose parameters under a coordinate system, assigning the transformed pose parameters to pose parameter components of an interactive operation command a, and sending the interactive command a transformed with the pose parameters to a virtual sub-scene s _k Virtual sub-scene s _k In response to the interactive operation command.

The beneficial effects of the embodiment are as follows:

the embodiment is suitable for a virtual-real fusion combined scene, particularly suitable for an MR head display, even if a virtual sub-scene is in a display section of the combined scene, and is not constantly and unidirectionally shielded from other sub-scene display sections, particularly from a real scene display section, the combined scene can also have a certain section, when a user enters, the virtual sub-scene and the other sub-scene display sections are unidirectionally shielded, and the virtual sub-scene and the other sub-scene display sections can be accurately shielded from other contents of the combined scene to calculate image synthesis without transmitting a depth image to a terminal, so that the network bandwidth for transmitting images from a cloud to the user terminal is remarkably saved.

Second embodiment:

instead of combining only virtual sub-scenes, and not including real scenes, the system is constructed as shown in fig. 8, except that the real scene imaging module 30 is omitted compared with the system diagram of the first embodiment of fig. 7. The experience generation method of the present embodiment is substantially the same as that of the first embodiment except for subtracting out the calculation work related to the real scene, and differs in that: when the image synthesis is performed at step S20, when Λ' is not empty,for any virtual sub-scene in Λ',and->Is s _k2 User binocular stereoscopic experience picture generated by scene instance and corresponding depth image>And->Initial setting->Initial setting E ^R And E is ^L Each element is k ₂ The method comprises the steps of carrying out a first treatment on the surface of the When Λ' is empty, ++>Is->Is->And->Is->User binocular stereoscopic experience picture generated by scene instance, initial setting +.>Initial setting E ^R And E is ^L Each element is k ₃ . After the initial setting is completed, the method of image composition in step S20 is the same as that of the first embodiment.

The beneficial effects are that:

a Virtual Reality (VR) terminal may be adapted.

Third embodiment:

as shown in fig. 9, compared with the system configuration of the second embodiment of fig. 8, the XR combined scene experience generation method of the present embodiment is the same as that of the second embodiment except that the image synthesis module 20 is deployed from the user terminal to the cloud.

The beneficial effects are that:

compared with the second embodiment, the computing work of image synthesis is put on the cloud, so that the computing amount of a user terminal is reduced, and the synthesized image is only transmitted from the cloud, so that the network bandwidth from the cloud to the user terminal can be further saved, and the cost is the bandwidth for transmitting the image inside the cloud is increased.

Fourth embodiment:

a person in a real scene may enter a display interval of a virtual sub-scene in a combined scene, if the entered virtual sub-scene does not generate a corresponding depth image when generating an experience picture, the person and the display content of the virtual sub-scene are possibly shielded to calculate distortion, in order to solve the distortion problem, the first embodiment is improved in a specific improvement mode that: when judging that the person in the real scene enters any virtual sub-scene s _i After the display section of the combined scene, if s _i If no depth image is generated in step S10, setting S _i It is necessary to generate the user p stereoscopic experience picture at the same timeForming corresponding depth image when people leave s in the real scene _i Resetting s after the display interval of the combined scene _i No corresponding depth image is generated while generating the user p stereoscopic experience picture, so that in step 10, s _i The generation of the depth image is stopped. Wherein, judging whether the person in the real scene enters the virtual scene sub-scene s _i In the display interval omega 'of the combined scene' _i The method comprises the following steps: calculating the combination scene of the real person from the real scene user experience picture by using a computer vision algorithmOr calculate the position of the real person figure in the combined scene +.>Occupied space, when real person is in the combined scene +.>Is positioned at omega' _i Or the real person shape in the combined scene +.>Occupied space and omega' _i When intersecting, the person in the real scene enters a virtual scene sub-scene s _i In the display interval omega 'of the combined scene' _i 。

The beneficial effects are that:

the occlusion calculations of the real person and the virtual sub-scene are not distorted.

In addition, in order to achieve the above objective, the present invention also proposes an XR combined scene experience generation system, which is characterized in that the system comprises a memory, a processor and an XR combined scene experience generation method program stored on the processor, wherein the XR combined scene experience generation method program is executed by the processor to perform the steps of the method as described above.

The invention provides an XR combined scene experience generation method, an XR combined scene experience generation system and a storage medium, wherein the XR combined scene experience generation method comprises the following steps: step S10, generating experience pictures of all sub-scenes of the combined scene, wherein all arbitrary virtual sub-scenes S _i Acquiring the pose of the user p in the coordinate system in real time, and according to the real-time pose and s _i Is the imaging interval of (a) and the user p is s _i Pupil distance in scene space, generating user p versus s _i Is an experience picture of the computer; step S20, each sub-scene experience picture is subjected to shielding calculation to be synthesized into a combined scene experience picture of the user p, and the combined scene experience picture is displayed to the user p for viewing, wherein any sub-scene S which is one-way shielding or non-shielding with all other sub-scenes is selected _i When experiencing picture composition, when s _i Experience picture image arbitrary pixel and arbitrary sub-scene s _j When the shielding relation exists among the picture image pixels, the method for calculating the shielding relation among the pixels is as follows: according to the real-time position of the user p in the combined scene, s is obtained _i And s _j The one-way occlusion relation between the pixels is s _i And s _j Is a one-way occlusion relationship; if the combined scene includes a real scene, the sub-scenes of the combined scene include a real scene and a virtual sub-scene, and each sub-scene may be a non-convex section in a display section of the combined scene. According to the technical scheme, under the condition that the combined scene does not meet the condition that separation surfaces exist among all sub-scenes, under the real-time pose of a user, whether each sub-scene is one-way blocked or not blocked with other sub-scenes is judged, and for the sub-scenes which are one-way blocked or not blocked with other sub-scenes, when the user experience picture is rendered, the depth image corresponding to the experience picture does not need to be generated and transmitted, and the depth image can be correctly synthesized with the experience picture images of other sub-scenes according to the one-way blocking relation, so that the bandwidth required for transmitting the depth image is remarkably saved, the calculated amount of image synthesis is reduced, and the method has clear economy Benefit is obtained.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather to utilize the equivalent structures or equivalent processes disclosed in the present specification and the accompanying drawings, or to be directly or indirectly applied to other related technical fields, which are all encompassed by the present invention.

Claims

1. An XR combined scene experience generation method, comprising the steps of:

2. The method of generating as recited in claim 1, further comprising the step of setting a combined scene experience generation parameter including setting an imaging interval of each virtual sub-scene, the user p being present, before said step S10Pupil distance of each virtual sub-scene, rotation translation transformation relation between the combined scene coordinate system and each virtual sub-scene coordinate system, display interval of each virtual sub-scene in the combined scene, calculation of user position interval corresponding to different shielding relation among all sub-scenes, and for any sub-scene s _i And s _j ，s _i And s _j The user position intervals corresponding to the different shielding relations comprise: s is(s) _i Non-shielding shelter _j User location intervals, s _j Does not shade s _i User location intervals, s _i And s _j And the user position interval is blocked in two directions.

3. The method of claim 2, wherein step S20 is followed by step S30, wherein the combined scene accepts user p interaction input, determines whether the interaction input is an interaction with sub-scenes, and if the interaction input is determined to be user p with any sub-scene S _i Is then converted into a scene s _i Interactive input in coordinate system s _i Responding to the converted interactive input.

4. A method of generating as claimed in claim 3, characterized in that step S00 further comprises: for arbitrary virtual sub-scene s _i If it is in a combined sceneDisplay interval omega 'of (2)' _i For non-convex region, several sub-convex regions omega 'are set' _i,0 、Ω′ _i′1 、...、/>Completely cover omega' _i I.e. +.>At the same time require->As small as possible, and all sub-convex sections do not intersect with the display sections and sub-convex sections of other sub-scenes.

5. The method of claim 4, wherein for any sub-scene pair s _i And s _j Step S00, calculating the user position interval corresponding to different shielding relations comprises the following steps:

s002, if S _i In a combined sceneDisplay interval omega 'of (2)' _i For convex regions, calculate Ω' _i And s _j The user position interval corresponding to each shielding relation of each sub-convex interval is further calculated to obtain s _i And s _j User position intervals corresponding to each shielding relation, wherein s is satisfied _i Does not shade s _j The user position interval of all the sub-convex intervals is s _i Does not shade s _j Interval satisfies s _j All sub-convex regions do not shade s _i The user location interval of (2) is s _j Does not shade s _i Section, if s _i For non-convex regions, traverse s _i All sub-convex regions are respectively calculated and s _j Corresponding to each shielding relation of the system, thereby further calculating s _i And s _j User position intervals corresponding to each shielding relation, wherein s is satisfied _i All sub-convex regions do not shade s _j The user location interval of (2) is s _i Does not shade s _j Interval satisfies s _j Does not shade s _i User location for all sub-convex regionsInterval is s _j Does not shade s _i Interval.

6. The method according to claim 5, wherein said step S10 further judges the scene S under the current real-time pose of the user p _i If not, generating a depth image corresponding to the experience picture is also needed, and if so, generating no depth image is needed.

7. The method according to claim 6, wherein said step S10 judges the scene S under the current real-time pose of the user p _i The way of whether all other sub-scenes are unidirectional occlusion or non-occlusion is as follows: the real-time position of the user p in the combined scene is W _j According to the user position intervals corresponding to different shielding relations among all the sub-scenes calculated in the step S00, respectively searching out W _j S where is located _i User position interval corresponding to different shielding relations of all other sub-scenes, when W _j S where is located _i If the user position interval corresponding to the different shielding relations of all other sub-scenes is not the bidirectional shielding user position interval, under the current real-time pose of the user p, the scene s _i Is one-way occlusion or non-occlusion with all other sub-scenes.

8. The method according to any one of claims 1-7, wherein step S10 further comprises determining in real time whether the person in the real scene enters the display section of the virtual sub-scene in the combined scene, and if it is determined that the person in the real scene enters any virtual sub-scene S _i After combining the display intervals of the scene, if s _i If the scene is one-way occlusion or non-occlusion with all other sub-scenes, setting s _i It is necessary to generate corresponding depth images while generating the user p stereoscopic experience picture when the person in the real scene leaves s _i Resetting s after the display interval of the combined scene _i The corresponding depth image is not generated while the user p stereoscopic experience picture is generated.

9. An XR combined scene experience generation system, comprising a memory, a processor, and an XR combined scene experience generation method program stored on the processor, which when executed by the processor performs the steps of the method of any one of claims 1 to 8.

10. A computer readable storage medium, characterized in that it has stored thereon a computer program which, when invoked by a processor, performs the steps of the XR combined scene experience generation method of any one of claims 1 to 8.