CN115576427A - XR-based multi-user online live broadcast and system - Google Patents

Abstract

The embodiment of this specification provides an XR-based multiplayer online live broadcast method and system. The method includes: establishing a communication connection with the terminals of at least two participants; The virtual character corresponding to each participant; through the preset 3D coordinate position algorithm, based on the obtained position data of the participant in the actual space, determine the position information of the virtual character corresponding to the participant in the virtual space; in the virtual space Display the avatar based on the location information of the avatar; obtain the shared data uploaded by the participants, and display the shared data in the virtual space.

Description

An XR-based multiplayer online live broadcast and system

Case Description

This application is a divisional application filed against a Chinese application with a filing date of September 28, 2022, an application number of 2022111915615, and an invention title of "An XR-Based Multi-person Collaboration Method and System".

technical field

This specification relates to the field of communication technology, and in particular to a method and system for multiplayer online live broadcast based on XR.

Background technique

In the current multi-person communication scene, due to time cost and transportation cost, it is often impossible to attend important meetings at any time because of being in a different place. However, in the current multi-person remote communication solutions in the market, only computers and mobile phones can be used to display flat images for explanation and communication of things.

Therefore, it is hoped to provide a method and system for multi-person collaboration based on XR, which can provide a more direct and effective communication method for the collaboration of participants in different places.

Contents of the invention

One of the embodiments of this specification provides an XR-based multiplayer online live broadcast method, including: establishing a communication connection with the terminals of at least two participants; creating a virtual space in which each of the at least two participants The virtual character corresponding to the participant; through the preset 3D coordinate position algorithm, based on the obtained position data of the participant in the actual space, determine the position information of the virtual character corresponding to the participant in the virtual space; in the virtual space based on the virtual The location information of the characters displays the virtual characters; obtains the shared data uploaded by the participants, and displays the shared data in the virtual space.

One of the embodiments of this specification provides an XR-based multiplayer online live broadcast system, including: a connection module, used to establish a communication connection with the terminals of at least two participants; a positioning module, used to create a virtual space, in the virtual space Create a virtual character corresponding to each participant in the at least two participants; the positioning module is further used to determine the virtual character corresponding to the participant based on the acquired position data of the participant in the actual space through a preset 3D coordinate position algorithm. The location information of the character in the virtual space; and it is used to display the virtual character based on the location information of the task in the virtual space; the download module is used to obtain the shared data uploaded by the participants; the display module is used to display the shared data in the virtual space data.

One of the embodiments of this specification provides an XR-based multiplayer online live broadcast device, which includes: at least one storage medium storing computer instructions; at least one processor executing computer instructions to realize the above-mentioned XR-based multiplayer Human online live broadcast method.

One of the embodiments of this specification provides a computer-readable storage medium, the storage medium stores computer instructions, and when the computer reads the computer instructions, the computer executes the XR-based multiplayer online live broadcast method described above.

Description of drawings

This specification will be further illustrated by way of exemplary embodiments, which will be described in detail with the accompanying drawings. These examples are non-limiting, and in these examples, the same number indicates the same structure, wherein:

FIG. 1 is a schematic diagram showing an application scenario of an XR-based multi-person collaboration system according to some embodiments of the present invention;

Fig. 2 is an exemplary module diagram of an XR-based multi-person collaboration system according to some embodiments of the present specification;

Fig. 3 is an exemplary flow chart of an XR-based multi-person collaboration method according to some embodiments of this specification;

Fig. 4 is a flow chart of an exemplary method for determining position information of a participant in a virtual space according to some embodiments of the present specification;

Fig. 5 is an exemplary flow chart of an XR-based multiplayer online live broadcast method according to some embodiments of this specification;

Fig. 6 is an exemplary flowchart of real-time updating of location information according to some embodiments of this specification;

Fig. 7 is an exemplary flow chart of determining the presentation priority of sub-action information according to some embodiments of this specification;

Fig. 8 is an exemplary flowchart of a data processing method for XR according to some embodiments of the present specification;

Fig. 9 is an exemplary flow chart of displaying content to be marked according to some embodiments of this specification;

Fig. 10 is an exemplary flow chart of determining predicted display content according to some embodiments of the present specification.

detailed description

In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the following briefly introduces the drawings that need to be used in the description of the embodiments. Apparently, the accompanying drawings in the following description are only some examples or embodiments of this specification, and those skilled in the art can also apply this specification to other similar scenarios. Unless otherwise apparent from context or otherwise indicated, like reference numerals in the figures represent like structures or operations.

It should be understood that "system", "device", "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts, parts or assemblies of different levels. However, the words may be replaced by other expressions if other words can achieve the same purpose.

As indicated in the specification and claims, the terms "a", "an", "an" and/or "the" are not specific to the singular and may include the plural unless the context clearly indicates an exception. Generally speaking, the terms "comprising" and "comprising" only suggest the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list, and the method or device may also contain other steps or elements.

The flowchart is used in this specification to illustrate the operations performed by the system according to the embodiment of this specification. It should be understood that the preceding or following operations are not necessarily performed in the exact order. Instead, various steps may be processed in reverse order or simultaneously. At the same time, other operations can be added to these procedures, or a certain step or steps can be removed from these procedures.

Fig. 1 is a schematic diagram showing an application scenario 100 of an XR-based multi-person collaboration system according to some embodiments of the present invention. XR (Extended Reality), also known as extended reality, is a general term for various new immersive technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR). XR can combine reality and virtuality through computers to create a virtual space that can interact with humans and computers.

As shown in FIG. 1 , an application scenario 100 of an XR-based multi-person collaboration system may include a processing device 110 , a network 120 , a storage device 130 , a terminal 140 and a data collection device 150 . Components in the application scenario 100 of the XR-based multi-person collaboration system may be connected in one or more different ways. For example, data collection device 150 may be connected to processing device 110 via network 120 . For example, as shown in FIG. 1 , data collection device 150 may be directly connected to processing device 110 .

In some embodiments, the application scenario 100 of the XR-based multi-person collaboration system may include a scenario where multiple people who are not in the same physical space need to collaborate. For example, the application scenario 100 may include academic conferences, remote consultations, teaching and training, surgical guidance, and live broadcasting. The XR-based multi-person collaboration system can create a virtual space. The application scenario 100 can be realized through a virtual space. For example, in the scenario of surgical guidance, the medical staff participating in the operation can interact and communicate in the virtual space, share the patient's information recorded by the medical equipment, and also conduct live broadcasts with the experts in the virtual space to assist the experts in remote operation guidance. Furthermore, in the virtual space, 3D models can also be shared, such as heart models, etc., and the models can be disassembled, and video images related to surgery can also be presented in the virtual space. Devices and other terminal devices can share data information.

The data acquisition device 150 can be configured as a device for acquiring audio and video data related to the participants and the space where the participants are located. The data collection device 150 may include a panoramic camera 151, a common camera 152, a motion sensor (not shown in the figure) and the like.

Processing device 110 may process data and/or information obtained from storage device 130 , terminal 140 , and/or data collection device 150 . Processing facility 110 may include a server data center. In some embodiments, processing device 110 may host a simulated virtual world, or meta-domain for terminal 140 . For example, processing device 110 may generate the participant's location data based on images of the participant collected by data collection device 150 . For another example, the processing device 110 may generate the participant's location information in the virtual space based on the participant's location data.

In some embodiments, processing device 110 may be a computer, a user console, a single server or a group of servers, or the like. Server groups can be centralized or distributed. For example, a designated area of a meta domain can be simulated by a single server. In some embodiments, processing device 110 may include multiple simulation servers dedicated to physics simulations to manage interactions and handle collisions between characters and objects in the metaverse.

In some embodiments, the processing device 110 may be implemented on a cloud platform. For example, cloud platforms may include private clouds, public clouds, hybrid clouds, community clouds, distributed and inter-cloud clouds, multi-cloud, etc., or combinations thereof.

In some embodiments, the processing device 110 may include a storage device dedicated to storing data related to objects and characters in the metaworld. Data stored in the storage device may include object shapes, avatar shapes and appearances, audio clips, metaworld-related scripts, and other metaworld-related objects. In some embodiments, processing device 110 may be implemented by a computing device having a processor, memory, input/output (I/O), communication ports, and the like. In some embodiments, the processing device 110 may be implemented on a processing circuit (eg, processor, CPU) of the terminal 140 .

Terminal 140 may be a device that allows a user to participate in a virtual reality experience. In some embodiments, the terminal 140 may include a VR helmet, VR glasses, a VR patch, a stereo head-mounted display or the like, a personal computer (personal computer, PC), a mobile phone, or any combination thereof. For example, terminal 140 may include Google Glass ^™ , OculusRift ^™ , Gear VR ^™ , etc. Specifically, the terminal 140 may include a display device 141 on which virtual content may be presented and displayed. The user can view virtual content (for example, content to be marked, marking information, etc.) through the display device 141 .

In some embodiments, a user may interact with virtual content through the display device 141 . For example, when the user wears the display device 141, the user's head movement and/or gaze direction can be tracked, thereby presenting virtual content in response to changes in the user's position and/or orientation, providing immersive and convincing visual effects that reflect changes in the user's perspective. virtual reality experience.

In some embodiments, terminal 140 may further include an input component 142 . The input component 142 enables user interaction with virtual content displayed on the display device 141 . Wherein, the virtual content may include data information uploaded by participants. For example, the input component 142 may include a touch sensor, a microphone, etc. configured to receive user input that may be provided to the terminal 140 and control the virtual world by changing visual content presented on the display device. In some embodiments, user input received by the input component may include, for example, touch, voice input, and/or gesture input, and may be sensed by any suitable sensing technology (eg, capacitive, resistive, acoustic, optical). In some embodiments, input component 142 may include a gamepad, glove, stylus, game console, or the like.

In some embodiments, display device 141 (or processing device 110 ) may track input component 142 and render virtual elements based on the tracking of input component 142 . A virtual element may include a representation of the input component 142 (eg, an image of a user's hand, fingers). The virtual element may be rendered in a 3D location in the virtual reality experience that corresponds to the real location of the input component 142 .

For example, one or more sensors may be used to track input assembly 142 . The display device 141 may receive signals collected from the input component 142 by one or more sensors through a wired or wireless network. The signal may include any suitable information capable of tracking the input assembly 142, such as the output of one or more inertial measurement units (e.g., accelerometers, gyroscopes, magnetometers) in the input assembly 142, the A Global Positioning System (GPS) sensor, or similar, or a combination thereof.

The signal may indicate the position (eg, in a three-dimensional coordinate) and/or orientation (eg, in a three-dimensional rotational coordinate) of the input component 142 . In some embodiments, the sensors may include one or more optical sensors for tracking input assembly 142 . For example, sensors may use visible light and/or depth cameras to locate input component 142 .

In some embodiments, input component 142 may include a haptic component that may provide tactile feedback to a user. For example, a haptic assembly may include multiple force sensors, motors, and/or actuators. The force sensor may measure the magnitude and direction of the force applied by the user and input these measurements to the processing device 110 .

Processing device 110 may translate the input measurements into motions of one or more virtual elements (eg, virtual fingers, virtual palms, etc.) that may be displayed on display device 141 . Processing device 110 may then calculate one or more interactions between one or more virtual elements and at least a portion of the participants, and output these interactions as computer signals (ie, signals representing feedback forces). The motors or actuators in the haptic components can apply feedback force to the user according to the computer signal received from the processing device 110, so that the participant feels the real tactile sensation of the object in the surgical instruction. In some embodiments, the magnitude of the feedback force can be set according to a default setting of the XR-based multi-person collaboration system or by a user or an operator, for example, a terminal device (eg, terminal 140 ) preset setting.

In some embodiments, the application scenario 100 of the XR-based multi-person collaboration system may further include an audio device (not shown) configured to provide audio signals to the user. For example, an audio device such as a speaker can play the voices of the participants. In some embodiments, the audio device may include an electromagnetic speaker (eg, moving coil speaker, moving iron speaker, etc.), a piezoelectric speaker, an electrostatic speaker (eg, condenser speaker), or the like, or any combination thereof. In some embodiments, an audio device may be integrated into terminal 140 . In some embodiments, the terminal 140 may include two audio devices respectively located on the left and right sides of the terminal 140 to provide audio signals to the left and right ears of the user.

The storage device 130 may be used to store data and/or instructions, for example, the storage device 130 may be used to store relevant information and/or data collected by the data collection device 150 . Storage device 130 may obtain data and/or instructions from, for example, processing device 110 or the like. In some embodiments, storage device 130 may store data and/or instructions that processing device 130 executes or uses to perform the exemplary methods described in this specification. In some embodiments, storage device 130 may be integrated on processing device 110 .

Network 120 may provide a conduit for information and/or data exchange. In some embodiments, the processing device 110 , the storage device 130 , the terminal 140 and the data collection device 150 may exchange information through the network 160 . For example, the terminal 140 may obtain data information and the like sent from the processing device 110 through the network 120 .

It should be pointed out that the above description of the application scenario 100 of the XR-based multi-person collaboration system is for illustrative purposes only, and is not intended to limit the scope of the present disclosure. For example, the assembly and/or functions of the application scenario 100 of the XR-based multi-person collaboration system may vary or change according to specific implementation scenarios. In some embodiments, the application scenario 100 of the XR-based multi-person collaboration system may include one or more additional components (for example, storage devices, networks, etc.) and/or may omit the above-mentioned application scenario of the XR-based multi-person collaboration system 100 of one or more components. In addition, two or more components of the application scenario 100 of the XR-based multi-person collaboration system can be integrated into one component. One component of the application scenario 100 of the XR-based multi-person collaboration system can be implemented on two or more subcomponents.

Fig. 2 is an exemplary block diagram of an XR-based multi-person collaboration system 200 according to some embodiments of the present specification. In some embodiments, the XR-based multi-person collaboration system 200 may include a connection module 210 , a positioning module 220 , a download module 230 , a presentation module 240 , and a generation module 250 .

The connection module 210 can be used to establish communication connections with terminals of at least two participants.

The positioning module 220 can be used to determine the position information of at least two participants in the virtual space through a preset 3D coordinate position algorithm.

In some embodiments, the position information of the at least two participants in the virtual space is related to the position data of the at least two participants in the real space, and the position data of the at least two participants in the real space are passed through the terminals of the at least two participants Obtain.

In some embodiments, the positioning module 220 can be further used to create a virtual space; create a virtual character corresponding to each participant in the at least two participants in the virtual space, wherein the virtual character has initial position information in the virtual space; Obtain the position data of the participant in the actual space, and associate the position data with the position information of the corresponding virtual character in the virtual space; based on the position data of the participant, obtain the movement data of the participant in the actual space; through preset 3D coordinates The location algorithm updates the initial location information based on the mobile data, and determines the updated location information.

In some embodiments, the positioning module 220 may be used to create a virtual space, and to create an avatar corresponding to each of the at least two participants in the virtual space. In some embodiments, the positioning module 220 is further configured to determine the position information of the virtual character corresponding to the participant in the virtual space based on the acquired position data of the participant in the real space through a preset 3D coordinate position algorithm. In some embodiments, the positioning module 220 can be used to display the avatar in the virtual space based on the location information of the task.

In some embodiments, the positioning module 220 can be further used to scan the actual space where the participant is, and spatially locate the participant; for the participant who has completed the scanning, determine the real-time position data of the participant in the actual space; based on the real-time position data Determine the first movement information of the participant in the actual space; determine the initial position information of the virtual character in the virtual space; obtain the first action information of the participant in the actual space; the first action information includes sub-action information of various parts of the participant's body ; Synchronously update the second movement information and/or the second movement information of the avatar based on the first movement information and/or the first movement information through the preset 3D coordinate position algorithm.

In some embodiments, the positioning module 220 can be further used to determine at least one core body part of the participant based on the current scene; determine the display priority of the sub-action information of each part of the participant's body based on the at least one core body part; The display priority of the sub-action information determines the display parameters of the action information, and the display parameters include display frequency and display accuracy; based on the display parameters, the second action information of the avatar corresponding to the participant is synchronized.

The download module 230 can be used to save data information uploaded by at least two participants, and provide data download service to at least two participants, wherein the data download service includes at least one of creating a data download channel and providing download resources.

In some embodiments, the download module 230 can be used to obtain shared data uploaded by participants.

The display module 240 can be used to display data information synchronously on the terminals of at least two participants.

In some embodiments, the terminal includes at least one of a VR display device, an AR display device, a mobile phone, and a PC computer.

In some embodiments, the display module 240 can be used to display shared data in a virtual space.

In some embodiments, the presentation module 240 can be used to create at least one second space and/or second window in the virtual space, wherein each of the at least one second space and/or second window is associated with a participant Correspondence; displaying the shared data of corresponding participants through the second space and/or the second window.

In some embodiments, the display module 240 can be used to display the content to be marked on the canvas, wherein the content to be marked is the data that has been marked and/or the original data that has not been marked; Marking information created on the canvas, where the marking information includes marking content and marking path; share the content to be marked and marking information to the terminals of other participants for display.

In some embodiments, the content to be marked is the content displayed in any window and any position of multiple windows on the terminal of the marking requester.

In some embodiments, the display module 240 can be further used to obtain the display settings of the mark requester, the display settings include real-time mark display and mark display after completion; based on the display settings, the content to be marked and its mark information are shared with other participants terminal for display.

In some embodiments, the display module 240 can be further used to determine the viewing angle information of each participant based on the position information of each participant among other participants; and determine the displaying angle information of each participant based on the viewing angle information of each participant content, and display it, the displayed content includes the content to be marked and/or marked information under the viewing angle information.

The generation module 250 may be configured to create a canvas within the virtual space in response to a request from an annotation requester.

It should be noted that the above description of the system and its modules is only for convenience of description, and does not limit the application to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to combine various modules arbitrarily, or form a subsystem to connect with other modules without departing from this principle. For example, the connection module 210, the positioning module 220, the download module 230 and the display module 240 can be combined to form an XR-based multiplayer online live broadcast system. For another example, the presentation module 240 and the generation module 250 may be combined to form a data processing system for XR. Such deformations are within the protection scope of the present application.

Fig. 3 is an exemplary flow chart of a multi-person collaboration method according to some embodiments of this specification. As shown in FIG. 3 , the process 300 may include the following steps.

Step 310, establishing communication connections with the terminals of at least two participants. In some embodiments, step 310 may be performed by the connection module 210 .

A participant may refer to a person participating in a collaboration. Different collaboration scenarios require different participants. For example, in the operating room VR scene, participants may include operating personnel (for example, doctors participating in the operation), remote experts, and operating personnel. Among them, operating personnel can manage user rights and archive and trace back the guidance process.

Multi-person collaboration can be used for academic conferences, remote consultations, medical training, surgical live broadcast, medical device training, etc. When multi-person collaboration, real-time multi-person live broadcast, real-time annotation sharing, etc. can be realized. For details of multi-person real-time live broadcast, refer to descriptions in other parts of this specification, for example, FIGS. 5, 6, and 7. For details of real-time annotation sharing, refer to descriptions in other parts of this specification, for example, FIGS. 8 , 9 , and 10 .

In some embodiments, a server data center can be established, and participants realize communication connections by connecting participants' terminals to the server data center.

The server data center can be a platform equipped with multi-person collaborative instant communication. The server data center can include at least one server. The performance of the server can meet the requirements of multi-person collaborative operation, not only can the server data center provide multi-person and multi-terminal access, but also can ensure the stability and stability of multiple terminals when accessing. Real-time performance can also ensure the security and integrity of server data.

In some embodiments, the communication connection can be used for audio instant communication, video instant communication, multi-person virtual space technology exchange, and the like. Wherein, audio instant communication may include audio information recording, transmission and reception. Instant video communication can include the recording, decoding, transmission and reception of video information. Multi-person collaboration can be realized through multi-person virtual space technology communication. For example, during the live broadcast of surgery, experts from different places can be invited to provide remote assistance. For another example, the invited participant can view the live broadcast of the invited participant, communicate with other participants, and provide assistance. For another example, participants can also use the marking function to mark locally, and display the marked content to other participants in real time.

A participant's terminal may refer to a device used by a participant to participate in collaboration. In some embodiments, the participant's terminal may include a device for realizing the connection between the participant and the server data center and a data collection device. Data collection equipment is used to collect data such as audio and video in the actual space where the participants are located, such as panoramic cameras, ordinary cameras, AR glasses, mobile phones, motion sensors, depth cameras, etc. The participant terminal can also include a display device, which can display the data obtained from the server data center.

In some embodiments, the terminal includes at least one of a VR display device, an AR display device, a mobile phone, and a PC computer. For example, the equipment that realizes the connection between the participants and the server data center may include AR glasses, VR helmets, PCs, mobile phones, etc.

Step 320, determine the position information of at least two participants in the virtual space through a preset 3D coordinate position algorithm. In some embodiments, step 320 may be performed by the positioning module 220 .

A virtual space may refer to a space where virtual objects are displayed. The virtual space can be created based on the information of the actual space or based on the preset virtual information; for details about creating the virtual space, refer to the description in other parts of this specification, for example, FIG. 4 .

In some embodiments, the virtual space can correspond to different scenarios, for example, it can include academic conferences, teaching and training, case interrogations, operating room VR scenarios, surgical procedure details scenarios, pathological data sharing scenarios, surgical navigation information scenarios, patient vital signs data scenarios, etc. In the virtual space, the location information and data information of the participants can be displayed. For details about the location information and data information, refer to the introduction of other parts of this specification, for example, step 340 in FIG. 3 .

As an example only, in the detailed scene of the operation process, the surgeon can wear a terminal device and connect to the server data center, and can project the operation screen in the actual space seen by the surgeon to the virtual space and broadcast it to other remote experts and scholars in real time , other remote experts and scholars can watch and understand the details of close-range surgery in real time by connecting to the server data center, and surgeons can also communicate with remote experts in audio and video and obtain remote guidance.

For another example, in the teaching and training scene, teachers and students can join the virtual space through the participant terminal, and the teacher can conduct live training in the virtual space, and can import and share 3D models, images, text and other materials into the virtual space , teachers and students can walk and interact in the virtual space, and can also edit and mark the shared information.

In some embodiments, a spatial coordinate system can be set in the virtual space, and the spatial coordinate system can be used to represent the spatial position relationship of the virtual object in the virtual space. Multiple participants can communicate and interact in the same virtual space through the participant terminals.

In some embodiments, virtual objects may include space backgrounds, virtual characters, virtual windows, canvases, data information, and the like. In some embodiments, the virtual space may include a space background, which may be a real-time image of the actual space or other preset images. In some embodiments, virtual characters corresponding to each participant may be included in the virtual space. For details about the virtual character, refer to the introduction of other parts of this specification, for example, FIG. 4 .

In some embodiments, the virtual space may include multiple second windows and/or multiple second spaces. For details about the second window and/or the second space, refer to the introduction of other parts of this specification, for example, FIG. 5 . In some embodiments, the virtual space may include a canvas. For details about the canvas, please refer to the introduction of other parts of this specification, for example, Figure 8.

Location information may refer to information related to a participant's location and/or actions in a virtual space. The location information may include initial location information and real-time location information of the participant in the virtual space. The initial location information may refer to the initial location of each participant in the virtual space. For details about the initial location information, refer to the introduction of other parts of this specification, for example, FIG. 4 .

In some embodiments, the location information may include action information of the virtual character corresponding to the participant in the virtual space. The motion information may refer to body motion information generated by the participant in the actual space. For details about action information, please refer to the introduction of other parts of this manual. For example, Figure 6.

In some embodiments, the motion information may also include the head motion information of the avatar in the virtual space corresponding to the actual motion of the participant, which may be used to determine the viewing angle information of the participant in the virtual space. For details about viewing angle information, refer to descriptions in other parts of this specification, for example, FIG. 9 .

In some embodiments, the position information of the at least two participants in the virtual space is related to the position data of the at least two participants in the real space, and the position data of the at least two participants in the real space are passed through the terminals of the at least two participants Obtain.

A physical space may refer to a space where a participant is actually located. For example, a physical space may refer to an office, a study, an outdoor location, etc. where a participant is located.

Location data may refer to data related to a participant's location and/or movement in real space. In some embodiments, location data may include a participant's location and/or motion in a physical space. Wherein, the position may be represented by coordinates in real space. For example, the coordinates may be represented by coordinates composed of longitude and latitude or coordinate information based on other preset coordinate systems. In some embodiments, the location data may include the participant's coordinate position, movement speed, acceleration, movement of body parts, direction of the participant's terminal (that is, the orientation of the participant), and the like. The location data may include real-time location data.

The participant's location data can be determined by the positioning device and the data collection device (eg, camera, sensor, etc.) For example, based on received data from the positioning device, the location of the user can be determined. As another example, the movements of participants can be determined through cameras and sensors. Location data can be obtained by connecting with a positioning device, a data acquisition device in the actual space where the participant is located.

Exemplary positioning devices may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation System, Galileo Positioning System, Quasi-Zenith Satellite System (QZSS), Base Station Positioning System, Wi-Fi Positioning System.

In some embodiments, the position information of the participant in the virtual space may be determined based on the position data of the participant in the real space. For example, a database can be preset in the server data center, and the location data of the participants can be corresponding to the location information of the avatar in the database. The database can be established based on the corresponding relationship between location data and location information in historical data. The corresponding relationship between the location data and the location information can be determined by a 3D coordinate location algorithm.

In some embodiments, the position information can be updated based on the position data of the participants in the real space, so as to realize the synchronization of the position information of the participants in the real space and the virtual space. For details on updating the location information, refer to other parts of this specification, for example, FIG. 4 .

In some embodiments, the location data (for example, 3D coordinates) in the actual space can be converted into location information in the virtual space through a projection transformation matrix by using a 3D coordinate location algorithm. For example, coordinates in the real space can be converted into coordinates in the virtual space coordinate system.

In some embodiments, determining the position information of the at least two participants in the virtual space through a preset 3D coordinate position algorithm includes: creating a virtual space; creating in the virtual space corresponding to each participant in the at least two participants A virtual character, wherein the virtual character has initial position information in the virtual space; acquiring the position data of the participant in the real space, and associating the position data with the corresponding position information of the virtual character in the virtual space; based on the position of the participant Data, to obtain the movement data of the participants in the actual space; through the preset 3D coordinate position algorithm, the initial position information is updated based on the movement data, and the updated position information is determined. For details about determining location information, refer to other parts of this specification, for example, FIG. 4 .

In some embodiments, the position information of the avatar corresponding to the participant in the virtual space is determined based on the acquired position data of the participant in the virtual space through a preset 3D coordinate position algorithm, including: scanning the participant The actual space where the participant is located is spatially positioned; for the participant who has completed the scan, the real-time location data of the participant in the actual space is determined; the first movement information of the participant in the actual space is determined based on the real-time location data; The initial position information of the virtual space; the first action information of the participant in the real space is obtained; the first action information includes the sub-action information of each part of the participant's body; through the preset 3D coordinate position algorithm, based on the first movement information and/or Or the first action information synchronously updates the second movement information and/or the second action information of the avatar corresponding to the participant. Further, the location information is determined based on the second movement information and/or the second action information. For details on determining location information, refer to content in other parts of this specification, for example, FIG. 6 .

Step 330, save the data information uploaded by at least two participants, and provide data download service to at least two participants, wherein the data download service includes at least one of creating a data download channel and providing download resources. In some embodiments, step 330 may be performed by the download module 230 .

Data information may refer to information shared in a virtual space. For example, data information may include 3D models, videos, documents, operation manuals, etc. In some embodiments, the data information may include content to be marked and marking information. For details about the content to be marked and the marking information, refer to the description of other content in this specification, for example, FIG. 8 .

In some embodiments, the data information may be data uploaded by participants, or data retrieved from other platforms (for example, a network cloud platform). The data information can be saved in the storage device of the server data center. In response to the data request of the participant, the server data center can connect with other platforms and retrieve the corresponding data information, or retrieve the data information uploaded by the participant to the server data center, or retrieve the storage device stored in the server data center data information within.

The data download service may refer to a service for data downloading of information by connecting a communication module (for example, an LTE communication module) to a corresponding communication network (for example, a 4G network). Participants can obtain data information through the data download service.

In some embodiments, a download channel can be created between the server data center and the participant terminal. There can be multiple download channels, corresponding to each participant. Participants can obtain the required information data through the data download channel.

In some embodiments, data information can be classified (for example, according to data type) and stored in the storage device of the server data center, each type of data information corresponds to a data download channel, and in response to the type of data request of the participant, it can be Obtain corresponding data information from different data download channels.

Step 340, display data information synchronously on the terminals of at least two participants. In some embodiments, step 340 may be performed by the presentation module 240 .

In some embodiments, the data information can be displayed in the virtual space, and the participant terminal can be connected to the server data center to obtain the data information, and the data information can be displayed synchronously through the display device of the participant terminal.

In some embodiments, different display modes may be determined according to different participant terminals. For example, PCs and mobile phones may display data information through the screens of PCs and mobile phones, and AR glasses and/or VR helmets may display data information through a screen projected inside AR glasses and/or VR helmets.

By establishing a 3D virtual space and synchronously sharing data information in the 3D virtual space, the synchronization between local and remote places is realized, and the problems of number limit and venue limit of offline multi-person meetings are solved. Participants can communicate face-to-face through the virtual space, and more intuitively interact with the object information in the virtual space. Participants support more compatible platforms, and can join the discussion with different devices anytime, anywhere, reducing a lot of time The cost can efficiently and quickly form a collaborative team. At the same time, through the virtual space, records can be formed, which is convenient for other personnel to learn and refer to, summarize experience and even investigate and collect evidence.

Fig. 4 is a flowchart of an exemplary method for determining position information of a participant in a virtual space according to some embodiments of the present specification. In some embodiments, the process 400 may be performed by the positioning module 220 .

Step 410, creating a virtual space; creating a virtual character corresponding to each of the at least two participants in the virtual space, wherein the virtual character has initial position information in the virtual space.

In some embodiments, a coordinate system can be established in any actual space, the model data of the actual space model can be created based on the actual space coordinate system and the actual space scan data, and the actual space coordinate system corresponding to the actual space model can be established, according to the actual space coordinate system Describe the model data of the actual space model, establish a virtual space model corresponding to the actual space model through mapping, and establish a virtual space coordinate system corresponding to the virtual space model.

In some embodiments, the virtual space may be created based on the design, for example, the virtual space may be a designed virtual operating room or the like.

A virtual character may refer to a character image corresponding to a participant in a virtual space. When the participant connects to the server data center, the corresponding avatar can be assigned to the participant according to the default settings, or the participant can be provided with multiple candidate avatars that have been created, and the participant can choose one of the avatars to determine the corresponding avatar. of virtual characters. The location information of the corresponding participant can be displayed synchronously through the avatar. For example, if participant 1 clicks on virtual character 1, the participant moves to the left in the real space, and the corresponding virtual character 1 also moves to the left in the virtual space.

In some embodiments, the initial position information of the avatar may be determined according to preset rules. For example, each avatar is preset with an initial position. When the participant selects the corresponding virtual character, the corresponding initial position can be determined, or the participant can choose the initial position of the corresponding virtual character in the virtual space by himself.

Step 420, acquiring the position data of the participant in the real space, and associating the position data with the position information of the corresponding avatar in the virtual space.

In some embodiments, the location data of the participants in the actual space can be obtained by connecting with the positioning device and the data collection device.

In some embodiments, a storage device for each avatar may be set in the server data center. After the participant's location data is obtained, it can be stored in the storage device corresponding to the participant, and the server data center can convert the location data into the location information of the avatar through a preset 3D coordinate location algorithm.

Step 430, based on the location data of the participant, the movement data of the participant in the real space is obtained.

Movement data may refer to data related to the movement of participants in physical space. Movement data can include, among other things, the direction and distance a participant moves.

In some embodiments, the movement data of the participant can be determined based on the moving direction of the participant and the coordinate points before and after the movement. The distance moved can be determined based on the coordinates of the participant before and after the movement and a distance formula. For example, the participant's position data includes moving to the left, the coordinates before the movement are (1, 2), and the coordinates after the movement are (1, 3). According to the coordinates before and after the movement, the moving distance can be calculated as 1 meter, then the moving data To move 1 meter to the left.

Step 440, update the initial position information based on the movement data through a preset 3D coordinate position algorithm, and determine the updated position information.

In some embodiments, specifically, the virtual space needs to obtain the spatial position information of each participant. When the participant enters the virtual space, he is in the initial position. After the user has a relative displacement, the mobile data is uploaded to the server through the participant terminal Data center, the server data center uses the 3D coordinate position algorithm to convert the mobile data into the mobile information of the virtual space through the projection transformation matrix, and updates the position of the virtual character, and then synchronizes it to other participants' terminals. The real-time movement of the participant's avatar is seen in the virtual space.

Based on the position data of the participants in the actual space, the position information of the virtual characters of the participants in the virtual space can provide participants with more realistic, all-round, multi-level rendering and display effects, creating a sense of reality that is close to face-to-face communication , to increase the effectiveness of communication.

Fig. 5 is an exemplary flow chart of an XR-based multiplayer online live broadcast method according to some embodiments of this specification. As shown in Fig. 5, the process 500 may include the following steps.

Step 510, establishing communication connections with terminals of at least two participants. In some embodiments, step 510 may be performed by the connection module 210 .

For the definition and description of participants and terminals, and the method for establishing a communication connection, please refer to FIG. 3 and related descriptions.

Step 520, create a virtual space, and create a virtual character corresponding to each of the at least two participants in the virtual space. In some embodiments, step 520 may be performed by the positioning module 220 .

For the definition and description of the virtual space and the virtual character, as well as the method for creating the virtual space and the virtual character, please refer to FIG. 4 and its related description.

Step 530: Determine the position information of the avatar corresponding to the participant in the virtual space based on the acquired position data of the participant in the real space through a preset 3D coordinate position algorithm. In some embodiments, step 530 may be performed by the positioning module 220 .

For the definition and description of the 3D coordinate location algorithm, location data, and location information, please refer to FIG. 3 and related descriptions. For the method of determining and updating location information in real time, refer to FIG. 4 and its related descriptions.

Step 540, displaying the avatar in the virtual space based on the location information of the avatar. In some embodiments, step 540 may be performed by the positioning module 220 .

In some embodiments, according to the location information of the avatar, the corresponding created avatar may be displayed at the coordinates corresponding to the location information. When the position information changes, the display of the avatar changes in real time along with the change of the position information. For a detailed description of the avatar, refer to FIG. 4 and its related descriptions.

Step 550, acquire the shared data uploaded by the participants, and display the shared data in the virtual space. In some embodiments, step 550 may be performed by the download module 230 and the display module 240 .

Shared data refers to data uploaded by participants to the virtual space. Shared data has multiple manifestations, such as video, audio, image, and model, and the shared data in different application scenarios can be different.

For example, when displaying the VR scene of the operating room, the shared data may include the panoramic view of the operating room (such as the spatial design of the operating room, location orientation, instrument placement, etc.) data. For another example, when presenting details of an operation, the shared data may include close-up data of the operation process (such as doctor's hand operation, instrument operation, patient's operation site, etc.). For another example, when sharing pathological data, the shared data may include patient 3D image models, pathological pictures and videos, and so on. For another example, when displaying surgical navigation information, the shared data may include a picture of a surgical robot (such as a picture of surgery planning), and the like. For another example, when displaying patient vital sign data, the shared data may include vital sign monitoring data (such as vital signs, blood pressure, heart rate, electrocardiogram, blood oxygen saturation, etc.) during the patient's operation. For another example, when displaying a video image of a remote expert, the shared data may include video data, audio data, etc. of the expert captured by a camera. For another example, in interactive scenarios, shared data can include model manipulation, spatial annotation, group chat message boards, and private chat dialog boxes.

In some embodiments, step 550 further includes creating at least one second space and/or second window in the virtual space, wherein each of the at least one second space and/or second window corresponds to a participant; The shared data of the corresponding participant is displayed through the second space and/or the second window.

The second space and/or second window refers to a space and/or window created in the virtual space for displaying shared data. In some embodiments, the second space and/or the second window may be a window visible only to corresponding participants, for example, a private chat window between two participants. In some embodiments, the second space and/or the second window can be preset by the system, and can also be dragged and moved by the participants themselves.

In some embodiments, the participant can create the second space and/or the second window as needed, for example, the participant can choose to create the second space and/or the second window on the creation interface of the terminal. In some embodiments, the second space and/or the second window may also be created by the system by default.

In some embodiments, different second spaces and/or second windows may correspond to different participants and display different shared data. In some embodiments, the second space and/or the second window can realize one-to-one correspondence with the participants through dynamic allocation. For example, when the participant enters the virtual space, the system automatically creates a corresponding second space and the second window for the participant. /or a second window. For another example, the second space and/or the second window created by the participant corresponds to the participant himself.

In some embodiments, a participant can upload the data to be shared received or stored by the terminal to the server of the system, and other participants can download the shared data from the server as required. For details about the method for sharing data, refer to FIG. 3 and its related descriptions.

The XR-based multi-person online live broadcast method can realize the immersive interactive operation of the participants in the first-person perspective in the virtual space, increase the participants' learning interest and skill mastering proficiency, and solve the problem that the actual space, venue and number of people are limited and cannot achieve the best guidance Effects and other issues, and the data shared by participants can be visually displayed in the virtual space, which facilitates information synchronization among participants, thereby improving the efficiency and effectiveness of discussions and guidance.

Fig. 6 is an exemplary flowchart of real-time updating of location information according to some embodiments of the present specification. In some embodiments, the process 600 may be performed by the positioning module 220 .

In some embodiments, the participant terminal can scan the actual space where the participant is located to locate the participant; for the participant who has completed the scanning, the participant terminal can determine the real-time position data 610 of the participant in the actual space.

In some embodiments, the participant's terminal can scan the actual space where the participant is located based on multiple methods. For example, the participant can hold the terminal and use the terminal's depth camera to scan the surrounding environment of the actual space.

For another example, the participant terminal can obtain the anchor point of the actual space, perform multi-point space scanning on the special plane of the actual space, and keep the anchor point positioning of the actual space successful, that is, the space positioning is successful; if the scanning fails, it will remind that the space scanning is not Finish.

In some embodiments, the participant terminal can determine the real-time location data 610 of the participant in the actual space based on various methods. For example, the participant terminal may draw the spatial outline after scanning the actual space, and determine the real-time position data 610 of the participant according to the relative position information of the participant and the spatial reference object. For another example, the participant terminal can also directly obtain the real-time location data 610 of the participant according to positioning methods such as GPS.

In some embodiments, the participant terminal may determine the first movement information 620 of the participant in the actual space based on the real-time location data 610 .

The first movement information 620 refers to information generated by a participant moving in an actual space. The first movement information may include movement information of the participant's orientation, distance, height, etc. in the actual space.

In some embodiments, the first movement information 620 can be obtained in various ways. For example, the first movement information can be determined based on the real-time position data of the participant in the actual space. When the real-time position data of the participant changes, the participant terminal Corresponding first movement information may be calculated according to the changed data.

For another example, it may be determined by the anchor point positioning of the participant in the actual space, that is, it may be determined by the movement information of the anchor point. For more content about acquiring the first movement information, refer to FIG. 4 and related descriptions.

In some embodiments, the server may determine initial position information 660 of the avatar in the virtual space.

For example, after scanning the actual space where the participant is located, the participant terminal can determine the current participant's location data in the actual space, and the server can obtain the location data and map it to the initial location information of the avatar in the virtual space. For another example, the initial location information may be preset by the server. For more description on determining the initial position information of the avatar in the virtual space, please refer to FIG. 4 and related descriptions.

In some embodiments, the participant terminal can obtain the first action information 630 of the participant in the real space; the first action information 630 includes sub-action information of various parts of the participant's body,

The first motion information 630 refers to body motion information generated by the participant in the actual space. The first action information 630 may also include body action information (for example, stretching arms, shaking the body, walking, squatting), facial expression information (for example, blinking, opening the mouth) and the like. In some embodiments, the first action information 630 includes sub-action information of various body parts of the participant.

The sub-action information refers to specific action information of each part of the participant's body, for example, leg action information and arm action information in a running action. In some embodiments, the participant terminal may divide the participant's actions into sub-actions of multiple body parts, and then acquire sub-action information of each body part.

In some embodiments, when a participant performs a movement, at least one body part participates in or constitutes the movement. For example, when a participant performs a running movement, the participant's feet, legs, arms, etc. Corresponding actions, wherein the feet and legs can be used as the core parts of the running action. For different scenes and different actions, the core parts of the participants can be different. For more descriptions of different scenarios and core parts, please refer to Figure 6 and its related descriptions.

In some embodiments, the first action information and the sub-action information can be obtained in multiple ways. For example, it can be obtained through cameras, wearable devices, sensors and other devices. Specifically, the camera can capture the real-time image information of the participant, and the real-time image processing can obtain the changes of various parts of the participant's body, and then obtain the first action information and sub-action information; the wearable device can be connected with the displacement sensor, Components such as angle sensors are used to obtain change information of various parts of the participant's body and convert it into first action information and sub-action information.

In some embodiments, the participant terminal can synchronously update the second movement information 670 and/or the second movement information 670 of the virtual character corresponding to the participant based on the first movement information 620 and/or the first action information 630 through a preset 3D coordinate position algorithm Or the second action information 680.

In some embodiments, synchronously updating the second movement information and/or the second movement information of the avatar corresponding to the participant based on the first movement information and/or the first movement information may include updating the second movement information based on the first movement information. Movement information, updating second movement information and second action information based on first movement information, updating second action information based on first action information, updating second movement information and second action information based on first movement information, updating second movement information and second action information based on first movement information, information and the first action information update the second movement information and the second action information and so on.

In some embodiments, the participant terminal may synchronously update the second movement information and/or the second movement information of the virtual character corresponding to the participant based on the first movement information and/or the first movement information through various methods. For example, the participant's terminal can scan the actual space in real time to obtain the participant's first movement information and/or first action information and transmit it to the server, and then perform coordinate conversion through the preset 3D coordinate position algorithm to convert the participant's first The movement information and/or the first action information are combined with the data of the avatar to obtain the second movement information and/or the second action information of the corresponding avatar.

For example, if the participant's terminal obtains the first movement information of the participant moving a distance of two meters straight ahead in the actual space, and the first movement information of walking with a 70-centimeter stride and swinging with both arms drooping at 15°, it can pass The preset 3D coordinate position algorithm performs coordinate conversion, merges the above data with the avatar, and obtains the second movement information that the avatar also moves two meters straight ahead in the virtual space, and moves on foot at a pace of 70 centimeters. The second action information accompanied by the 15° swing of the arms drooping.

In some embodiments, the server can determine at least one core body part of the participant based on the current scene; determine the presentation priority 640 of the sub-action information of each part of the participant's body based on the at least one core body part; The display priority determines the display parameters 650 of the action information, and the display parameters 650 include display frequency and display precision; based on the display parameters, the second action information 680 of the avatar corresponding to the participant is synchronized.

The current scene refers to the scene in the current virtual space, such as an academic conference, a remote consultation, etc. For more descriptions of the scene, please refer to Figure 1 and its related contents.

Core body parts are those body parts that are most important to the participant's movements. For example, in the surgical guidance scenario, the core part of the doctor's operation can be the hand; another example, during the interrogation process, the core part of the interrogated person can be the face.

In some embodiments, the core body parts can be determined in multiple ways, for example, a comparison table of core body parts corresponding to different stages in different scenes can be preset, and the core body parts in the current scene can be determined based on the preset comparison table. For another example, the core body part can also be determined according to the continuous movement time of the part. For a part with a long continuous movement time, it can be considered that it undertakes the current main action of the participant and is the core body part.

The presentation priority 640 refers to the presentation priority of the sub-action information of each part of the participant's body. The display priority 640 can be expressed by sorting or level. For example, the numerical value of 1-10 can be used to reflect the order of the display priority. The smaller the numerical value, the higher the ranking, indicating that the corresponding sub-action information should be displayed first. For another example, the numerical value of 1-10 may also reflect the level of display priority. The larger the numerical value, the higher the level, and the corresponding sub-action information shall be displayed first.

In some embodiments, the server may preset display priorities of various parts in different scenes, and may display sub-action information based on a preset display priority comparison table in practical applications. For example, it can be preset that in the surgical live broadcast scene, the display priority of the hand sub-motion information is the highest, followed by the arm, and the leg is the lowest. Then in the actual live broadcast of the operation, it can be based on the above information in the preset priority comparison table Show the doctor's actions. In some embodiments, the presentation priority may also be determined based on scene information and action information of various parts of the body. For details on determining the presentation priority, refer to FIG. 7 and its description.

The display parameters 650 refer to parameters related to the display of sub-actions. For example, the display parameters may include display frequency and display accuracy of an action.

The display frequency refers to the update frequency of the sub-action. For example, the display frequency range can be set to 30-60 Hz as the low display frequency, 60-90 Hz as the middle display frequency, and 90-120 Hz as the high display frequency. In some embodiments, the display frequency may be a fixed numerical option, or may be freely changed within the display frequency range. In some embodiments, the display frequency can be preset by the server, and can also be determined based on the display priority of the sub-action information, for example, the higher the display priority, the higher the display frequency.

The display accuracy refers to the display accuracy of the sub-action, and the display accuracy can be expressed in pixels. For example, 1280*720 pixels can be used as smooth display precision, 1920*1080 pixels can be used as standard display precision, and 2560*1440 and above pixels can be used as high-definition display precision. In some embodiments, the display accuracy can be preset by the server, and can also be determined based on the display priority of the sub-action information, for example, the higher the display priority, the higher the display accuracy.

In some embodiments, the display parameters may be determined based on display priorities of sub-action information, and sub-actions with higher priority may be displayed with higher frequency and precision. For example, in an operation, the change of the doctor's hand movements can be displayed with a higher display frequency and display accuracy, and the movements of other parts such as shaking the body can be displayed with a lower display frequency; another example, during the interrogation process, The facial expression changes of the person being interrogated can be displayed at a higher display frequency and display accuracy.

In some embodiments, the server can preset parameter tables corresponding to different display priorities. For example, the preset display frequency corresponding to the first display priority is 120hz, the display accuracy is 2560*1440 pixels, and the display frequency corresponding to the second display priority is The frequency is 90hz, and the display accuracy is 1920*1080 pixels. In some embodiments, the presentation parameters can also be set by the participants themselves.

In some embodiments, after determining the display parameters of each part of the participant based on the display priority of the sub-action information, the server can obtain the display parameter data of each part, and then perform the second action information of the avatar according to the display parameters of different parts. Make a sync update. For example, according to the display frequency of different parts (for example, for the same participant, the display frequency of its hands is 120hz, and the display frequency of legs is 60hz) to collect and update the second action information of the corresponding part; and for example, according to the display accuracy of different parts ( For example, for the above-mentioned participants, the display accuracy of their hands is 2560*1440 pixels, and the display accuracy of their legs is 1280*720 pixels) to display the second action information.

In some embodiments, the location information may be updated based on the second movement information by using a preset 3D coordinate location algorithm. For specific description, please refer to FIG. 4 and related contents.

Determine the display parameters according to the display priority to synchronize the second action information, use high-frequency, high-precision display for more important action changes, and use low frequency and low-precision display for unimportant action changes, which can ensure action display Effectively save server resources at the same time.

According to the position update method shown in Figure 6, the action information and movement information of the participants in the real space can be accurately mapped in the virtual space in real time, so that the remote participants can watch and understand the operation details of the operators in the real space in real time , so as to provide real-time guidance information, avoid interference caused by unnecessary actions, and enhance the immersive experience of participants.

Fig. 7 is an exemplary diagram of determining display priorities of sub-action information according to some embodiments of the present specification. In some embodiments, the process 700 may be performed by the positioning module 220 .

In some embodiments, the presentation priority of sub-action information may be implemented based on a processing model.

In some embodiments, a processing model may be used to determine the presentation priority of sub-action information. The processing model may be a machine learning model, for example, the processing model may include a convolutional neural network model (Convolutional Neural Networks, CNN) or a deep neural network model (Deep Neural Networks, DNN).

Step 710, based on the motion image, the motion trajectory of each body part and the motion feature vector of each body part can be determined through a convolutional neural network model.

Action images may refer to images of sub-actions of various parts of a participant. The action information is obtained through the data acquisition equipment in the actual space where the participants are located. For example, the action image may be an action video or picture of participant A captured by a panoramic camera.

In some embodiments, the convolutional neural network model may be used to process at least one motion image to determine at least one motion trajectory and motion feature vector corresponding to the motion image.

A motion trajectory may refer to a movement trajectory of a participant's body part. The motion trajectory can be represented by a sequence or matrix of position coordinates of corresponding body parts at continuous time points, etc., where each sequence or matrix element can represent the position coordinates of the center position of a part of the body at the corresponding moment. For example, the action trajectory sequence can be ((1,0), (1,1), (1,2)), where (1,0), (1,1), (1,2) are the participants The position coordinates of A's right hand at three consecutive time points.

Motion feature vectors may refer to feature vectors of actions of various parts of the body. The elements of the motion feature vector may include the name of the part, the importance of the action of the part in each scene, the frequency of the action of the part, and the like. There may be a plurality of part names that are parts acquired based on motion images. The actions of each part can be preset with different degrees of importance according to different scenes. The frequency of movements of the parts can be represented by the number of times of movements within a preset time period. For example, in a training course, the teacher's finger movements and facial movements can all be set with higher importance. As an example only, the motion feature vector can be (1,40,3), where 1 can represent the hand, 40 can represent the importance of the hand to the current scene, and 3 can represent the number of times the hand has moved 3 times .

In some embodiments, the deep neural network model can be used to process the action trajectory, action feature vector and scene information to determine the presentation priority of the sub-action information.

Scene information can be represented based on various forms, for example, it can be represented by vectors. According to the preset relationship between preset scenes and numbers and/or letters, elements in the scene information may correspond to a scene. For example, 1 in scene vector (1) may represent a training scene.

Step 720, based on the scene information, the motion trajectories of each part of the body, and the motion feature vectors of each part of the body, the display priority of the sub-action information can be determined through the deep neural network model.

In some embodiments, the deep neural network model may be used to process at least one action track, action feature vector, and scene information corresponding to the action image to determine the presentation priority of the sub-action information. For details about display priorities of sub-action information, refer to descriptions in other parts of this specification, for example, FIG. 6 .

In some embodiments, the processing model can be obtained through joint training of a convolutional neural network model and a deep neural network model. For example, input training samples to the initial convolutional neural network model, that is, historical action images, and obtain at least one historical action track and historical action feature vector corresponding to the historical action images; then the output of the initial convolutional neural network model and the historical action images The corresponding historical scene information is used as the input of the initial deep neural network model. During the training process, a loss function is established based on the labels of the training samples and the output of the initial deep neural network model, and the parameters of the initial convolutional neural network model and the initial deep neural network model are iteratively updated based on the loss function until the preset conditions are met Training is complete. After the training is completed, the parameters of the convolutional neural network model and the deep neural network model in the processing model can also be determined.

In some embodiments, the training samples can be obtained based on the historical action images collected by the data collection device and the corresponding historical scene information. The label of the training sample may be the historical presentation priority of the corresponding sub-action information. Labels can be manually annotated.

Determining the display priority of the sub-action information through the machine learning model can increase the speed of determining the display priority, and can also improve the accuracy of the display priority.

In some embodiments, the presentation priority of the sub-action information can be implemented through a vector database. Specifically, the scene action vector can be constructed based on the scene information and the sub-action information of each part of the participant's body, and then, based on the scene action vector, the reference vector is retrieved in the vector database, and the display priority of the sub-action information corresponding to the reference vector is used as this second priority.

Sub-action information can be represented by a sub-action information vector. Elements in the sub-action information vector may represent body part names and corresponding actions. Different actions can be represented based on different numbers or letters. For example, the sub-action information vector is (1, 2), where 1 indicates a hand, and 2 indicates that the hand action is a fist. In some embodiments, scene information and sub-action information may be combined to determine a scene action vector. Scene action vectors may be multi-dimensional vectors. For example, a in the scene action vector (a, b) may represent a consultation scene, and b may represent a sub-action information vector.

In some implementations, scene motion vectors can be obtained through an embedding layer. The embedding layer may be a machine learning model, for example, the embedding layer may be a recurrent neural network model (Recurrent Neural Network, RNN) or the like. The input of the embedding layer can be the scene information, the sub-action information of each part of the participant's body, and the output can be the scene action vector.

A vector database may refer to a database containing historical scene action vectors. In some embodiments, the preset database includes historical scene action vectors and presentation priorities of sub-action information corresponding thereto.

The reference vector may refer to a historical scene action vector whose similarity with the scene action vector exceeds a preset threshold. For example, the preset threshold is 80%, and the similarity between the historical scene action vector 1 and the scene action vector in the vector database is 90%, then the historical scene action vector 1 is the reference vector. In some embodiments, the reference vector may be a historical scene action vector with the highest similarity to the scene action vector.

The reference vector may refer to the similarity with the scene action vector and may be determined based on the vector distance between the scene action vector and the historical scene action vector. Vector distances may include Manhattan distance, Euclidean distance, Chebyshev distance, cosine distance, Mahalanobis distance, etc. According to the formulas corresponding to different distance types, values can be substituted for mathematical calculations.

In some embodiments, the embedding layer can be jointly trained with the deep neural network model. Input training samples to the initial embedding layer to obtain scene action vectors; then use the output of the initial embedding layer as the input of the initial deep neural network model. During the training process, a loss function is established based on the label and the output of the initial deep neural network model, and the parameters of the initial embedding layer and the initial deep neural network model are iteratively updated based on the loss function at the same time, until the preset conditions are met and the training is completed. After the training is completed, the parameters of the embedding layer and the deep neural network model can also be determined.

In some embodiments, the training samples may be historical scene information, historical sub-action information of various parts of the participant's body. The label of the training sample may be the historical presentation priority of the corresponding sub-action information. Labels can be manually annotated.

Obtaining the parameters of the embedding layer through the above training method is helpful in some cases to solve the problem that it is difficult to obtain labels when training the embedding layer alone, and it can also enable the embedding layer to better obtain scene motion vectors that reflect scene information and sub-action information.

Presetting the vector database based on the historical data, and then determining the display priority of the sub-action information can make the determined display priority more in line with the actual situation.

Fig. 8 is a schematic flowchart of an exemplary process 800 of a data processing method for XR according to some embodiments of the present specification. The process 800 can be performed by the presentation module 240 and the generation module 250 .

Step 810, creating a canvas in the virtual space in response to the request of the annotation requester.

An annotation requester refers to a participant who makes an annotation request.

In some embodiments, the labeling requester may send the labeling request at the terminal and be received by the server.

The canvas refers to the canvas in the virtual scene that displays the content to be marked. Canvas can have various forms, for example, three-dimensional canvas and so on.

In some embodiments, the server may create a default canvas in the virtual space after receiving the request from the label requester. In some embodiments, the annotation requester can change the size and shape of the canvas through manual operations or preset options, and in some embodiments, the annotation requester can drag the canvas to move as needed.

Step 820, displaying the content to be marked on the canvas, wherein the content to be marked is data that has been marked and/or original data that has not been marked.

In some embodiments, the content to be marked can come from various data. For example, the content to be marked may be data prepared by the participant in advance. For another example, the content to be marked may also include shared data corresponding to the scene, for example, the patient's 3D image model, pathological pictures and videos when sharing pathological data. The shared data in different scenarios is different, and the content to be marked is also different. For more descriptions on the content to be marked in different scenarios, please refer to the relevant content in FIG. 5 .

Labeled data refers to data with historical labels. In some embodiments, labeling can continue on the labeled data. In some embodiments, when labeling data for the second time, it is possible to choose whether to display historical labeling situations.

Unlabeled raw data refers to data that does not have historical annotations, for example, raw data downloaded from the server by participants, real-time data in live broadcasts, etc.

In some embodiments, the content to be marked is the content displayed in any window and any position of the multiple windows on the terminal of the marking requester.

In some embodiments, different presentation methods may be adopted for different content to be marked. For example, pictures can be displayed statically, and videos can be displayed dynamically using video sources. In some embodiments, different display methods can also be used for incompatible terminals. For example, VR devices can perform 3D display through different images of the eyes and an appropriate interpupillary distance, mobile terminal devices can use screens for display, and computer terminals can display display through a display, etc.

Step 830, acquire the mark information created on the canvas by the ray interactive system by the mark requester, wherein the mark information includes mark content and mark path.

Marking information refers to the information generated by marking the content to be marked. In some embodiments, the marker information may also include marker time, marker position information corresponding to the marker time, and the like. For example, the marking time is nine o'clock in the morning, and the marking is at the position of coordinates (20, 30, 40) in the virtual space at this time.

The marked content refers to the specific content to be marked. The marked content may include content drawn with a brush, inserted pictures, resizing operations, and the like.

The marker path refers to the stroke path of the marker. For example, if the labeling requester marks the character "人" on the canvas, the marking path is the left and right sides of the character "人".

The ray interactive system refers to the system used for marking. In some embodiments, through the ray interaction system, the participant can point the ray to the content to be marked displayed on the canvas, and mark by touching, pressing and other gesture operations.

In some embodiments, the terminal may automatically save the tagging information of the tagging requester locally in real time. In some embodiments, the annotation requester can actively choose to save the annotation information by touching the canvas, clicking a button, and other operations.

Step 840, sharing the content to be marked and marking information to terminals of other participants for display.

In some embodiments, the terminal can collect the marking information of the corresponding participant and upload it to the server, and then the server sends it to other participant terminals, and creates a display window on other participant terminals, so as to realize the content to be marked and the marking information Share to the terminals of other participants for display.

In some embodiments, based on the display settings, the content to be marked and its marking information can be shared with the terminals of other participants for display. Displaying based on the display settings can realize the personalization of the display and meet the needs of the participants. Specifically, it can be See Figure 9 and its related contents.

The data processing method for XR shown in Figure 8 can realize real-time annotation of shared data. The results are saved locally for future reference, summary and comparison. In addition, the annotation information can also be shared with other participants to facilitate discussions on complex issues among participants.

Fig. 9 is a schematic diagram of an exemplary process 900 of displaying content to be marked according to some embodiments of the present specification. The process 900 can show the execution of the module 240 .

Step 910, acquire the display setting of the mark requester, and the display setting includes real-time mark display and mark display after completion.

The display settings refer to the settings related to displaying the content to be marked and marking information. The display setting may also include 3D position information of the display window, size, color, precision and other information of the display screen.

Real-time marking display refers to synchronizing the marking process of the marking requester to other participants in real time, that is, other participants can see the creation process of marking information. Display after marking refers to sharing the final result of marking with other participants, that is, other participants can obtain the marking result, but cannot see the creation process of marking information.

In some embodiments, the display setting may be determined by default by the terminal, for example, the terminal defaults to display after completion of marking. In some embodiments, the display setting can also be determined by the participant's selection of options in the terminal display setting window, for example, the participant can check the option of real-time marker display by clicking, touching and other operations. In some embodiments, the terminal can record the participant's presentation settings and transmit the setting data to the server.

Step 920, based on the display settings, share the content to be marked and its marking information to terminals of other participants for display.

In some embodiments, the server may share the content to be marked and its marking information to terminals of other participants for display according to the display settings. For example, the server can obtain the display setting selected by the participant as a marked display, and can also obtain the content to be marked and its marking information of the participant, and then transmit the real-time data of the content to be marked and its marking information to other participants according to the display settings real-time display on the terminal of the user.

In some embodiments, the displayed situation may be different for different scenarios. For example, during surgical guidance, in order to avoid affecting the surgical process, the display of the content to be marked and its marked information needs to avoid the patient's surgical site and equipment display screen. For another example, in the training and teaching process, in order to ensure that each participant can clearly see the content to be marked and its marking information, a display window can be created in front of each participant; while in the academic lecture scene, only one Large display window for display.

In some embodiments, sharing the content to be marked and its marking information to terminals of other participants for presentation includes: determining the viewing angle information of each participant based on the position information of each participant in other participants; The viewing angle information of the participants is used to determine and display the display content of each participant. The display content includes the content to be marked and/or the marking information under the viewing angle information.

The angle of view information refers to the angle of view information of the participant relative to the content to be marked and its marking information. The angle of view information may include information such as azimuth, angle, height, and distance. The position of the participants is different, and the corresponding perspective information is also different. For example, for the same display model, the viewing angle information of the participant at the front right of the display model mainly includes part of the right view information and part of the front view information of the display model; the viewing angle information of the participant at the upper left of the display model mainly includes the part of the display model Left view information and partial top view information.

In some embodiments, the server can compare the position information of the participant acquired by the terminal with the display position information of the content to be marked and its marking information, and determine the relative position between the participant and the display position, thereby determining the angle of view information. For example, three-dimensional space coordinates (x, y, z) can be constructed in the virtual space. When displaying a three-dimensional image such as a model, the participant stands facing the y direction, and its position coordinates are (1, 1, 1), and the position coordinates of the display model (such as center coordinates) is (1, 2, 2), then the relative position between the display model and the participants can be obtained based on the calculation between the two coordinates, the participants can see the partial front view of the model and For the information at the bottom of the model, the server can determine the specific viewing angle information through an algorithm. For another example, in the above example, there is another participant whose position coordinates are (1, 0, 1), then in the viewing angle information corresponding to this participant, the distance from the display model is greater than the position coordinates of (1, 1, 1 ) between the participant and the displayed model, that is, the proportion of the model seen by the corresponding participant is smaller than the proportion of the model seen by the participant whose position coordinates are (1, 1, 1).

In some embodiments, based on the view angle information of each participant, the display content that the participant can see under the view angle can be calculated, and the content can be displayed. For example, according to the perspective information of the participant, it is obtained that the participant is located on the right side of the displayed 3D model, that is, it is determined that the participant can see the right view of the model and the right view is displayed to the participant. In some embodiments, the content that a participant can see is also related to the distance. For example, for a participant who is far away, the proportion of the corresponding displayed content may be smaller than that of a participant who is closer.

Displaying the content to be marked and its marking information to other participants according to the display settings can facilitate participants' discussions on complex issues, and adopt different display settings for different content to be marked and their marking information to achieve the best display effect. In turn, the effect of discussion and guidance can be improved; and different display settings can realize the personalization of the display to meet the needs of the participants. At the same time, determining the display content according to the perspective information of the participants can provide a more realistic, all-round, multi-level rendering display effect and enhance the immersive experience of the participants.

Fig. 10 is a schematic diagram of an exemplary schematic flow 1000 for determining predicted display content according to some embodiments of the present specification. In some embodiments, the process 1000 can be performed by the presentation module 240 .

Step 1010, predicting the future trajectory of the participant.

The future trajectory refers to the trajectory of the participant after the current time. The motion trajectory may include time information and corresponding location information. In some embodiments, the time length of the time period corresponding to the future motion trajectory can be set according to requirements. In some embodiments, the time period corresponding to the future motion trajectory may be a time period starting from the current time, or a time period with a certain interval from the current time. For example, the future movement trajectory may be a movement trajectory within the next 5 seconds, or a movement trajectory within the next 10 minutes. For another example, the future movement track may be a movement track within 5 seconds from the current time, or a movement track within 10 minutes after a 5-minute interval from the current time (that is, within 5-15 minutes from the current time).

In some embodiments, the future trajectory of the participant can be predicted based on the scene and the current sub-action information of the participant. For example, in the scene of live surgery, if the doctor picks up the instruments or sutures in the surgical suture bag, it can be predicted that the doctor will suture the surgical site next. In some embodiments, the future trajectory of the participants can also be predicted based on the historical scenes and the time when the participants entered the scenes. For example, for the teaching scene of the same topic, if the teaching staff showed demonstration actions 30 minutes into the scene in the historical teaching scene, it can be predicted that in the current teaching scene, the teaching staff will show demonstration actions at 30 minutes. For another example, for the surgical guidance scene of the same theme, in the historical surgical guidance scene, the guided personnel will gather at the operating bedside to watch the guidance after 10 minutes of entering the scene, then it can be predicted that in the current surgical guidance scene, the guided personnel will Move around the bed at 10 minutes.

In some embodiments, the prediction of the future trajectory of the participant can be realized based on a prediction model; the structure of the prediction model is a recurrent neural network model; the input of the prediction model is a preset historical time period up to the current time, and participants The positioning data sequence of the person; the output of the prediction model is the predicted position data sequence of the preset future time period.

The preset historical time period refers to the time period up to the current time. In some embodiments, the preset historical time period may be the time period from when the participant enters the scene to the current time. For example, if the participant enters the virtual scene at nine o'clock and the current time is ten o'clock, then nine to ten o'clock is the preset time period. historical time period. In some embodiments, the preset historical time period can also be the time period from the beginning of a certain action to the current time. For example, if the participant performs a squat action, the time period from the moment the participant starts squatting to the current time is Preset historical time period.

In some embodiments, a preset historical time period may also include multiple time point information. In some embodiments, the time point information may be information of a single time point. For example, if the time point collection interval is 1 second, and the preset historical time period is the past 2 minutes, then the preset historical time period includes 120 time point information. For another example, if the time point collection interval is 1 minute, and the preset historical time period is the past 1 hour, then the preset historical time period includes 60 time point information.

In some embodiments, one time point information also corresponds to one sub-time period information. For example, the preset historical time period is from 9:00 to 10:00, and this time period can be divided into three sub-time periods. For example, from 9:00 to 9:25 is divided into the first sub-time period, 9:45 is divided into the second sub-time period, and 9:45 to 10:00 is the third sub-time period, that is, the preset time period includes three time point information. In some embodiments, the time point collection interval and the length of the sub-time period can be preset by the server, or can be set by the participants themselves.

The positioning data sequence refers to the positioning data sequence of the participant in the virtual space. Positioning data sequences can reflect the movement of participants within a preset historical time period. Each element value in the positioning data sequence corresponds to the participant's position data at a point in time. For example, in a set of sequences ((1, 1, 1), (2, 2, 2), (1, 2, 1), (1, 2, 2)), if each coordinate element value corresponds to an interval of one Second, then (1, 2, 1) indicates the position data of the participant in the virtual space at the third second in the preset time period. For another example, in the above sequence, if the time point corresponding to each coordinate element value is a sub-time period, then (1, 2, 1) indicates that the participant corresponding to the third sub-time period within the preset time period is in the virtual Location data within a space.

In some embodiments, the training data of the prediction model may be multiple groups of labeled training samples, and the training samples may be a preset historical time period up to the current time, and a sequence of historical positioning data of participants. The training samples may come from historical data stored on the server. The label of the training sample of the prediction model can be used to preset the actual location data sequence of the participants in the future time period for the history, and the label can be obtained by manual labeling.

In some embodiments, a loss function is constructed by corresponding labels and initial prediction model results, and parameters of the prediction model are iteratively updated by gradient descent or other methods based on the loss function. When the preset conditions are met, the model training is completed, and the trained prediction model is obtained. Wherein, the preset condition may be that the loss function converges, the number of iterations reaches a threshold, and the like.

Step 1020, based on the future trajectory, determine the viewing angle information at the future time point.

A future time point refers to a certain moment after the current time. The future time point is included in the time period corresponding to the future motion trajectory.

The angle of view information refers to the angle of view information of the participant relative to the content to be marked and its marking information. For more information about the angle of view information, please refer to FIG. 9 and related descriptions.

In some embodiments, based on the future trajectory of the participant, the location information of the participant at the future time point can be determined, according to the location information of the participant at the future time point and the location information of the display window of the content to be marked and its marking information By comparison, the relative position between the participant and the display window can be determined, thereby determining the viewing angle information. For more description on how to determine the viewing angle information, please refer to related content in FIG. 9 .

Step 1030, based on the viewing angle information at the future time point, determine the corresponding predicted display content.

In some embodiments, the predicted display content may include the content to be marked in the virtual scene, the marked content and its marking information, and the like. In some embodiments, when the predicted display content is determined, based on the perspective information of each participant at a future time point, the predicted display content that the participant can see under the perspective can be obtained through calculation. For example, when the predicted display content is the marked three-dimensional model of the heart and its marking information, according to the viewing angle information of the future time point, it can be known that the predicted display window of the content to be marked and its marking information is located at the corresponding predicted future time point. If the position is 30° to the front right of the position, it can be predicted that the predicted display content corresponding to the participant is a side perspective view of the marked content and its marked information at a future time point at this angle.

In some embodiments, when the predicted display content changes in real time, pre-acquisition preparations may be made for the predicted display content at the future time point based on the perspective information of each participant at the future time point. For example, if the content to be marked in the virtual scene is a doctor's surgery operation, and the doctor's surgery operation corresponding to the future time point is a chest surgery operation, then the camera at the corresponding position of the participant's perspective information can enter the standby state for chest shooting in advance.

Predicting the display content by predicting the future trajectory of the participants can pre-prepare the display content at the corresponding future time point, or pre-prepare the collection of the display content at the future time point, thereby improving the loading speed and optimizing the user experience of the participants.

It should be noted that different embodiments may have different beneficial effects. In different embodiments, the possible beneficial effects may be any one or a combination of the above, or any other possible beneficial effects.

The basic concept has been described above, obviously, for those skilled in the art, the above detailed disclosure is only an example, and does not constitute a limitation to this description. Although not expressly stated here, those skilled in the art may make various modifications, improvements and corrections to this description. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Meanwhile, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and/or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that two or more references to "an embodiment" or "an embodiment" or "an alternative embodiment" in different places in this specification do not necessarily refer to the same embodiment . In addition, certain features, structures or characteristics in one or more embodiments of this specification may be properly combined.

In addition, unless explicitly stated in the claims, the order of processing elements and sequences described in this specification, the use of numbers and letters, or the use of other names are not used to limit the sequence of processes and methods in this specification. While the foregoing disclosure has discussed, by way of various examples, some embodiments that are presently believed to be useful, it should be understood that such detail is for purposes of illustration only and that the appended claims are not limited to the disclosed embodiments, but instead claim It is intended to cover all modifications and equivalent combinations consistent with the spirit and scope of the embodiments of this specification. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by a software-only solution, such as installing the described system on an existing server or mobile device.

In the same way, it should be noted that in order to simplify the expression disclosed in this specification and help the understanding of one or more embodiments, in the foregoing description of the embodiments of this specification, sometimes multiple features are combined into one embodiment, appended Figures or their descriptions. This method of disclosure does not, however, imply that the subject matter of the specification requires more features than are recited in the claims. Indeed, embodiment features are less than all features of a single foregoing disclosed embodiment.

In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of the embodiments use the modifiers "about", "approximately" or "substantially" in some examples. grooming. Unless otherwise stated, "about", "approximately" or "substantially" indicates that the stated figure allows for a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that can vary depending upon the desired characteristics of individual embodiments. In some embodiments, numerical parameters should take into account the specified significant digits and adopt the general digit reservation method. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of the range are approximations, in specific embodiments, such numerical values are set as precisely as practicable.

Each patent, patent application, patent application publication, and other material, such as article, book, specification, publication, document, etc., cited in this specification is hereby incorporated by reference in its entirety. Application history documents that are inconsistent with or conflict with the content of this specification are excluded, and documents (currently or later appended to this specification) that limit the broadest scope of the claims of this specification are also excluded. It should be noted that if there is any inconsistency or conflict between the descriptions, definitions, and/or terms used in the accompanying materials of this manual and the contents of this manual, the descriptions, definitions and/or terms used in this manual shall prevail .

Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other modifications are also possible within the scope of this description. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims (10)

1. A multiplayer online live broadcast method based on XR, comprising: establishing a communication connection with the terminals of at least two participants; creating a virtual space in which avatars corresponding to each of the at least two participants are created; determining the position information of the avatar corresponding to the participant in the virtual space based on the acquired position data of the participant in the real space through a preset 3D coordinate position algorithm; displaying the avatar in the virtual space based on the location information of the avatar; Obtaining the shared data uploaded by the participants, and displaying the shared data in the virtual space. 2. The method according to claim 1, wherein the preset 3D coordinate position algorithm is used to determine the corresponding position of the participant based on the obtained position data of the participant in the actual space. The location information of the virtual character in the virtual space includes: scanning the actual space where the participant is located, and spatially locating the participant; For said participant who has completed the scan, determine real-time location data of said participant in said physical space; determining first movement information of the participant in the real space based on the real-time location data; determining initial position information of the avatar in the virtual space; Acquiring first action information of the participant in the actual space; the first action information includes sub-action information of various parts of the participant's body; Synchronously update the second movement information and/or second movement information of the virtual character corresponding to the participant based on the first movement information and/or the first movement information through the preset 3D coordinate position algorithm . 3. The method according to claim 2, wherein the participant is updated synchronously based on the first movement information and/or the first action information through the preset 3D coordinate position algorithm The corresponding second movement information and/or second action information of the avatar includes: determining at least one core body part of the participant based on the current scene; prioritizing presentation of the sub-action information for various parts of the participant's body based on the at least one core body part; determining display parameters of the action information based on the display priority of the sub-action information, where the display parameters include display frequency and display accuracy; The second action information of the avatar corresponding to the participant is synchronized based on the presentation parameter. 4. The method according to claim 1, wherein the method further comprises: creating at least one second space and/or second window in the virtual space, wherein each of the at least one second space and/or second window corresponds to one of the participants; The shared data of the corresponding participant is displayed through the second space and/or the second window. 5. An XR-based multiplayer online live broadcast system, comprising: A connection module, configured to establish a communication connection with the terminals of at least two participants; A positioning module, configured to create a virtual space in which a virtual character corresponding to each of the at least two participants is created; The positioning module is further configured to determine the position of the avatar corresponding to the participant in the virtual space based on the acquired position data of the participant in the real space through a preset 3D coordinate position algorithm information; and displaying the avatar in the virtual space based on the location information of the task; a download module, configured to obtain the shared data uploaded by the participants; A display module, configured to display the shared data in the virtual space. 6. The system according to claim 5, wherein the positioning module is further used for: scanning the actual space where the participant is located, and spatially locating the participant; For the participant who has completed scanning, determine the real-time position data of the participant in the real space; determine the first movement information of the participant in the real space based on the real-time position data; determining initial position information of the avatar in the virtual space; Acquiring first action information of the participant in the actual space; the first action information includes sub-action information of various parts of the participant's body; Through the preset 3D coordinate position algorithm, the second movement information and/or second movement information of the avatar is synchronously updated based on the first movement information and/or the first movement information. 7. The system according to claim 6, wherein the positioning module is further used for: determining at least one core body part of the participant based on the current scene; prioritizing presentation of the sub-action information for various parts of the participant's body based on the at least one core body part; determining display parameters of the action information based on the display priority of the sub-action information, where the display parameters include display frequency and display accuracy; The second action information of the avatar corresponding to the participant is synchronized based on the presentation parameter. 8. The system according to claim 5, wherein the display module is further used for: creating at least one second space and/or second window in the virtual space, wherein each of the at least one second space and/or second window corresponds to one of the participants; The shared data of the corresponding participant is displayed through the second space and/or the second window. 9. An XR-based multiplayer online live broadcast device, said device comprising: at least one storage medium storing computer instructions; At least one processor executes the computer instructions to implement the XR-based multiplayer online live broadcast method according to any one of claims 1 to 4. 10. A computer-readable storage medium, characterized in that the storage medium stores computer instructions, and when a computer reads the computer instructions, the computer executes the method based on any one of claims 1 to 4 XR's multiplayer online live broadcast method.

Images