WO2022160867A1 - 远程重现方法、系统、装置、设备、介质及程序产品 - Google Patents

远程重现方法、系统、装置、设备、介质及程序产品 Download PDF

Info

Publication number
WO2022160867A1
WO2022160867A1 PCT/CN2021/130532 CN2021130532W WO2022160867A1 WO 2022160867 A1 WO2022160867 A1 WO 2022160867A1 CN 2021130532 W CN2021130532 W CN 2021130532W WO 2022160867 A1 WO2022160867 A1 WO 2022160867A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
terminal
image
human body
body model
Prior art date
Application number
PCT/CN2021/130532
Other languages
English (en)
French (fr)
Inventor
公维蒙
李军
涂丹丹
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Priority to EP21922457.3A priority Critical patent/EP4280226A4/en
Publication of WO2022160867A1 publication Critical patent/WO2022160867A1/zh
Priority to US18/358,228 priority patent/US20230368467A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/547—Remote procedure calls [RPC]; Web services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H04N7/157—Conference systems defining a virtual conference space and using avatars or agents
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation

Definitions

  • the present application relates to the field of computer technology, and in particular, to a remote reproduction method, system, apparatus, device, computer-readable storage medium, and computer program product.
  • Remote assistance means that the facilitator assists the assisted person through remote communication to solve the problems encountered by the assisted person.
  • the facilitator assists the assisted person through remote communication to solve the problems encountered by the assisted person.
  • experts with diagnostic or maintenance capabilities guide equipment operators at the fault site through remote communication.
  • the industry provides a holographic capture and reproduction solution, which reproduces the images of remote helpers at the fault site, thereby providing a more natural interaction.
  • the present application provides a remote reproduction method.
  • the method pre-builds a realistic human body model in the preparation stage, and then uses the pre-built human body model to generate action images in the remote reproduction stage, and reproduces the action images to the user to achieve face-to-face communication. Effect.
  • This solves the problem that the model reconstruction effect is poor and makes it difficult to meet business needs, and there is no need to transmit multi-view images in real time, which reduces the occupation of network resources, avoids frame loss and freezes in the case of poor network quality, and improves the interactive experience.
  • the present application also provides systems, apparatuses, devices, computer-readable storage media, and computer program products corresponding to the above methods.
  • the present application provides a remote reproduction method.
  • the method can be performed by a remote reproduction system.
  • the remote reproduction system includes a first terminal and a second terminal, the first terminal is worn by a first user, the second terminal is worn by a second user, and the first user and the second user are in different geographical scenes.
  • the second terminal may acquire an action image of the first user, wherein the action image of the first user includes a sequence of at least one action formed by a human body model of the first user, and the human body model of the first user is a pre-built model, Then, the second terminal reproduces the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the human body model of the first user is pre-built in the preparation stage, the human body model is directly used to generate motion images in the remote reproduction stage, thereby avoiding a lot of time spent on building a human body model during remote reproduction, and meeting the requirements of remote reproduction.
  • the current application requires real-time performance, and it can spend a lot of time in the preparation stage for reconstruction to obtain a realistic human body model, which has good results and can meet business needs.
  • the remote reproduction stage there is no need to send a large number of images to the cloud or the terminal to reconstruct the human body model in real time, which avoids large-scale data transmission occupying a large amount of network resources, and problems such as frame loss and freezing in the case of poor network quality. interactive experience.
  • the remote rendering system may also include a server.
  • the remote reproduction system can utilize the computing power of the server to generate an action image of the first user.
  • the server obtains the location information of the key points of the first user, the server can determine the movement trajectory of the key points of the first user according to the location information of the key points of the first user, and then the server can determine the movement trajectory of the key points of the first user according to the above movement trajectory and the data stored in the server
  • the human body model of the first user in the obtains the motion image of the first user, and the server sends the motion image of the first user to the second terminal.
  • the movement trajectory of the key points can be represented as the position sequence of the key points.
  • the key points of the first user are bound to the human body model of the first user, the server can determine the human body model corresponding to each position according to the position sequence of the key points, and can form a continuous human body model according to the human body model corresponding to each position, and then generate the first human body model.
  • the motion image of the first user includes a sequence of at least one motion formed by the user's mannequin.
  • the second terminal Since the image generation process that requires a lot of computing power is performed on the server side, the second terminal only needs to receive the motion image of the first user, and present the motion image of the first user to the second user in the geographic scene where the first user is located. In this way, the requirements for the second terminal are reduced, and the lightweight second terminal can also be used to implement the remote reproduction method, with high usability.
  • the remote reproduction system may include servers in the cloud, such as a central server in a central cloud and an edge server in an edge cloud.
  • the edge server and the central server can cooperate to generate motion images.
  • the edge server can obtain the position information of the key points of the first user, for example, by performing key point detection on the image of the key parts of the first user collected by the image sensor, and obtaining the position information of the corresponding key points, the central server obtains the position information of the corresponding key points according to the edge server.
  • the position information of the key points of the first user is determined, and the movement trajectory of the key points of the first user is determined, and then the action image of the first user is obtained according to the movement trajectory and the human body model of the first user stored in the central server.
  • the server may acquire the image of the key part of the first user, and then acquire the position information of the corresponding key point according to the image of the key part of the first user.
  • the server may use the key point detection technology to process the image of the key part of the first user to obtain the position information of the key point of the first user.
  • the server may construct a keypoint detection model, and then input the keypoint image of the first user into the keypoint detection model, extract keypoints from the image through the keypoint detection model, and obtain location information of the first user's keypoints.
  • the key point detection model may be a traditional key point detection model.
  • Traditional keypoint detection models include keypoint detection models based on template matching, such as graph structure models.
  • the keypoint detection model may also be a neural network-based keypoint detection model.
  • Neural network-based keypoint detection models include, but are not limited to, cascaded feature network models, regional multi-person pose estimation models, mask-based regional convolutional neural network models, and cascaded pyramid network models.
  • the method obtains the position information of the key points by using the key point detection model to detect the image, which reduces the type and number of sensors in the remote reproduction system, and reduces the complexity of the remote reproduction system.
  • the server may acquire position information of key points corresponding to the key parts from a pose sensor, such as an inertial measurement unit, deployed at the key parts of the first user. Since there is no need for detection through complex models, the efficiency of obtaining location information of key points is improved, thereby improving the efficiency of remote reproduction.
  • a pose sensor such as an inertial measurement unit
  • the first terminal may also acquire an image of the geographic scene where the second user is located, and then the first terminal reproduces the image of the geographic scene where the second user is located to the first user to in the geographic scene where the first user is located.
  • the first user can remotely assist the second user to solve problems in the geographic scene where the second user is located, such as problems at the fault site, according to the image of the geographic scene where the second user is located.
  • the first terminal is a virtual reality terminal
  • the second terminal is an augmented reality terminal
  • the augmented reality terminal can superimpose and present a live image and an image of the first user, thereby improving the sense of immersion.
  • the first terminal may acquire a motion image of the second user, where the motion image of the second user includes a sequence of at least one motion formed by a human body model of the second user, the second user
  • the human body model is a pre-built model, and the first terminal reproduces the action image of the second user to the first user in the geographic scene where the first user is located.
  • the first user can make targeted corrections to the first user according to the action image of the second user, thereby improving the efficiency of online learning.
  • the first terminal and the second terminal are virtual reality terminals.
  • virtual reality terminals can be used to virtualize the real environment, which can improve interactivity.
  • the computing power of the first terminal when the computing power of the first terminal is strong enough, for example, it can support the generation of motion images according to the human body model, the motion images may not be generated by the cloud computing power, but the first terminal can directly generate motion images according to the preset
  • the constructed human body model of the first user generates a motion image of the first user locally, and then delivers the motion image of the first user to the second terminal.
  • the first terminal acquires the location information of the key points of the first user, and then the first terminal determines the movement trajectory of the key points of the first user according to the location information of the key points of the first user, and according to the movement trajectory and the storage
  • the human body model of the first user in the first terminal obtains a motion image of the first user.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image.
  • the three-dimensional action image can vividly reproduce the image of the first user, providing a better immersive experience for the second user.
  • the human body model may also be a two-dimensional human body model, and the rendering efficiency of the two-dimensional human body model is high. In this way, the second user can view the action image of the first user in time, which reduces the delay.
  • the motion image can be used in scenarios such as remote assistance (eg, troubleshooting) or online learning, so as to provide the user with the effect of face-to-face communication.
  • remote assistance eg, troubleshooting
  • online learning so as to provide the user with the effect of face-to-face communication.
  • the present application provides a remote reproduction system.
  • the system includes a first terminal and a second terminal, the first terminal is worn by a first user, the second terminal is worn by a second user, and the first user and the second user are in different geographical scenes;
  • the second terminal is configured to acquire the motion image of the first user, reproduce the motion image of the first user to the second user in the geographic scene where the second user is located, and the first user
  • a motion image of a user includes a sequence of at least one motion formed by a mannequin of the first user, which is a pre-built model.
  • the remote reproduction system further includes a server
  • the server is configured to acquire the position information of the key points of the first user, determine the movement trajectory of the key points of the first user according to the position information of the key points of the first user, and determine the movement trajectory of the key points of the first user according to the movement trajectory and the human body model of the first user stored in the server to obtain the motion image of the first user, and the server is further configured to send the motion image of the first user to the second terminal.
  • the server is specifically used for:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the server is specifically used for:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the first terminal is used for:
  • the image of the geographic scene where the second user is located is reproduced to the first user into the geographic scene where the first user is located.
  • the first terminal is a virtual reality terminal
  • the second terminal is an augmented reality terminal.
  • the first terminal is used for:
  • the motion image of the second user includes a sequence of at least one motion formed by a human body model of the second user, and the human body model of the second user is a pre-built model;
  • the motion image of the second user is reproduced to the first user in a geographic scene where the first user is located.
  • the first terminal and the second terminal are virtual reality terminals.
  • the first terminal is used for:
  • a motion image of the first user is obtained.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • an embodiment of the present application provides a remote reproduction apparatus.
  • the remote reproduction device is deployed in a second terminal, the second terminal is worn by a second user, and the second user and the first user are in different geographical scenes, and the device includes:
  • an image acquisition unit configured to acquire a motion image of the first user, the motion image of the first user includes a sequence of at least one motion formed by a human body model of the first user, and the human body model of the first user is pre-built models;
  • the image reproducing unit is configured to reproduce the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the apparatus further includes:
  • an image sending unit configured to send a motion image of the second user to the first terminal, where the motion image of the second user includes at least one motion sequence formed by the human body model of the second user, the human body of the second user Models are pre-built models.
  • the apparatus further includes:
  • An image generation unit configured to acquire the position information of the key points of the second user, determine the movement trajectory of the key points of the second user according to the position information of the key points of the second user, and determine the movement trajectory of the key points of the second user according to the movement trajectory and the human body model of the second user stored in the second terminal to obtain a motion image of the second user.
  • the image generation unit is specifically used for:
  • a key part image of the second user is acquired, and position information of a corresponding key point is obtained according to the key part image of the second user.
  • the image generation unit is specifically used for:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the second user.
  • the apparatus further includes:
  • the image sending unit is configured to send the image of the geographic scene where the second user is located to the first terminal.
  • the image acquisition unit is specifically used for:
  • the motion video of the first user is acquired from the server or the first terminal.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • an embodiment of the present application provides a remote reproduction apparatus.
  • the remote reproduction device is deployed in the server, and the device includes:
  • a position obtaining unit used for obtaining the position information of the key point of the first user
  • a trajectory determination unit configured to determine the movement trajectory of the key point of the first user according to the position information of the key point of the first user
  • an image generating unit configured to obtain a motion image of the first user according to the movement trajectory and the human body model of the first user stored in the server, where the motion image of the first user includes the first user A sequence of at least one action formed by a user's human body model, the first user's human body model being a pre-built model;
  • the image sending unit is configured to send the motion image of the first user to the second terminal, so that the second terminal reproduces the motion image of the first user to the second user to the location where the second user is located.
  • the first user and the second user are in different geographic scenes.
  • the location acquisition unit is specifically used for:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the location acquisition unit is specifically used for:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the location acquisition unit is further used for:
  • the trajectory determination unit is also used for:
  • the image generation unit is also used for:
  • a motion image of the second user is obtained, and the motion image of the second user includes the human body model of the second user.
  • the sequence of at least one action of the second user, the human body model of the second user is a pre-built model;
  • the image sending unit is also used for:
  • the image generation unit is further configured to:
  • the image sending unit is also used for:
  • the image of the geographic scene where the second user is located is sent to the first terminal.
  • an embodiment of the present application provides a remote reproduction apparatus.
  • the remote reproduction device is deployed in a first terminal, the first terminal is worn by a first user, and the first user and the second user are in different geographical scenes, and the device includes:
  • An image acquisition unit configured to acquire an image of the geographic scene where the second user is located and/or a motion image of the second user, where the motion image of the second user includes images formed by a human body model of the second user. a sequence of at least one action, the human body model of the second user being a pre-built model;
  • An image reproduction unit configured to reproduce to the first user the image of the geographic scene where the second user is located and/or the motion image of the second user to the geographic scene where the first user is located .
  • the apparatus further includes:
  • an image sending unit configured to send a motion image of the first user to the second terminal, where the motion image of the first user includes at least one motion sequence formed by the human body model of the first user, the human body of the first user Models are pre-built models.
  • the apparatus further includes:
  • an image generation unit configured to acquire the position information of the key points of the first user, determine the movement trajectory of the key points of the first user according to the position information of the key points of the first user, and determine the movement trajectory of the key points of the first user according to the movement trajectory and the human body model of the first user stored in the first terminal to obtain a motion image of the first user.
  • the image generation unit is specifically used for:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the image generation unit is specifically used for:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the apparatus further includes:
  • the image sending unit is configured to send the image of the geographic scene where the first user is located to the second terminal.
  • the image acquisition unit is specifically used for:
  • the motion video of the second user is acquired from the server or the second terminal.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • the present application provides a second terminal, where the second terminal includes a processor and a memory.
  • the processor and the memory communicate with each other.
  • the processor is configured to execute the instructions stored in the memory, so that the second terminal implements the function of the remote reproduction apparatus according to the third aspect.
  • the present application provides a server including a processor and a memory.
  • the processor and the memory communicate with each other.
  • the processor is configured to execute the instructions stored in the memory, so that the server implements the function of the remote reproduction apparatus according to the fourth aspect.
  • the present application provides a first terminal, where the first terminal includes a processor and a memory.
  • the processor and the memory are in mutual communication.
  • the processor is configured to execute the instructions stored in the memory, so that the server implements the function of the remote reproduction apparatus according to the fifth aspect.
  • the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs the second terminal to execute the above-mentioned first aspect or any implementation manner of the first aspect. Steps performed by the second terminal in the remote reproduction method described above.
  • the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs a server to execute the first aspect or any implementation manner of the first aspect. Steps performed by the server in the remote reproduction method.
  • the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs a first terminal to execute the first aspect or any implementation manner of the first aspect The steps performed by the first terminal in the remote reproduction method.
  • the present application provides a computer program product including instructions, which, when running on a second terminal, enables the second terminal to execute the first aspect or any one of the implementations of the first aspect. Steps performed by the second terminal in the remote reproduction method.
  • the present application provides a computer program product containing instructions, which, when running on a server, cause the server to execute the remote reproduction method described in the first aspect or any implementation manner of the first aspect in the steps performed by the server.
  • the present application provides a computer program product including instructions, which, when running on a first terminal, enables the first terminal to execute the first aspect or any one of the implementation manners of the first aspect. Steps performed by the first terminal in the remote reproduction method.
  • the present application may further combine to provide more implementation manners.
  • FIG. 1 is a system architecture diagram of a remote reproduction system provided by an embodiment of the present application.
  • FIG. 2 is a system architecture diagram of another remote reproduction system provided by an embodiment of the present application.
  • FIG. 3 is a flowchart of a method for constructing a human body model according to an embodiment of the present application
  • FIG. 6 is a schematic structural diagram of a remote reproduction device provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a remote reproduction device provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a remote reproduction apparatus provided by an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of a second terminal according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a server according to an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a first terminal according to an embodiment of the present application.
  • first and second in the embodiments of the present application are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as “first” or “second” may expressly or implicitly include one or more of that feature.
  • Remote reproduction refers to the reproduction of images through remote communication technology, thereby breaking the geographical restrictions, so that people in different geographical locations can communicate on the same scene or event, and achieve the effect of face-to-face communication.
  • the reproduced images include images of people, or images of geographic scenes where people are located.
  • Remote reproduction can be applied in different scenarios in many fields.
  • remote reproduction can be used in remote assistance scenarios, or in online learning scenarios.
  • the remote assistance scenario refers to a scenario in which an assisted person assists the assisted person through remote communication to solve the problem encountered by the assisted person.
  • the helper may be an expert with diagnostic or repair capabilities
  • the assisted person may be an operator at the fault site.
  • Experts can guide operators at the fault site through remote communication to troubleshoot the cause of the fault and perform fault recovery according to the cause of the fault.
  • teachers or coaches can guide students or students to learn online through remote communication.
  • the remote reproduction scheme based on holographic capture and reproduction is specifically: synchronous cameras deployed in different directions simultaneously shoot the object to be reproduced, such as a shooting assistant, obtain a multi-view image of the assistant, and based on the multi-view image
  • the three-dimensional human body can be reconstructed in real time to obtain a three-dimensional human body model.
  • the augmented reality (AR) device of the assisted person such as AR glasses, establishes a coordinate system based on the observation position of the assisted person, places the reconstructed 3D human model in the coordinate system, and then renders it into a binocular video.
  • the AR glasses point to the assisted person.
  • the facilitator presents the binocular video, thereby realizing the reproduction of the facilitator's image on the scene.
  • the embodiments of the present application provide a remote reproduction method.
  • the method is performed by a remote reproduction system.
  • the remote reproduction system includes a first terminal and a second terminal.
  • the first terminal is worn by the first user, and the second terminal is worn by the second user.
  • the first user and the second user are participants in the communication process.
  • the first user may be an assistant, such as an expert with diagnostic or maintenance capabilities
  • the second user may be an assisted person, such as an operator at a fault site.
  • the first user may be a teacher or a coach, and the second user may be a student, a student, or the like.
  • the second terminal may acquire a motion image of the first user, wherein the motion image of the first user includes a sequence of at least one motion formed by a human body model of the first user, and the human body model of the first user is a pre-built model, Then, the second terminal reproduces the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the method divides the remote reproduction process into two stages, specifically the preparation phase before the remote reproduction starts and the remote reproduction phase.
  • the human body model is directly used to generate motion images, thus avoiding the need for a large amount of time to construct a human body model during remote reproduction, meeting the real-time requirements of remote reproduction applications, and it can spend more time in the preparation stage for reconstruction.
  • the realistic human body model has good effect and can meet the business needs.
  • the remote reproduction stage there is no need to send a large number of images to the cloud or the terminal to reconstruct the human body model in real time, which avoids large-scale data transmission occupying a lot of network resources, and problems such as frame loss and freezes in the case of poor network quality. interactive experience.
  • the remote reproduction method provided by the embodiments of the present application can be applied to different scenarios.
  • the remote reproduction method can be applied to a remote assistance scenario, so that an expert with diagnostic or repair capabilities can remotely assist an operator at the fault site in troubleshooting and/or fault recovery.
  • the remote reproduction method can be applied to online learning scenarios, such as online fitness learning, online experimental course learning and other scenarios, so that teachers or coaches can remotely guide students or students to learn.
  • the remote reproduction system 100 includes a first terminal 102 and a second terminal 104 .
  • the first terminal 102 is worn by the first user
  • the second terminal 104 is worn by the second user
  • the first user and the second user are in different geographical scenarios.
  • the second terminal 104 is specifically configured to acquire an action image of the first user, the action image of the first user includes at least one action sequence formed by a human body model of the first user, and the human body model of the first user is a pre-built model, Then, the motion image of the first user is reproduced and presented to the second user in the geographic scene where the second user is located.
  • the first terminal 102 is specifically configured to acquire an image of a geographic scene where the second user is located, or acquire an action image of the second user, where the action image of the second user includes a human body model of the second user A sequence of at least one action is formed, the human body model of the second user is a pre-built model, and then the image of the geographic scene where the second user is located is reproduced to the first user into the geographic scene where the first user is located. , or reproduce the action image of the second user to the first user in the geographic scene where the first user is located.
  • the remote reproduction system 100 can also Generate motion images with cloud computing power.
  • a cloud computing device such as a central server may generate a motion image of the first user according to the human body model of the first user, and then deliver the motion image of the first user to the second terminal 104 .
  • the remote reproduction system 100 further includes a central server 106 in a cloud environment.
  • the central server 106 may generate a motion image of the first user according to the pre-built human body model of the first user, and then send the motion image of the first user to the second terminal 104.
  • the second terminal 104 presents the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the remote rendering system 100 may also include an edge server 108 in an edge environment.
  • the edge server 108 may obtain location information of the key points of the first user.
  • key points refer to points that represent user actions, including but not limited to facial key points such as pupils, eyebrows, and corners of the mouth, or body joint points such as fingers, elbows, and knees.
  • the central server 106 generates an action image of the first user according to the position information of the key points of the first user and the human body model of the first user. In this way, it can avoid sending a large number of images to the cloud, which not only saves network resources, but also saves computing resources, and also avoids large-scale data transmission when the network quality is not good. problem occurs.
  • the step of acquiring the location information of the key points of the first user may also be performed by the central server 106 or performed by the first terminal 102 . This embodiment of the present application does not limit this.
  • the computing power of the first terminal 102 when the computing power of the first terminal 102 is strong enough, for example, it can support the generation of motion images according to the human body model, the motion images can be generated directly by the first terminal 102 without using the cloud computing power. According to the pre-built human body model of the first user, a motion image of the first user is generated locally, and then the motion image of the first user is delivered to the second terminal 104 .
  • the remote reproduction system 100 includes a first terminal 102 and a second terminal 104 .
  • the first terminal 102 can generate a motion image of the first user according to the pre-built human body model of the first user, and then the first terminal 102 can communicate through wired communication (such as optical fiber communication, coaxial cable communication) or wireless communication (such as The motion image of the first user is delivered to the second terminal 104 by a fifth-generation mobile communication network (5th generation mobile networks, 5G), a wireless hotspot (wireless fidelity, Wi-Fi)).
  • the second terminal 104 may present the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the second terminal 104 may also generate a motion image of the second user according to a pre-built human body model of the second user, and then send the second user's motion image to the first terminal 102 through a wireless communication method such as 5G.
  • the first terminal 102 may present the motion image of the second user to the first user in the geographic scene where the first user is located.
  • the second terminal 104 may also generate an image of the geographic scene where the second user is located, and then send the image of the geographic scene where the second user is located to the first terminal 102, and the first terminal 102 may send the image of the geographic scene where the second user is located to the first terminal 102.
  • the user presents the image of the geographic scene where the second user is located to the geographic scene where the first user is located.
  • the remote reproduction method provided by the embodiments of the present application relies on a human body model pre-built in the preparation stage.
  • the embodiments of the present application further describe in detail the process of building a human body model in the preparation stage.
  • the method includes:
  • S302 The server obtains a multi-view image of the user.
  • a sensor such as an infrared camera
  • the terminal such as the first terminal 102, the second terminal 104 worn by the user or the geographical scene where the user is located
  • the server may obtain the sensor from the sensor, such as an infrared camera.
  • the camera obtains images from different perspectives of the user, thereby obtaining multi-perspective images of the user.
  • S304 The server performs reconstruction according to the multi-view image to obtain the user's human body model.
  • the server may perform three-dimensional reconstruction according to the multi-view image to obtain the three-dimensional human body model of the user.
  • the server may perform two-dimensional reconstruction according to the multi-view images to obtain a two-dimensional human body model of the user.
  • the server may obtain a depth map according to the multi-view image, and then reconstruct a three-dimensional mesh (3D mesh) according to the depth map.
  • the three-dimensional mesh may specifically be a polygonal network composed of triangles, and the polygonal network is used to simulate the surface of a complex object, for example, the surface of a human body.
  • the server can use the reconstructed 3D mesh as a human body model.
  • Each grid of this 3D mesh has a gradient spherical lighting image that can be used to generate albedo, normal, glossiness, and ambient occlusion maps. These maps are compatible with standard rendering engines and can be used to reproduce rendered images under any set of lighting conditions.
  • the server can obtain multi-view images of the user in different actions, such as different expressions, different mouth shapes or different body movements, and model respectively according to the multi-view images to obtain the user’s different expressions and different mouth shapes. Or mannequins with different body movements.
  • the models constructed by this method are very detailed and realistic. And the model can also be repaired offline, such as beautifying, changing hairstyles, changing clothes, etc. to improve user satisfaction, or to simplify the model to reduce resource consumption. Moreover, for each user, the server can save the human body model after modeling and obtain the human body model, for example, save the human body model to the cloud for subsequent reuse.
  • the remote reproduction method provided by the embodiments of the present application may be implemented through the device-cloud collaboration, for example, may be implemented by the remote reproduction system 100 shown in FIG. 1 .
  • the implementation is described in detail below with reference to the accompanying drawings.
  • the method is executed by the remote reproduction system 100 shown in FIG. 1, and the remote reproduction system 100 includes a first terminal 102, a second terminal 104, a central server 106, The edge server 108, the method includes:
  • the edge server 108 acquires the image of the key part of the first user collected by the sensor.
  • the key parts are the parts where the key points are located, such as the eyes, mouth, eyebrows, etc.
  • Keypoints are points that characterize user actions.
  • the user actions include facial actions and/or body actions.
  • the key points include, but are not limited to, facial key points such as pupils, eyebrows, and corners of the mouth, or body joint points such as fingers, elbows, and knees.
  • the first terminal 102 worn by the first user or the geographic scene where the first user is located may deploy sensors, such as an infrared camera, and the infrared camera may collect images of key parts of the first user.
  • the edge server 108 can use infrared A sensor such as a camera acquires an image of a key part of the first user, for example, an image of an eye, an image of a mouth, or an image of other key parts.
  • the first terminal 102 worn by the first user is a VR terminal for illustration.
  • the VR terminal may be VR glasses, VR helmet, and the like.
  • Infrared cameras can be deployed inside VR glasses or VR headsets to capture eye images. Infrared cameras can also be deployed under VR glasses or VR headsets to capture images of the mouth.
  • the edge server 108 obtains the position information of the key point of the first user according to the image of the key part of the first user.
  • the edge server 108 may use the key point detection technology to process the image of the key part of the first user to obtain location information of the key point of the first user. Specifically, the edge server 108 can construct a key point detection model, and then input the image of the key part of the first user into the key point detection model, extract key points from the image through the key point detection model, and obtain the position information of the key point of the first user.
  • the key point detection model can be obtained according to the open source model, for example, it can be obtained by fine-tuning the open source model.
  • the keypoint detection model may be a conventional keypoint detection model.
  • Traditional keypoint detection models include keypoint detection models based on template matching, such as a Pictorial Structure model.
  • the keypoint detection model may also be a neural network-based keypoint detection model.
  • Neural network-based keypoint detection models include but are not limited to Google's Research & Machine Intelligence (G-RMI) model, cascaded feature network (CFN) model, regional multi-person pose estimation (regional multi-person pose estimation) -person pose estimation, RMPE) model, mask-based region convolutional neural network (mask region convolutional neural network, mask R-CNN) model and cascaded pyramid network (cascaded pyramid network, CPN) model, etc.
  • G-RMI Research & Machine Intelligence
  • CNN regional multi-person pose estimation
  • RMPE regional multi-person pose estimation
  • mask-based region convolutional neural network mask-based region convolutional neural network
  • mask R-CNN cascaded pyramid network
  • CPN cascaded pyramid network
  • the foregoing S402 to S404 are an implementation manner of acquiring the location information of the key points of the first user in the embodiment of the present application.
  • the edge server 108 may also acquire the first user in other manners.
  • the location information of the key points For example, the edge server 108 may directly acquire the position information of the key points corresponding to the key parts from the pose sensors deployed on the key parts of the first user.
  • the edge server 108 sends the location information of the key point of the first user to the central server 106.
  • the central server 106 generates an action image of the first user according to the position information of the key points of the first user and the human body model of the first user.
  • the central server 106 stores the human body model of the first user.
  • the human body model of the first user includes the human body model of the first user during different actions, such as different facial movements (different expressions, different mouth shapes) and different body movements.
  • the mannequin of the first user is bound with the key points of the first user.
  • the key points of the first user may be points representing the actions of the first user, for example, key points representing the facial actions of the first user, such as between the eyebrows, pupils, and corners of the mouth.
  • the central server 106 may determine the movement trajectory of the key point of the first user according to the position information of the key point of the first user. Then the central server 106 can obtain the action image of the first user according to the movement trajectory of the key points of the first user and the human body model of the first user stored in the central server 106 . For example, the central server 106 may perform rendering according to the movement trajectory of the key points of the first user and the human body model of the first user, so as to generate a motion image of the first user.
  • the movement trajectory of the key points can be represented as the position sequence of the key points.
  • the key points of the first user are bound to the human body model of the first user, the central server 106 can determine the human body model corresponding to each position according to the position sequence of the key points, and a continuous human body model can be formed according to the human body model corresponding to each position, and then A motion image of the first user is generated.
  • the motion image of the first user includes a sequence of at least one motion formed by the user's mannequin.
  • the center server 106 sends the motion image of the first user to the second terminal 104 .
  • S412 The second terminal 104 reproduces the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the second terminal 104 may play the motion image through the display of the second terminal 104, thereby reproducing the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the geographic scenario where the second user is located may be the residence or study place of the second user, including but not limited to a residence, a dormitory, a hotel, a classroom, a library, a study room, and the like.
  • the second terminal 104 may be a flat terminal, for example, a large-screen terminal such as a TV or a portable terminal such as a smart phone and a tablet computer.
  • the motion image may be a two-dimensional motion image
  • the flat terminal may play the two-dimensional motion image, thereby reproducing the two-dimensional image of the first user to the second user in the geographic scene where the second user is located.
  • the second terminal 104 may also be a stereo terminal, such as a virtual reality (virtual reality, VR) terminal or an AR terminal.
  • the VR terminal may include devices such as VR glasses and a VR head-mounted display, and the AR terminal may include AR glasses and an AR head-mounted display.
  • the motion image may be a 3D motion image, and the VR terminal and AR terminal may play the 3D motion image, so as to reproduce the 3D image of the first user to the second user in the geographic scene where the second user is located.
  • the rendering efficiency of the two-dimensional motion image is relatively high, and the second user can view the motion image of the first user in time, thereby reducing the delay.
  • the three-dimensional action image can vividly present the image of the first user, providing a better immersive experience for the second user.
  • the central server 106 can also convert the human body model of the first user to the coordinate system corresponding to the second terminal 104 for rendering, so that multiple second users can The image of the first user observed by the second terminal 104 is in the same position, thereby further enhancing the sense of immersion.
  • the edge server 108 acquires the image of the key part of the second user collected by the sensor.
  • the edge server 108 obtains the position information of the key point of the second user according to the image of the key part of the second user.
  • the edge server 108 obtains the image of the key part of the second user, and obtains the position information of the key point of the second user according to the image of the key part of the second user, and the edge server 108 obtains the image of the key part of the first user, and according to The image of the key part of the first user is similar to obtaining the position information of the key point of the first user, and details are not repeated here.
  • the edge server 108 may also directly acquire the position information of the key points corresponding to the key parts from the pose sensors deployed on the key parts of the second user.
  • the edge server 108 sends the location information of the key point of the second user to the central server 106.
  • S420 The central server 106 generates a motion image of the second user according to the position information of the key points of the second user and the human body model of the second user.
  • the central server 106 can determine the movement trajectory of the key points of the second user according to the position information of the key points of the second user, and then according to the movement trajectory of the key points of the second user and the human body of the second user stored in the central server 106 The model obtains the motion image of the second user.
  • the center server 106 sends the motion image of the second user to the first terminal 102 .
  • the first terminal 102 presents the motion image of the second user to the first user in the geographic scene where the first user is located.
  • the first terminal 102 may play the motion image of the second user through the display of the first terminal 102, thereby reproducing the motion image of the second user to the first user in the geographic scene where the first user is located.
  • the geographic scenario where the first user is located may be the residence of the first user, a teaching place, such as a residence, a dormitory, a hotel, a classroom, an office, and the like.
  • the first terminal 102 may be a flat terminal, for example, a large-screen terminal such as a TV or a portable terminal such as a smart phone and a tablet computer.
  • the motion image may be a two-dimensional motion image
  • the flat terminal may play the two-dimensional motion image, thereby reproducing the two-dimensional image of the second user to the first user in the geographic scene where the first user is located.
  • the first terminal 102 may also be a stereoscopic terminal, such as a VR terminal or an AR terminal.
  • the VR terminal may include devices such as VR glasses and a VR head-mounted display, and the AR terminal may include AR glasses and an AR head-mounted display.
  • the motion image may be a 3D motion image, and the VR terminal and AR terminal may play the 3D motion image, thereby reproducing the 3D image of the second user to the first user in the geographic scene where the first user is located.
  • the embodiment shown in FIG. 4 is illustrated in an online learning scenario.
  • the above S414 to S424 may not be performed.
  • the central server 106 may also obtain the image of the geographic scene where the second user is located, and send the image of the geographic scene where the second user is located to the first terminal 102, and the first terminal 102 reproduces the moving image of the second user to the first user. to the geographic scene where the first user is located.
  • the central server 106 may also acquire the image of the geographic scene where the second user is located and the motion image of the second user, and send the image of the geographic scene where the second user is located and the motion image of the second user to the first terminal 102 In this way, the first terminal 102 presents to the first user the image of the geographic scene where the second user is located and the motion image of the second user into the geographic scene where the first user is located.
  • an embodiment of the present application provides a remote reproduction method.
  • the method divides the remote reproduction process into two stages, specifically, the preparation stage before the remote reproduction starts and the remote reproduction stage.
  • the preparation stage a human body model of the first user is pre-built, and the human body is used in the remote reproduction stage.
  • the model generates motion images, which avoids spending a lot of time in the remote reproduction stage to rebuild the human body model, and meets the real-time requirements of remote reproduction applications.
  • more time can be spent on reconstruction to obtain a realistic human body model, which has good effects and can meet business needs.
  • the method extracts key points from images of key parts, obtains the location information of the key points, and sends the location information of the key points to the cloud instead of sending a large number of images, which not only saves network resources, but also saves computing resources, and avoids the need for It solves the problems that may seriously affect the subjective quality, such as frame loss and freezing, which may be caused by large-scale data transmission when the network quality is not good.
  • the method also allows the first user and the second user to move in a large range, and has less restriction on the movement range of the user, thereby improving the user experience.
  • the embodiment shown in FIG. 4 is mainly illustrated by the cloud server such as the central server 106 generating motion images.
  • the process of generating motion images may also be performed by a terminal (for example, the first terminal 102, the second terminal 104) Implementation.
  • the premise for the terminal to generate the action image is that the terminal has pre-built a human body model corresponding to the user, or pre-obtains the user's human body model from other devices such as a server.
  • the specific implementation of the terminal building a human body model is similar to the implementation process of the server building a human body model in the embodiment shown in FIG. 3 , and details are not described herein again.
  • the following describes a remote reproduction method of a moving image generated by a terminal with reference to the embodiments.
  • the remote reproduction system 100 includes a first terminal 102 and a second terminal 104 , and the method includes:
  • the first terminal 102 acquires the location information of the key points of the first user.
  • Keypoints are points that characterize user actions.
  • the user actions include facial actions and/or body actions.
  • the key points include, but are not limited to, facial key points such as pupils, eyebrows, and corners of the mouth, or body joint points such as fingers, elbows, and knees.
  • sensors may be deployed on the body of the first user, the first terminal 102 or the geographic scene where the first user is located to capture the information of the first user or the geographic scene where the first user is located. information.
  • the sensor includes at least one of a camera (such as an infrared camera, a visible light camera) and a pose sensor (such as an inertial measurement unit (IMU)).
  • the sensor may also include a capacitive sensor, which may be used for eye tracking.
  • the first terminal 102 may acquire images of key parts of the first user through a camera deployed on the body of the first user, the first terminal 102, or the environment (geographical scene where the first user is located) where the first user is located.
  • the key parts are the parts where key points are located, such as eyes, mouth, eyebrows, etc., and then the first terminal 102 obtains the position information of the corresponding key points according to the image of the key parts of the first user.
  • the first terminal 102 can use the key point detection model to detect the image of the key part, so as to obtain the position information of the key point.
  • the first terminal 102 may also directly capture the location information of the key points of the first user through the sensor. Specifically, the first terminal 102 may obtain the pose information of the key points of the first user from a pose sensor such as an IMU deployed in a key part, thereby obtaining the position information of the key points of the first user.
  • a pose sensor such as an IMU deployed in a key part
  • the first terminal 102 generates an action image of the first user according to the position information of the key points of the first user and the human body model of the first user.
  • a mannequin is a model used to describe the human body.
  • the human body models may be different.
  • the first terminal 102 may acquire the human body model of the first user during different actions (eg, different mouth shapes, different expressions, and different body movements).
  • the first terminal 102 can also optimize the human body model of the first user, for example, it can support beautifying, changing hair, changing clothes, etc. on the human body model, so that the human body model retains the first user's body model. In the case of features, it can also meet personalized needs, thereby improving user experience.
  • the first terminal 102 may also simplify the human body model. Specifically, the first terminal 102 can perform sparse processing on the human body model, thereby simplifying the human body model, so that the amount of rendering calculation can be reduced, and the rendering speed and rendering efficiency can be improved.
  • the first terminal 102 may determine the movement track of the key point of the first user according to the position information of the key point of the first user.
  • the key points of the first user are bound to the human body model of the first user, and the first terminal 102 can drive the human body model of the first user according to the movement trajectory of the key points of the first user, for example, according to the movement trajectory and corresponding
  • the human body model is rendered to generate an action image of the first user.
  • the movement trajectory of the key points can be represented as the position sequence of the key points.
  • the key points of the first user are bound to the human body model of the first user, the first terminal 102 can determine the human body model corresponding to each position according to the position sequence of the key points, and can form a continuous human body model according to the human body model corresponding to each position, Furthermore, an action image of the first user is generated.
  • the motion image of the first user includes a sequence of at least one motion formed by the user's mannequin.
  • the first terminal 102 sends the motion image of the first user to the second terminal 104 .
  • the second terminal 104 presents the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the second terminal 104 may play the motion image through the display of the second terminal 104, thereby reproducing the motion image of the first user to the second user in the geographic scene where the second user is located.
  • the second terminal 104 may be a flat terminal, such as a TV, a whiteboard device, a desktop computer, a notebook computer or a smart phone, etc.
  • the motion image of the first user may be a two-dimensional motion image.
  • the second terminal 104 may also be a stereo terminal, such as a VR terminal or an AR terminal, including VR glasses, VR helmet, AR glasses, and AR helmet.
  • the motion image of the first user may be a three-dimensional motion image.
  • the rendering efficiency of the two-dimensional motion image is relatively high, and the second user can view the motion image of the first user in time, thereby reducing the delay.
  • the three-dimensional action image can vividly reproduce the image of the first user, providing a better immersive experience for the second user.
  • the first terminal 102 when rendering the action image of the first user, can also convert the human body model of the first user to the coordinate system corresponding to the second terminal 104 for rendering, so that multiple second users can pass through The images of the first user observed by the respective second terminals 104 are in the same position, thereby further enhancing the sense of immersion.
  • an embodiment of the present application provides a remote reproduction method.
  • the method generates an action image by using the pre-built human body model of the first user, thereby avoiding the time-consuming construction of the human body model during remote reproduction, and meeting the real-time requirement of the remote reproduction application.
  • the preparation stage more time can be spent on reconstruction to obtain a realistic human body model, which has good effects and can meet business needs.
  • the remote reproduction stage there is no need to send a large number of images to the cloud to reconstruct the human body model in real time, which avoids large-scale data transmission occupying a lot of network resources, and problems such as frame loss and freezes in the case of poor network quality, which improves the interactive experience.
  • the embodiment shown in FIG. 5 mainly describes that the second terminal 104 plays the motion image of the first user, so as to reproduce the motion image of the first user to the second user.
  • the first terminal 102 may also play an image related to the second user, so as to present the image related to the second user to the first user.
  • the image related to the second user may be an image of a geographic scene where the second user is located.
  • the first terminal 102 may also perform the following steps:
  • the first terminal 102 receives the image of the geographic scene where the second user is located and sent by the second terminal 104.
  • the second user may be an operator at the fault site.
  • the fault site refers to the site where the fault occurs, for example, the site where the faulty equipment is located in a factory or a laboratory.
  • the image of the geographic scene where the second user is located may be an image of a fault site, such as an image showing a faulty device.
  • the first terminal 102 may be a VR terminal, such as VR glasses.
  • the second terminal may be an AR terminal, such as AR glasses.
  • the image of the first user can be superimposed with the image of the fault site, and then projected to the AR terminal of the on-site operator.
  • the operator at the scene can observe the first user through the AR terminal, and the image of the first user may not move with the operator. From the perspective of the second user, the first user can walk around, point, speak, etc., just like Visiting the scene in person enhances the interactive experience.
  • the first terminal 102 presents the image of the geographic scene where the second user is located to the geographic scene where the first user is located to the first user.
  • the first user may be an expert with fault diagnosis or fault maintenance capabilities
  • the first terminal 102 may play an image of the geographic scene where the second user is located, thereby reproducing the fault scene where the second user is located to the first user to the geographic scene where the first user is located.
  • the image of the geographic scene where the second user is located and sent by the second terminal 104 may also be an image of the fault scene from the perspective of the first user.
  • the second terminal 104 can obtain the pupil position of the first user, and the pupil position can be obtained by eye tracking, and then the second terminal 104 can determine the first user's perspective according to the pupil position of the first user, and the second terminal 104 can use the The fault scene image from the first user's perspective is obtained by means of on-site shooting, view synthesis or environmental simulation.
  • on-site shooting refers to driving a binocular camera through a robotic arm
  • the robotic arm is driven by the pupil position of the first user
  • the binocular camera is driven by the robotic arm to shoot the fault scene
  • the second terminal 104 can access the binocular camera from the binocular camera.
  • the eye camera obtains the fault scene image from the first user's perspective.
  • Perspective synthesis refers to pre-deploying several fixed cameras in the geographic scene (such as the fault site) where the second user is located, and using the videos captured by cameras with multiple perspectives close to the first user's perspective to perform video synthesis to obtain the first user.
  • Environment simulation refers to modeling the environment of the fault site in advance to obtain an environment model, then establishing a coordinate system from a first user's perspective (eg, an expert's perspective), and rendering the environment model, thereby obtaining an image of the fault site from the first user's perspective.
  • a first user's perspective eg, an expert's perspective
  • the above-mentioned S510 to S512 may also be performed before S502.
  • the second terminal 104 may first send an image of the geographic scene where the second user, such as an operator, is located, such as a fault scene image, and the first terminal 102 plays the fault scene image and reproduces the fault scene image to the first user, such as an expert to the geographic scene where the first user is located.
  • the first user can take corresponding actions according to the fault scene, for example, look to the left with the eyes, and the first terminal 102 can determine the position information of key points including the pupil position according to the image of the eye part or the eye tracking result, An action image of the first user is generated according to the position information of the key points and the human body model.
  • the first terminal 102 transmits the moving image to the second terminal 104 .
  • the second terminal 104 may determine the first user's perspective according to the motion image, acquire the fault scene image from the first user's perspective, and return to the first terminal 102 .
  • the first user finds out the cause of the fault according to the fault scene image, he can also instruct the fault maintenance method through actions and voice, and the first terminal 102 can generate an action image indicating the fault maintenance method and send it to the second terminal 104 .
  • the image related to the second user may be a motion image of the second user.
  • the first terminal 102 may also perform the following steps:
  • the first terminal 102 receives the motion image of the second user sent by the second terminal 104 .
  • the first user may be a coach or teacher with professional ability
  • the second user may be a student or student learning professional skills or knowledge.
  • the first user pays more attention to the second user itself, rather than the geographic scene where the second user is located.
  • the second terminal 104 may acquire the motion image of the second user, and send the motion image of the second user to the first terminal 102 .
  • the first terminal 102 and the second terminal 104 may both be VR terminals, such as VR glasses.
  • the first terminal 102 and the second terminal 104 may be other types of terminals.
  • S516 The first terminal 102 presents a motion image of the second user to the first user.
  • the first terminal 102 presents the motion image of the second user to the first user in the geographic scene where the first user is located.
  • the first user such as coaches and teachers can judge the mastery of professional skills and professional knowledge of students and students according to the action images of second users such as students and students, and make targeted corrections according to the situation, thereby improving the efficiency of online learning. As well as improving the user experience of online learning.
  • S514 and S516 may be performed after S508, or may be performed before S502, which is not limited in this embodiment of the present application.
  • the system 100 includes a first terminal 102 and a second terminal 104 , the first terminal 102 is worn by the first user, and the second terminal 104 Worn by a second user, the first user and the second user are in different geographic scenarios:
  • the second terminal 104 is configured to acquire the motion image of the first user, and reproduce the motion image of the first user to the second user in the geographic scene where the second user is located, the
  • the action image of the first user includes a sequence of at least one action formed by the human body model of the first user, and the human body model of the first user is a pre-built model.
  • the remote reproduction system 100 further includes a server, such as a central server 106 and an edge server 108;
  • the edge server 108 configured to obtain the location information of the key points of the first user
  • the central server 106 is configured to determine the movement track of the key point of the first user according to the position information of the key point of the first user, according to the movement track and the first user stored in the server The human body model is obtained, and the action image of the first user is obtained.
  • the server (eg, edge server 108 ) is specifically used to:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the server (eg, edge server 108 ) is specifically used to:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the first terminal 102 is used for:
  • the image of the geographic scene where the second user is located is reproduced to the first user into the geographic scene where the first user is located.
  • the first terminal is a virtual reality terminal
  • the second terminal is an augmented reality terminal.
  • the first terminal 102 is used for:
  • the motion image of the second user includes a sequence of at least one motion formed by a human body model of the second user, and the human body model of the second user is a pre-built model;
  • the motion image of the second user is reproduced to the first user in a geographic scene where the first user is located.
  • the first terminal and the second terminal are virtual reality terminals.
  • the first terminal 102 is used for:
  • a motion image of the first user is obtained.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • the remote reproduction system 100 may correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and/or functions of the various components of the remote reproduction system 100 are for the purpose of realizing FIG. 4 and FIG. 4 , respectively.
  • FIG. 4 and FIG. 4 respectively.
  • the corresponding processes of each method in the embodiment shown in 5 will not be repeated here.
  • the embodiment of the present application provides a corresponding remote reproduction apparatus.
  • the remote reproduction device may be a software device, and the remote reproduction device is deployed in a second terminal 104, the second terminal 104 being worn by a second user, who is in a different geographic scene from the first user.
  • the apparatus 600 includes:
  • the image acquisition unit 602 is configured to acquire a motion image of the first user, the motion image of the first user includes a sequence of at least one motion formed by the human body model of the first user, and the human body model of the first user for pre-built models;
  • the image reproduction unit 604 is configured to reproduce the action image of the first user to the second user in the geographic scene where the second user is located.
  • the apparatus 600 further includes:
  • the image sending unit 606 is configured to send a motion image of the second user to the first terminal, where the motion image of the second user includes a sequence of at least one motion formed by the human body model of the second user, and the motion image of the second user Mannequins are pre-built models.
  • the apparatus 600 further includes:
  • the image generation unit 608 is configured to obtain the position information of the key points of the second user, determine the movement trajectory of the key points of the second user according to the position information of the key points of the second user, and determine the movement trajectory of the key points of the second user according to the position information of the key points of the second user.
  • the trajectory and the human body model of the second user stored in the second terminal 104 are used to obtain a motion image of the second user.
  • the image generation unit 608 is specifically configured to:
  • a key part image of the second user is acquired, and position information of a corresponding key point is obtained according to the key part image of the second user.
  • the image generation unit 608 is specifically configured to:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the second user.
  • the apparatus 600 further includes:
  • the image sending unit 606 is configured to send the image of the geographic scene where the second user is located to the first terminal.
  • the image acquisition unit 602 is specifically configured to:
  • the motion video of the first user is acquired from the server or the first terminal.
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • the remote reproducing apparatus 600 may correspond to executing the methods described in the embodiments of the present application, and the above-mentioned and other operations and/or functions of the respective modules/units of the remote reproducing apparatus 600 are intended to realize FIG. 4 , Corresponding processes of each method in the embodiment shown in FIG. 5 are not repeated here for brevity.
  • the embodiment of the present application provides another remote reproduction apparatus.
  • the remote reproduction means may be a software means deployed in the server.
  • the apparatus 700 includes:
  • a location acquiring unit 702 configured to acquire location information of key points of the first user
  • a trajectory determination unit 704 configured to determine the movement trajectory of the key point of the first user according to the position information of the key point of the first user;
  • the image generation unit 706 is configured to obtain the motion image of the first user according to the movement trajectory and the human body model of the first user stored in the server (for example, the central server 106 ).
  • the motion image of the first user including a sequence of at least one action formed by a mannequin of the first user, the mannequin of the first user being a pre-built model;
  • the image sending unit 708 is configured to send the motion image of the first user to the second terminal, so that the second terminal reproduces the motion image of the first user to the second user to the location where the second user is located In the geographic scene, the first user and the second user are in different geographic scenes.
  • the location obtaining unit 702 is specifically configured to:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the location obtaining unit 702 is specifically configured to:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the location obtaining unit 702 is further configured to:
  • the trajectory determination unit 704 is also used for:
  • the image generation unit 706 is also used for:
  • a motion image of the second user is obtained, and the motion image of the second user includes at least one action formed by the human body model of the second user
  • the sequence of the second user is determined, and the human body model of the second user is a pre-built model
  • the image sending unit 708 is also used for:
  • the image generation unit 706 is further configured to:
  • the image sending unit 708 is also used for:
  • the image of the geographic scene where the second user is located is sent to the first terminal.
  • the remote reproducing apparatus 700 may correspond to executing the methods described in the embodiments of the present application, and the above-mentioned and other operations and/or functions of the respective modules/units of the remote reproducing apparatus 700 are for the purpose of realizing FIG. 4 , Corresponding processes of each method in the embodiment shown in FIG. 5 are not repeated here for brevity.
  • the embodiment of the present application provides another remote reproduction apparatus.
  • the remote reproduction means may be a software means deployed in the first terminal 102 .
  • the first terminal 102 is worn by a first user, and the first user and the second user are in different geographical scenes.
  • the apparatus 800 includes:
  • An image acquisition unit 802 configured to acquire an image of the geographic scene where the second user is located and/or an action image of the second user, where the action image of the second user includes a human body model formed by the second user The sequence of at least one action of the second user, the human body model of the second user is a pre-built model;
  • An image reproduction unit 804 configured to reproduce to the first user the image of the geographic scene where the second user is located and/or the motion image of the second user to the geographic scene where the first user is located middle.
  • the apparatus 800 further includes:
  • the image sending unit 806 is configured to send a motion image of the first user to the second terminal, where the motion image of the first user includes a sequence of at least one motion formed by the human body model of the first user, and the motion image of the first user Mannequins are pre-built models.
  • the apparatus 800 further includes:
  • the image generation unit 808 is configured to acquire the position information of the key points of the first user, determine the movement trajectory of the key points of the first user according to the position information of the key points of the first user, and determine the movement trajectory of the key points of the first user according to the position information of the key points of the first user.
  • the trajectory and the human body model of the first user stored in the first terminal are used to obtain a motion image of the first user.
  • the image generation unit 808 is specifically configured to:
  • a key part image of the first user is acquired, and position information of a corresponding key point is obtained according to the key part image of the first user.
  • the image generation unit 808 is specifically configured to:
  • the position information of the key point corresponding to the key part is acquired from the pose sensor deployed on the key part of the first user.
  • the apparatus 800 further includes:
  • the image sending unit 806 is configured to send the image of the geographic scene where the first user is located to the second terminal.
  • the image acquisition unit 802 is specifically configured to:
  • the motion video of the second user is acquired from the server or the second terminal 104 .
  • the human body model is a three-dimensional human body model
  • the motion image is a three-dimensional motion image
  • the remote reproducing apparatus 800 may correspond to executing the methods described in the embodiments of the present application, and the above-mentioned and other operations and/or functions of the respective modules/units of the remote reproducing apparatus 800 are for the purpose of realizing FIG. 4 , For the sake of brevity, the corresponding processes of each method in the embodiment shown in FIG. 5 will not be repeated here.
  • the embodiment of the present application further provides a second terminal 104 .
  • the second terminal 104 is specifically used to implement the functions of the remote reproduction apparatus 600 in the embodiment shown in FIG. 6 above.
  • FIG. 9 provides a schematic structural diagram of a second terminal 104 .
  • the second terminal 104 includes a bus 1041 , a processor 1042 , a display 1043 and a memory 1044 . Communication between the processor 1042 , the memory 1044 and the display 1043 is through the bus 1041 .
  • the bus 1041 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.
  • the processor 1042 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc. any one or more of the devices.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • the display 1043 is an input/output (I/O) device. According to different manufacturing materials, the display 1043 can be classified into a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and the like. Specifically, the display 1043 can reproduce the motion image of the first user to the second user in the geographic scene where the second user is located.
  • LCD liquid crystal display
  • OLED organic light emitting diode
  • Memory 1044 may include volatile memory, such as random access memory (RAM).
  • RAM random access memory
  • the memory 1044 may also include non-volatile memory (non-volatile memory) such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (solid state drive) , SSD).
  • non-volatile memory such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (solid state drive) , SSD).
  • Executable code is stored in the memory 1044, and the processor 1042 executes the executable code to perform the aforementioned remote reproduction method.
  • the image acquisition unit in FIG. 6 is executed.
  • the image reproduction unit 604 may be partially or fully stored in the memory 1044 .
  • the processor 1042 executes the program code corresponding to each unit stored in the memory 1044 to execute the remote reproduction method.
  • the embodiment of the present application also provides a server.
  • the server is specifically used to implement the functions of the remote reproduction apparatus 700 in the embodiment shown in FIG. 7 above.
  • the server may be the central server 106 or the edge server 108, and the embodiment of the present application uses the central server 106 as an example for description.
  • FIG. 10 provides a schematic structural diagram of a central server 106 .
  • the central server 106 includes a bus 1061 , a processor 1062 , a communication interface 1063 and a memory 1064 .
  • the processor 1062 , the memory 1064 and the communication interface 1063 communicate through the bus 1061 .
  • the communication interface 1063 is mainly used to communicate with the outside, for example, to obtain the position information of the key points of the first user from the pose sensor or the edge server 108, and to send the motion image of the first user to the second terminal 104, for example, from The pose sensor or the edge server 108 acquires the position information of the key points of the second user, and sends the motion image of the first user to the first terminal 102 .
  • the position obtaining unit in FIG. 7 is executed.
  • 702 , the trajectory determination unit 704 , the image generation unit 706 , and the software or program codes required for the functions of the image transmission unit 708 may be partially or fully stored in the memory 1064 .
  • the processor 1062 executes the program code corresponding to each unit stored in the memory 1064 to execute the remote reproduction method.
  • This embodiment of the present application further provides a first terminal 102 .
  • the first terminal 102 is specifically used to implement the functions of the remote reproduction apparatus 800 in the embodiment shown in FIG. 8 above.
  • FIG. 11 provides a schematic structural diagram of a first terminal 102 .
  • the first terminal 102 includes a bus 1021 , a processor 1022 , a display 1023 and a memory 1024 . Communication between the processor 1022 , the memory 1024 and the display 1023 is through the bus 1021 .
  • the specific implementation of the bus 1021 , the processor 1022 , the display 1023 and the memory 1024 may refer to the description of the relevant content of the embodiment shown in FIG. 9 .
  • the image acquisition unit in FIG. 8 is executed.
  • the image reproduction unit 804 , the image transmission unit 806 , and the software or program codes required for the functions of the image generation unit 808 may be partially or fully stored in the memory 1024 .
  • the processor 1022 executes the program code corresponding to each unit stored in the memory 1024 to execute the remote reproduction method.
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer-readable storage medium may be any available medium that a computing device can store, or a data storage device such as a data center that contains one or more available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state drives), and the like.
  • the computer-readable storage medium includes instructions, the instructions instruct the computing device to execute the above-mentioned remote reproduction method applied to the remote reproduction apparatus 600 , the remote reproduction apparatus 700 , or the remote reproduction apparatus 800 .
  • Embodiments of the present application also provide a computer program product, where the computer program product includes one or more computer instructions.
  • the computer instructions When the computer instructions are loaded and executed on the computing device, all or part of the processes or functions described in the embodiments of the present application are generated.
  • the computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted over a wire from a website site, computer or data center. (eg coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) to another website site, computer or data center.
  • a website site e.g coaxial cable, fiber optic, digital subscriber line (DSL)
  • wireless eg infrared, wireless, microwave, etc.
  • the computer program product When the computer program product is executed by a computer, the computer executes any of the aforementioned remote reproduction methods.
  • the computer program product can be a software installation package, which can be downloaded and executed on a computer if any one of the aforementioned remote reproduction methods needs to be used.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Geometry (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Remote Sensing (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请提供了一种远程重现方法，应用于远程重现系统，该系统包括由第一用户佩戴的第一终端和由第二用户佩戴的第二终端，第一用户与第二用户处于不同地理场景，该方法包括：第二终端获取第一用户的动作影像，第一用户的动作影像包括预先构建的、第一用户的人体模型形成的至少一个动作的序列，然后第二终端向第二用户重现第一用户的动作影像至第二用户所处的地理场景中。该方法在第二终端侧重现第一用户的影像时无需实时传输多视角图像，减少了网络资源占用，避免在网络质量不好情况下出现丢帧、卡顿，提升了交互体验。

Description

远程重现方法、系统、装置、设备、介质及程序产品
本申请要求于2021年01月26日提交中国国家知识产权局、申请号为202110105037.0、发明名称为“远程重现方法、系统、装置、设备、介质及程序产品”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种远程重现方法、系统、装置、设备以及计算机可读存储介质、计算机程序产品。
背景技术
在很多应用场景中，由于地理位置的限制，人和人之间难以就同一场景或事件进行面对面交流。真实的面对面交流需要花费较高的成本，而且效率也比较低下。随着计算机技术和通信技术的不断发展，远程重现对方的影像，从而达到如同面对面交流的效果逐渐成为了一种可能。
远程重现影像的一种典型应用场景是远程协助。远程协助是指协助者通过远程通信的方式协助被协助者，以解决该被协助者遇到的问题。例如，设备维修场景中，具有诊断或维修能力的专家通过远程通信的方式指导位于故障现场的设备操作员。目前，业界提供了全息捕捉重现的方案，将远程的协助者的影像在故障现场进行重现，从而提供更为自然的交互。
然而，上述方案通常是利用多视角图像对三维人体进行实时重建。比较理想的人体模型通常需要花费数小时的运算，因此实时重建的效果通常较差，难以满足业务需求。并且实时传输多视角图像至云端或终端进行远程重建，需要占用大量网络资源，在网络质量不好的情况下，还容易出现丢帧、卡顿等问题，影响交互体验。
发明内容
本申请提供了一种远程重现方法，该方法在准备阶段预先构建逼真的人体模型，然后在远程重现阶段利用预先构建的人体模型生成动作影像，向用户重现动作影像以实现如同面对面交流的效果。由此解决模型重建效果较差导致难以满足业务需求的问题，并且无需实时传输多视角图像，减少了网络资源占用，避免在网络质量不好情况下出现丢帧、卡顿，提升了交互体验。本申请还提供了上述方法对应的系统、装置、设备、计算机可读存储介质以及计算机程序产品。
第一方面，本申请提供了一种远程重现方法。该方法可以由远程重现系统执行。远程重现系统包括第一终端和第二终端，第一终端由第一用户佩戴，第二终端由第二用户佩戴，第一用户与第二用户处于不同地理场景。
具体地，第二终端可以获取第一用户的动作影像，其中，第一用户的动作影像包括第一用户的人体模型形成的至少一个动作的序列，第一用户的人体模型为预先构建的模型， 然后第二终端向第二用户重现上述第一用户的动作影像至第二用户所处的地理场景中。
一方面，由于在准备阶段预先构建了第一用户的人体模型，在远程重现阶段直接利用该人体模型生成动作影像，由此避免了远程重现时花费大量时间构建人体模型，满足了远程重现应用对实时性的需求，而且在准备阶段可以花费较多的时间进行重建，得到逼真的人体模型，具有较好的效果，能够满足业务需求。另一方面，远程重现阶段无需发送大量图像到云端或终端实时重建人体模型，避免了大规模数据传输占用大量网络资源，以及网络质量不好的情况下出现丢帧、卡顿等问题，提升了交互体验。
在一些可能的实现方式中，远程重现系统还可以包括服务器。远程重现系统可以利用服务器的算力生成第一用户的动作影像。具体地，服务器获取第一用户的关键点的位置信息，服务器可以根据第一用户的关键点的位置信息，确定第一用户的关键点的移动轨迹，然后服务器可以根据上述移动轨迹以及存储在服务器中的第一用户的人体模型，获得所述第一用户的动作影像，服务器发送第一用户的动作影像至第二终端。
其中，关键点的移动轨迹可以表示为关键点的位置序列。第一用户的关键点与第一用户的人体模型绑定，服务器可以根据关键点的位置序列确定每一个位置对应的人体模型，根据各个位置对应的人体模型可以形成连续的人体模型，进而生成第一用户的动作影像。第一用户的动作影像中包括用户的人体模型形成的至少一个动作的序列。
由于需要耗费大量算力的影像生成过程在服务器侧执行，第二终端仅需接收第一用户的动作影像，向第二用户呈现第一用户的动作影像至第一用户所处的地理场景中，如此降低了对第二终端的要求，轻量级的第二终端也可以用于实现远程重现方法，具有较高可用性。
在一些可能的实现方式中，远程重现系统可以包括云端的服务器，例如是中心云中的中心服务器和边缘云中的边缘服务器。边缘服务器和中心服务器可以协同生成动作影像。具体地，边缘服务器可以获取第一用户的关键点的位置信息，例如通过图像传感器采集的第一用户的关键部位图像进行关键点检测，获得对应的关键点的位置信息，中心服务器根据边缘服务器获得的关键点的位置信息，确定第一用户的关键点的移动轨迹，然后根据移动轨迹和存储在中心服务器中的第一用户的人体模型，获得第一用户的动作影像。
由于边缘云服务器和中心云服务器之间仅传输关键点的位置信息，而不需要传输大规模的图像数据，减少了网络资源占用，避免了网络质量不好情况下出现丢帧、卡顿等问题。
在一些可能的实现方式中，服务器可以获取第一用户的关键部位图像，然后根据第一用户的关键部位图像获得对应的关键点的位置信息。具体地，服务器可以利用关键点检测技术对第一用户的关键部位图像进行处理，获得第一用户的关键点的位置信息。例如，服务器可以构建关键点检测模型，然后将第一用户的关键部位图像输入关键点检测模型，通过关键点检测模型对图像提取关键点，获得第一用户的关键点的位置信息。
其中，关键点检测模型可以是传统的关键点检测模型。传统的关键点检测模型包括基于模板匹配的关键点检测模型，例如是图结构模型。关键点检测模型也可以是基于神经网络的关键点检测模型。基于神经网络的关键点检测模型包括但不限于级联特征网络模型、区域多人位姿估计模型、基于掩膜的区域卷积神经网络模型和级联金字塔网络模型等。
该方法通过利用关键点检测模型对图像进行检测获得关键点的位置信息，减少了远程 重现系统中传感器的类型、数量，降低了远程重现系统的复杂度。
在一些可能的实现方式中，服务器可以从部署在第一用户的关键部位的位姿传感器，如惯性测量单元，获取与所述关键部位对应的关键点的位置信息。由于无需通过复杂的模型进行检测，提高了获取关键点的位置信息的效率，从而提高了远程重现的效率。
在一些可能的实现方式中，第一终端还可以获取第二用户所处的地理场景的影像，然后第一终端向所述第一用户重现所述第二用户所处的地理场景的影像至所述第一用户所处的地理场景中。如此，第一用户可以根据第二用户所处的地理场景的影像，远程协助第二用户解决第二用户所处的地理场景中的问题，例如是故障现场的问题。
在一些可能的实现方式，第一终端为虚拟现实终端，第二终端为增强现实终端，该增强现实终端可以将现场影像与第一用户的影像叠加进行呈现，从而提高沉浸感。
在一些可能的实现方式中，第一终端可以获取所述第二用户的动作影像，该第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，该第二用户的人体模型为预先构建的模型，第一终端向所述第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。如此，第一用户可以根据第二用户的动作影像，对第一用户进行针对性纠正，由此可以提高在线学习的效率。
在一些可能的实现方式中，所述第一终端和所述第二终端为虚拟现实终端。在线学习等场景中，通过虚拟现实终端对现实环境进行虚拟，可以提高交互性。
在一些可能的实现方式中，当第一终端的算力足够强大，例如是能够支持根据人体模型生成动作影像时，则可以不用借助云端计算能力生成动作影像，而是由第一终端直接根据预先构建的第一用户的人体模型，在本地生成第一用户的动作影像，然后向第二终端下发第一用户的动作影像。
具体地，第一终端获取第一用户的关键点的位置信息，然后第一终端根据第一用户的关键点的位置信息，确定第一用户的关键点的移动轨迹，根据所述移动轨迹以及存储在第一终端中的第一用户的人体模型，获得所述第一用户的动作影像。
由于无需经过服务器中转上述动作影像，一方面降低了远程重现系统的成本，另一方面减少了延迟。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。三维动作影像可以生动地重现第一用户的形象，为第二用户提供较好的沉浸式体验。在一些实施例中，人体模型也可以为二维人体模型，二维人体模型的渲染效率较高，如此，第二用户可以及时查看第一用户的动作影像，降低了延迟。
在一些可能的实现方式中，所述动作影像可以用于远程协助(例如故障维修)或在线学习等场景，从而为用户提供如同面对面交流的效果。
第二方面，本申请提供了一种远程重现系统。所述系统包括第一终端和第二终端，所述第一终端由第一用户佩戴，所述第二终端由第二用户佩戴，所述第一用户与所述第二用户处于不同地理场景；
所述第二终端，用于获取所述第一用户的动作影像，向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述远程重现系统还包括服务器；
所述服务器，用于获取所述第一用户的关键点的位置信息，根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹，根据所述移动轨迹和存储在服务器中的第一用户的人体模型，获得所述第一用户的动作影像，服务器还用于发送所述第一用户的动作影像至所述第二终端。
在一些可能的实现方式中，所述服务器具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述服务器具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述第一终端用于：
获取所述第二用户所处的地理场景的影像；
向所述第一用户重现所述第二用户所处的地理场景的影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述第一终端为虚拟现实终端，所述第二终端为增强现实终端。
在一些可能的实现方式中，所述第一终端用于：
获取所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
向所述第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述第一终端和所述第二终端为虚拟现实终端。
在一些可能的实现方式中，所述第一终端用于：
获取所述第一用户的关键点的位置信息；
根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
根据所述移动轨迹以及存储在第一终端中的第一用户的人体模型，获得所述第一用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
第三方面，本申请实施例提供了一种远程重现装置。所述远程重现装置部署在第二终端中，所述第二终端由第二用户佩戴，所述第二用户与第一用户处于不同地理场景，所述装置包括：
影像获取单元，用于获取所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
影像重现单元，用于向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中。
在一些可能的实现方式中，所述装置还包括：
影像发送单元，用于向第一终端发送第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述装置还包括：
影像生成单元，用于获取所述第二用户的关键点的位置信息，根据所述第二用户的关键点的位置信息，确定所述第二用户的关键点的移动轨迹，根据所述移动轨迹以及存储在所述第二终端中的所述第二用户的人体模型，获得所述第二用户的动作影像。
在一些可能的实现方式中，所述影像生成单元具体用于：
获取所述第二用户的关键部位图像，根据所述第二用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述影像生成单元具体用于：
从部署在所述第二用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述装置还包括：
影像发送单元，用于向第一终端发送第二用户所处的地理场景的影像。
在一些可能的实现方式中，所述影像获取单元具体用于：
从服务器或者第一终端获取所述第一用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
第四方面，本申请实施例提供了一种远程重现装置。所述远程重现装置部署在服务器中，所述装置包括：
位置获取单元，用于获取第一用户的关键点的位置信息；
轨迹确定单元，用于根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
影像生成单元，用于根据所述移动轨迹以及存储在所述服务器中的所述第一用户的人体模型，获得所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
影像发送单元，用于向第二终端发送所述第一用户的动作影像，以使所述第二终端向第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户与所述第二用户处于不同地理场景。
在一些可能的实现方式中，所述位置获取单元具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述位置获取单元具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述位置获取单元还用于：
获取第二用户的关键点的位置信息；
所述轨迹确定单元还用于：
根据所述第二用户的关键点的位置信息，确定所述第二用户的关键点的移动轨迹；
所述影像生成单元还用于：
根据所述移动轨迹以及存储在所述服务器中的所述第二用户的人体模型，获得所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
所述影像发送单元还用于：
向第一终端发送所述第二用户的动作影像，以使所述第一终端向第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述影像生成单元还用于：
生成第二用户所处的地理场景的影像；
所述影像发送单元还用于：
向第一终端发送所述第二用户所处的地理场景的影像。
第五方面，本申请实施例提供了一种远程重现装置。所述远程重现装置部署在第一终端中，所述第一终端由第一用户佩戴，所述第一用户与第二用户处于不同地理场景，所述装置包括：
影像获取单元，用于获取所述第二用户所处的地理场景的影像和/或所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
影像重现单元，用于向所述第一用户重现所述第二用户所处的地理场景的影像和/或所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述装置还包括：
影像发送单元，用于向第二终端发送第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述装置还包括：
影像生成单元，用于获取所述第一用户的关键点的位置信息，根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹，根据所述移动轨迹以及存储在所述第一终端中的所述第一用户的人体模型，获得所述第一用户的动作影像。
在一些可能的实现方式中，所述影像生成单元具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述影像生成单元具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述装置还包括：
影像发送单元，用于向第二终端发送第一用户所处的地理场景的影像。
在一些可能的实现方式中，所述影像获取单元具体用于：
从服务器或者第二终端获取所述第二用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
第六方面，本申请提供一种第二终端，所述第二终端包括处理器和存储器。所述处理器、所述存储器进行相互的通信。所述处理器用于执行所述存储器中存储的指令，以使得所述第二终端实现如第三方面所述的远程重现装置的功能。
第七方面，本申请提供一种服务器，所述服务器包括处理器和存储器。所述处理器、所述存储器进行相互的通信。所述处理器用于执行所述存储器中存储的指令，以使得所述服务器实现如第四方面所述的远程重现装置的功能。
第八方面，本申请提供一种第一终端，所述第一终端包括处理器和存储器。所述处理器和所述存储器进行相互的通信。所述处理器用于执行所述存储器中存储的指令，以使得所述服务器实现如第五方面所述的远程重现装置的功能。
第九方面，本申请提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，所述指令指示第二终端执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述第二终端执行的步骤。
第十方面，本申请提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，所述指令指示服务器执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述服务器执行的步骤。
第十一方面，本申请提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，所述指令指示第一终端执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述第一终端执行的步骤。
第十二方面，本申请提供了一种包含指令的计算机程序产品，当其在第二终端上运行时，使得第二终端执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述第二终端执行的步骤。
第十三方面，本申请提供了一种包含指令的计算机程序产品，当其在服务器上运行时，使得服务器执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述服务器执行的步骤。
第十四方面，本申请提供了一种包含指令的计算机程序产品，当其在第一终端上运行时，使得第一终端执行上述第一方面或第一方面的任一种实现方式所述的远程重现方法中由所述第一终端执行的步骤。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
为了更清楚地说明本申请实施例的技术方法，下面将对实施例中所需使用的附图作以简单地介绍。
图1为本申请实施例提供的一种远程重现系统的系统架构图；
图2为本申请实施例提供的另一种远程重现系统的系统架构图；
图3为本申请实施例提供的一种人体模型的构建方法的流程图；
图4为本申请实施例提供的一种远程重现方法的流程图；
图5为本申请实施例提供的一种远程重现方法的流程图；
图6为本申请实施例提供的一种远程重现装置的结构示意图；
图7为本申请实施例提供的一种远程重现装置的结构示意图；
图8为本申请实施例提供的一种远程重现装置的结构示意图；
图9为本申请实施例提供的一种第二终端的结构示意图；
图10为本申请实施例提供的一种服务器的结构示意图；
图11为本申请实施例提供的一种第一终端的结构示意图。
具体实施方式
本申请实施例中的术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
首先对本申请实施例中所涉及到的一些技术术语进行介绍。
远程重现是指通过远程通信技术进行影像重现，从而打破地理位置的限制，使得不同地理位置的人可以针对同一场景或事件进行交流，并达到如同面对面交流的效果。其中，重现的影像包括人的影像，或者是人所处的地理场景的影像。
远程重现可以应用于多个领域的不同场景中。例如，远程重现可以用于远程协助场景，或者是在线学习场景。其中，远程协助场景是指协助者通过远程通信的方式协助被协助者，以解决该被协助者遇到的问题的场景。例如在远程协助场景中，协助者可以是具有诊断或维修能力的专家，被协助者可以是位于故障现场的操作员。专家可以通过远程通信的方式指导位于故障现场的操作员，以排查故障原因，并根据故障原因进行故障恢复。又例如在线学习场景中，老师或者教练可以通过远程通信的方式指导学生或学员在线学习。
针对不同地理位置的人的交流问题，业界提出了基于全息捕捉重现的远程重现方案，该方案能够将人的影像在现场重现，从而提供更为自然的交互体验。
其中，基于全息捕捉重现的远程重现方案具体为，部署在不同方位的同步摄像头同时拍摄待重现影像的对象，例如是拍摄协助者，获得协助者的多视角图像，基于该多视角图像可以对三维人体进行实时重建，得到三维人体模型。被协助者的增强现实(augmented reality，AR)设备如AR眼镜以被协助者的观察位置建立坐标系，将重建的三维人体模型放置在该坐标系，然后渲染成双目视频，AR眼镜向被协助者呈现该双目视频，由此实现将协助者影像在现场重现。
然而，比较理想的三维人体重建往往需要数小时的计算，而远程重现应用对实时性要求非常高，实时重建的人体模型效果较差，难以满足业务的需求。并且，实时传输多视角图像至云端或终端进行远程重建，需要占用大量网络资源，并且在网络质量不好的情况下，容易出现丢帧、卡顿等问题，影响交互体验。
有鉴于此，本申请实施例提供了一种远程重现方法。该方法由远程重现系统执行。该远程重现系统包括第一终端和第二终端。其中，第一终端由第一用户佩戴，第二终端由第 二用户佩戴。第一用户和第二用户为交流过程的参与方。远程协助场景中，第一用户可以为协助者，例如是具有诊断或维修能力的专家，第二用户可以为被协助者，例如是故障现场的操作员。在线学习场景中，第一用户可以为老师或教练，第二用户可以为学生、学员等。
具体地，第二终端可以获取第一用户的动作影像，其中，第一用户的动作影像包括第一用户的人体模型形成的至少一个动作的序列，第一用户的人体模型为预先构建的模型，然后第二终端向第二用户重现上述第一用户的动作影像至第二用户所处的地理场景中。
一方面，该方法通过将远程重现过程分为两个阶段，具体是远程重现开始前的准备阶段和远程重现阶段，在准备阶段预先构建第一用户的人体模型，在远程重现阶段直接利用该人体模型生成动作影像，由此避免了远程重现时花费大量时间构建人体模型，满足了远程重现应用对实时性的需求，而且在准备阶段可以花费较多的时间进行重建，得到逼真的人体模型，具有较好的效果，能够满足业务需求。另一方面，远程重现阶段无需发送大量图像到云端或者终端实时重建人体模型，避免了大规模数据传输占用大量网络资源，以及网络质量不好的情况下出现丢帧、卡顿等问题，提升了交互体验。
本申请实施例提供的远程重现方法可以应用到不同场景。在一些实施例中，该远程重现方法可以应用于远程协助场景，以使具备诊断或维修能力的专家远程协助故障现场的操作员进行故障排查和/或故障恢复。在另一些实施例中，该远程重现方法可以应用于在线学习场景，例如是在线健身学习、在线实验课程学习等场景，以使教师或教练远程辅导学生或学员进行学习。
具体可以参见如图1所示的远程重现系统100的系统架构图，该远程重现系统100包括第一终端102和第二终端104。第一终端102由第一用户佩戴，第二终端104由第二用户佩戴，第一用户与第二用户处于不同地理场景。
其中，第二终端104具体用于获取第一用户的动作影像，第一用户的动作影像包括第一用户的人体模型形成的至少一个动作的序列，第一用户的人体模型为预先构建的模型，然后向第二用户重现呈现所述第一用户的动作影像至所述第二用户所处的地理场景中。
在一些可能的实现方式中，第一终端102具体用于获取第二用户所处的地理场景的影像，或者是获取第二用户的动作影像，第二用户的动作影像包括第二用户的人体模型形成的至少一个动作的序列，该第二用户的人体模型为预先构建的模型，然后向第一用户重现第二用户所处的地理场景的影像至所述第一用户所处的地理场景中，或者是向第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
考虑到第一终端102、第二终端104为轻量级终端，计算能力较弱，不足以支持根据人体模型生成动作影像或者不足以支持生成地理场景的影像的情况，远程重现系统100还可以借助云端计算能力生成动作影像。具体地，云端的计算设备如中心服务器可以根据第一用户的人体模型生成第一用户的动作影像，然后向第二终端104下发第一用户的动作影像。
具体可以参见如图1所示的远程重现系统100的系统架构图，该远程重现系统100还包括云环境中的中心服务器106。中心服务器106可以根据预先构建的、第一用户的人体 模型生成第一用户的动作影像，然后向第二终端104发送第一用户的动作影像。第二终端104向第二用户呈现第一用户的动作影像至第二用户所处的地理场景中。
进一步地，远程重现系统100还可以包括边缘环境中的边缘服务器108。边缘服务器108可以获取第一用户的关键点的位置信息。其中，关键点是指表征用户动作的点，包括但不限于瞳孔、眉心、嘴角等面部关键点，或者手指、手肘、膝盖等身体关节点。对应地，中心服务器106根据第一用户的关键点的位置信息和第一用户的人体模型生成第一用户的动作影像。如此可以避免发送大量图像到云端，既节省了网络资源，又节省了计算资源，而且还避免了大规模数据传输在网络质量不好的时候可能带来的丢帧、卡顿等严重影响影像质量的问题发生。
需要说明的是，获取第一用户的关键点的位置信息的步骤也可以由中心服务器106执行，或者是由第一终端102执行。本申请实施例对此不作限定。
在一些可能的实现方式中，当第一终端102的计算能力足够强大，例如是能够支持根据人体模型生成动作影像时，则可以不用借助云端计算能力生成动作影像，而是由第一终端102直接根据预先构建的第一用户的人体模型，在本地生成第一用户的动作影像，然后向第二终端104下发第一用户的动作影像。
参见图2所示的远程重现系统100的系统架构图，如图2所示，远程重现系统100包括第一终端102和第二终端104。其中，第一终端102可以根据预先构建的、第一用户的人体模型生成第一用户的动作影像，然后第一终端102可以通过有线通信(如光纤通信、同轴电缆通信)或无线通信(如第五代移动通信网络(5th generation mobile networks，5G)、无线热点(wireless fidelity，Wi-Fi))方式向第二终端104下发第一用户的动作影像。第二终端104可以向第二用户呈现第一用户的动作影像至第二用户所处的地理场景中。
在一些可能的实现方式中，第二终端104也可以根据预先构建的、第二用户的人体模型生成第二用户的动作影像，然后通过5G等无线通信方式向第一终端102发送第二用户的动作影像，第一终端102可以向第一用户呈现第二用户的动作影像至第一用户所处的地理场景中。在一些实施例中，第二终端104也可以生成第二用户所处的地理场景的影像，然后向第一终端102发送第二用户所处的地理场景的影像，第一终端102可以向第一用户呈现第二用户所处的地理场景的影像至第一用户所处的地理场景中。
本申请实施例提供的远程重现方法依赖于在准备阶段预先构建的人体模型。为了使得本申请的技术方案更加清楚、易于理解，本申请实施例还对准备阶段构建人体模型的过程进行详细说明。
参见图3所示的人体模型的构建方法的流程图，该方法包括：
S302：服务器获取用户的多视角图像。
具体地，对于第一用户或者第二用户，用户佩戴的终端(如第一终端102、第二终端104)或者用户所处的地理场景中可以部署传感器，如红外摄像头，服务器可以从传感器如红外摄像头，获取用户的不同视角图像，从而得到用户的多视角图像。
S304：服务器根据多视角图像进行重建，获得用户的人体模型。
具体地，服务器可以根据多视角图像进行三维重建，获得用户的三维人体模型。在一 些实施例中，多视角图像包括在一个平面内的视角的图像时，服务器可以根据多视角图像进行二维重建，获得用户的二维人体模型。
为了便于理解，下面以三维重建过程进行示例说明。
具体地，服务器可以根据多视角图像获得深度图，然后根据深度图重建得到三维网格(3D mesh)。该三维网格具体可以是三角形组成的多边形网络，该多边形网络用于模拟复杂物体的表面，例如用于模拟人体的表面。基于此，服务器可以将重建得到的三维网格作为人体模型。该三维网格的每个栅格都有可用于生成反照率、法线、光泽度和环境光遮挡贴图的渐变球形照明图像。这些贴图与标准渲染引擎兼容，可用于在任何一组照明条件下重新生成渲染图像。
在一些可能的实现中，服务器可以获取用户在不同动作，如不同表情、不同口型或者不同肢体动作时的多视角图像，根据该多视角图像分别建模，得到用户在不同表情、不同口型或者不同肢体动作时人体模型。
通过该方法构建的模型非常精细逼真。而且该模型还可以离线进行修补，比如进行美颜、换发型、换衣服等以用于提升用户满意度，或者对模型进行简化，以降低资源消耗。并且，针对每个用户，服务器可以在建模得到人体模型之后，保存该人体模型，如保存至云端，以便后续重复使用。
接下来，将结合具体实施例对本申请提供的远程重现方法进行介绍。
本申请实施例提供的远程重现方法可以通过端云协同实现，例如可以通过如图1所示的远程重现系统100实现。下面结合附图对该实现方式进行详细说明。
参见图4所示的远程重现方法的流程图，该方法由如图1所示的远程重现系统100执行，远程重现系统100包括第一终端102、第二终端104和中心服务器106、边缘服务器108，该方法包括：
S402：边缘服务器108获取传感器采集的、第一用户的关键部位图像。
关键部位是在关键点所在部位，如眼睛、嘴巴、眉毛等部位。关键点是指表征用户动作的点。其中，用户动作包括面部动作和/或肢体动作。相应地，关键点包括但不限于瞳孔、眉心、嘴角等面部关键点，或者手指、手肘、膝盖等身体关节点。
第一用户佩戴的第一终端102或者是第一用户所处的地理场景中可以部署传感器，例如是红外摄像头，该红外摄像头可以采集第一用户的关键部位图像，如此，边缘服务器108可以通过红外摄像头等传感器获取第一用户的关键部位图像，例如获取眼部图像、嘴部图像或者其他关键部位图像。
为了便于理解，以第一用户佩戴的第一终端102是VR终端进行示例说明。其中，VR终端可以是VR眼镜、VR头盔等。VR眼镜或VR头盔内可以部署红外摄像头，用于拍摄眼部图像。VR眼镜或VR头盔下也可以部署红外摄像头，用于拍摄嘴部图像。
S404：边缘服务器108根据第一用户的关键部位图像，获得第一用户的关键点的位置信息。
边缘服务器108可以利用关键点检测技术对第一用户的关键部位图像进行处理，获得第一用户的关键点的位置信息。具体地，边缘服务器108可以构建关键点检测模型，然后 将第一用户的关键部位图像输入关键点检测模型，通过关键点检测模型对图像提取关键点，获得第一用户的关键点的位置信息。
其中，关键点检测模型可以根据开源模型得到，例如可以对开源模型进行微调(fine tune)获得。在一些实施例中，关键点检测模型可以是传统的关键点检测模型。传统的关键点检测模型包括基于模板匹配的关键点检测模型，例如是图结构(Pictorial Structure)模型。关键点检测模型也可以是基于神经网络的关键点检测模型。基于神经网络的关键点检测模型包括但不限于谷歌研究和机器智能(Google's Research&Machine Intelligence，G-RMI)模型、级联特征网络(cascaded feature network，CFN)模型,区域多人位姿估计(regional multi-person pose estimation，RMPE)模型、基于掩膜的区域卷积神经网络(mask region convolutional neural network，mask R-CNN)模型和级联金字塔网络(cascaded pyramid network，CPN)模型等。
上述S402至S404为本申请实施例中获取第一用户的关键点的位置信息的一种实现方式，在本申请实施例其他可能的实现方式中，边缘服务器108也可以通过其他方式获取第一用户的关键点的位置信息。例如边缘服务器108可以直接从部署在第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
S406：边缘服务器108向中心服务器106发送第一用户的关键点的位置信息。
S408：中心服务器106根据第一用户的关键点的位置信息和第一用户的人体模型，生成第一用户的动作影像。
中心服务器106存储有第一用户的人体模型。第一用户的人体模型包括第一用户在不同动作，例如不同面部动作(不同表情、不同口型)、不同肢体动作时的人体模型。第一用户的人体模型绑定有第一用户的关键点。其中，第一用户的关键点可以是表征第一用户动作的点，例如表征第一用户面部动作的眉心、瞳孔、嘴角等关键点。
中心服务器106可以根据第一用户的关键点的位置信息，确定第一用户的关键点的移动轨迹。然后中心服务器106可以根据第一用户的关键点的移动轨迹以及存储在中心服务器106中的第一用户的人体模型，获得第一用户的动作影像。例如，中心服务器106可以根据第一用户的关键点的移动轨迹以及第一用户的人体模型进行渲染，生成第一用户的动作影像。
其中，关键点的移动轨迹可以表示为关键点的位置序列。第一用户的关键点与第一用户的人体模型绑定，中心服务器106可以根据关键点的位置序列确定每一个位置对应的人体模型，根据各个位置对应的人体模型可以形成连续的人体模型，进而生成第一用户的动作影像。第一用户的动作影像中包括用户的人体模型形成的至少一个动作的序列。
S410：中心服务器106向第二终端104发送第一用户的动作影像。
S412：第二终端104向第二用户重现第一用户的动作影像至第二用户所处的地理场景中。
具体地，第二终端104可以通过该第二终端104的显示器播放动作影像，从而向第二用户重现第一用户的动作影像至第二用户所处的地理场景中。在线学习场景中，上述第二用户所处的地理场景可以是第二用户的居所或学习场所，包括但不限于住宅、宿舍、酒店、教室、图书馆、自习室等。
其中，第二终端104可以是平面终端，例如可以是电视等大屏终端或者是智能手机、平板电脑等便携终端。动作影像可以是二维动作影像，上述平面终端可以播放上述二维动作影像，从而向第二用户重现第一用户的二维影像至第二用户所处的地理场景中。
在一些实施例中，第二终端104也可以是立体终端，如虚拟现实(virtual reality，VR)终端或者是AR终端。其中，VR终端可以包括VR眼镜、VR头显等设备，AR终端可以包括AR眼镜、AR头显。动作影像可以是三维动作影像，上述VR终端、AR终端可以播放三维动作影像，从而向第二用户重现第一用户的三维影像至第二用户所处的地理场景中。
其中，二维动作影像的渲染效率较高，第二用户可以及时查看第一用户的动作影像，降低了延迟。三维动作影像可以生动地呈现第一用户的形象，为第二用户提供较好的沉浸式体验。
需要说明的是，中心服务器106在渲染第一用户的动作影像时，还可以将第一用户的人体模型转换到第二终端104对应的坐标系进行渲染，如此可以使得多个第二用户通过各自的第二终端104观察到的第一用户的形象在同一位置，由此可以进一步提升沉浸感。
S414：边缘服务器108获取传感器采集的、第二用户的关键部位图像。
S416：边缘服务器108根据第二用户的关键部位图像，获得第二用户的关键点的位置信息。
其中，边缘服务器108获取第二用户的关键部位图像，以及根据第二用户的关键部位图像，获得第二用户的关键点的位置信息，与边缘服务器108获取第一用户的关键部位图像，以及根据第一用户的关键部位图像，获得第一用户的关键点的位置信息类似，在此不再赘述。
在一些实施例中，边缘服务器108也可以直接从部署在第二用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
S418：边缘服务器108向中心服务器106发送第二用户的关键点的位置信息。
S420：中心服务器106根据第二用户的关键点的位置信息和第二用户的人体模型，生成第二用户的动作影像。
中心服务器106可以根据第二用户的关键点的位置信息，确定第二用户的关键点的移动轨迹，然后根据第二用户的关键点的移动轨迹以及存储在中心服务器106中的第二用户的人体模型，获得第二用户的动作影像。
中心服务器106生成第二用户的动作影像的具体实现可以参见S408中心服务器106生成第一用户的动作影像的相关内容描述，在此不再赘述。
S422：中心服务器106向第一终端102发送第二用户的动作影像。
S424：第一终端102向第一用户呈现第二用户的动作影像至第一用户所处的地理场景中。
具体地，第一终端102可以通过该第一终端102的显示器播放第二用户的动作影像，从而向第一用户重现第二用户的动作影像至第一用户所处的地理场景中。在线学习场景中，第一用户所处的地理场景可以为第一用户的居所、教学场所，如住宅、宿舍、酒店、教室、办公室等。
与第二终端104类似，第一终端102可以是平面终端，例如可以是电视等大屏终端或 者是智能手机、平板电脑等便携终端。动作影像可以是二维动作影像，上述平面终端可以播放上述二维动作影像，从而向第一用户重现第二用户的二维影像至第一用户所处的地理场景中。
在一些实施例中，第一终端102也可以是立体终端，如VR终端或者是AR终端。其中，VR终端可以包括VR眼镜、VR头显等设备，AR终端可以包括AR眼镜、AR头显。动作影像可以是三维动作影像，上述VR终端、AR终端可以播放三维动作影像，从而向第一用户重现第二用户的三维影像至第一用户所处的地理场景中。
图4所示实施例是以在线学习场景进行示例说明，在一些可能的实现方式中，例如在远程协助场景中，也可以不执行上述S414至S424。例如中心服务器106还可以获取第二用户所处地理场景的影像，向第一终端102发送第二用户所处的地理场景的影像，第一终端102向第一用户重现第二用户的动作影像至第一用户所处的地理场景中。
进一步地，中心服务器106也可以获取第二用户所处的地理场景的影像以及第二用户的动作影像，向第一终端102发送第二用户所处的地理场景的影像以及第二用户的动作影像，如此，第一终端102向第一用户呈现第二用户所处的地理场景的影像以及第二用户的动作影像至第一用户所处的地理场景中。
基于上述内容描述，本申请实施例提供了一种远程重现方法。该方法通过将远程重现过程分为两个阶段，具体是远程重现开始前的准备阶段和远程重现阶段，在准备阶段预先构建第一用户的人体模型，在远程重现阶段利用该人体模型生成动作影像，避免了远程重现阶段花费大量时间重建人体模型，满足了远程重现应用对实时性的需求。而且在准备阶段可以花费较多的时间进行重建，得到逼真的人体模型，具有较好的效果，能够满足业务需求。另外，该方法从关键部位图像提取关键点，获得关键点的位置信息，向云端发送关键点的位置信息，而不再需要发送大量图像，既节省了网络资源，又节省了计算资源，还避免了大规模数据传输在网络质量不好的时候可能带来的丢帧、卡顿等严重影响主观质量的问题。该方法还允许第一用户、第二用户大范围地移动，对用户移动范围的限制较小，提升了用户体验。
图4所示实施例主要是以云端服务器如中心服务器106生成动作影像进行示例说明，在一些可能的实现方式中，生成动作影像的过程也可以由终端(例如是第一终端102、第二终端104)实现。需要说明的是，终端生成动作影像的前提是终端预先构建有对应用户的人体模型，或者预先从其他设备如服务器获取用户的人体模型。终端构建人体模型的具体实现与图3所示实施例中服务器构建人体模型的实现过程类似，在此不再赘述。下面结合实施例对由终端生成动作影像的远程重现方法进行说明。
参见图5所示的远程重现方法的流程图，该方法由如图2所示的远程重现系统100执行，远程重现系统100包括第一终端102和第二终端104，该方法包括：
S502：第一终端102获取第一用户的关键点的位置信息。
关键点是指表征用户动作的点。其中，用户动作包括面部动作和/或肢体动作。相应地，关键点包括但不限于瞳孔、眉心、嘴角等面部关键点，或者手指、手肘、膝盖等身体关节点。
在一些可能的实现方式中，第一用户的身体、第一终端102或者是第一用户所处的地理场景中可以部署传感器，用于捕捉第一用户或者是第一用户所处的地理场景的信息。其中，传感器包括相机(如红外相机、可见光相机)和位姿传感器(如惯性测量单元(inertial measurement unit，IMU))中的至少一种。进一步地，传感器还可以包括电容传感器，该电容传感器可以用于眼球追踪。
第一终端102可以通过部署在第一用户的身体、第一终端102或者是第一用户所在环境(所处的地理场景)中的相机，获取第一用户的关键部位图像。其中，关键部位是在关键点所在部位，如眼睛、嘴巴、眉毛等部位，然后第一终端102根据所述第一用户的关键部位图像获得对应的关键点的位置信息。具体地，第一终端102可以利用关键点检测模型对关键部位图像进行检测，从而获得关键点的位置信息。
第一终端102也可以直接通过传感器捕捉第一用户的关键点的位置信息。具体地，第一终端102可以从部署在关键部位的位姿传感器如IMU获取第一用户的关键点的位姿信息，从而获得第一用户的关键点的位置信息。
S504：第一终端102根据所述第一用户的关键点的位置信息和第一用户的人体模型生成第一用户的动作影像。
人体模型是指用于描述人体的模型。针对同一用户，该用户执行不同动作，如执行不同肢体动作，或者不同面部动作(例如不同口型、不同表情)时，人体模型可以是不同的。第一终端102可以获取第一用户在不同动作(如不同口型、不同表情、不同肢体动作)时的人体模型。
在一些可能的实现方式中，第一终端102还可以对第一用户的人体模型进行优化，例如可以支持对人体模型进行美颜、换发、换装等，使得人体模型在保留第一用户的特征的情况下，还能够满足个性化的需求，由此可以提高用户体验。
考虑到渲染速度和渲染效率，第一终端102还可以对人体模型进行简化。具体地，第一终端102可以通过对人体模型进行稀疏处理，从而实现对人体模型的简化，如此可以减少渲染计算量，提高渲染速度和渲染效率。
第一终端102可以根据第一用户的关键点的位置信息，确定第一用户的关键点的移动轨迹。第一用户的关键点与第一用户的人体模型绑定，第一终端102可以根据该第一用户的关键点的移动轨迹，驱动第一用户的人体模型，例如是根据该移动轨迹以及对应的人体模型进行渲染，生成第一用户的动作影像。
其中，关键点的移动轨迹可以表示为关键点的位置序列。第一用户的关键点与第一用户的人体模型绑定，第一终端102可以根据关键点的位置序列确定每一个位置对应的人体模型，根据各个位置对应的人体模型可以形成连续的人体模型，进而生成第一用户的动作影像。第一用户的动作影像中包括用户的人体模型形成的至少一个动作的序列。
S506：第一终端102向第二终端104发送第一用户的动作影像。
S508：第二终端104向第二用户呈现第一用户的动作影像至第二用户所处的地理场景中。
具体地，第二终端104可以通过该第二终端104的显示器播放动作影像，从而向第二用户重现第一用户的动作影像至第二用户所处的地理场景中。第二终端104可以是平面终 端，例如为电视、白板设备、台式机、笔记本电脑或者是智能手机等等，对应地，第一用户的动作影像可以是二维动作影像。第二终端104也可以是立体终端，例如为VR终端、AR终端，包括VR眼镜、VR头盔、AR眼镜、AR头盔，对应地，第一用户的动作影像可以是三维动作影像。
其中，二维动作影像的渲染效率较高，第二用户可以及时查看第一用户的动作影像，降低了延迟。三维动作影像可以生动地重现第一用户的形象，为第二用户提供较好的沉浸式体验。
需要说明的是，第一终端102在渲染第一用户的动作影像时，还可以将第一用户的人体模型转换到第二终端104对应的坐标系进行渲染，如此可以使得多个第二用户通过各自的第二终端104观察到的第一用户的影像在同一位置，由此可以进一步提升沉浸感。
基于上述内容描述，本申请实施例提供了一种远程重现方法。该方法通过利用预先构建的、第一用户的人体模型生成动作影像，由此避免了远程重现时花费大量时间构建人体模型，满足了远程重现应用对实时性的需求。而且在准备阶段可以花费较多的时间进行重建，得到逼真的人体模型，具有较好的效果，能够满足业务需求。此外，远程重现阶段无需发送大量图像到云端实时重建人体模型，避免了大规模数据传输占用大量网络资源，以及网络质量不好的情况下出现丢帧、卡顿等问题，提升了交互体验。
图5所示实施例主要描述了第二终端104播放第一用户的动作影像，从而向第二用户重现第一用户的动作影像。在一些可能的实现方式中，第一终端102还可以播放第二用户相关的影像，从而向第一用户呈现第二用户相关的影像。
其中，第二用户相关的影像可以是第二用户所处的地理场景的影像。例如，在远程协助场景中，第一终端102还可以执行如下步骤：
S510：第一终端102接收第二终端104发送的、第二用户所处的地理场景的影像。
具体地，第二用户可以是故障现场的操作员。其中，故障现场是指故障发生的现场，例如可以是工厂或实验室中故障设备所在现场。第二用户所处的地理场景的影像可以是故障现场的影像，如呈现故障设备的影像。
其中，第一终端102可以是VR终端，例如是VR眼镜。第二终端可以是AR终端，例如是AR眼镜。如此，第一用户的影像可以与故障现场的影像叠加，然后投影到现场操作员的AR终端。如此，现场的操作员可以通过AR终端观测到第一用户，该第一用户的影像可以不随操作员移动，从第二用户的视角来看，第一用户可以四处走动、指点、说话等，如同亲临现场，提升了交互体验。
S512：第一终端102向第一用户呈现第二用户所处的地理场景的影像至第一用户所处的地理场景中。
具体地，第一用户可以是具有故障诊断或故障维修能力的专家，第一终端102可以播放第二用户所处的地理场景的影像，从而向第一用户重现第二用户所处的故障现场的影像至第一用户所处的地理场景中。
上述S510至S512可以是在S508之后执行，基于此，第二终端104发送的第二用户所处的地理场景的影像还可以是第一用户视角的故障现场影像。具体地，第二终端104可以 获取第一用户的瞳孔位置，该瞳孔位置可以通过眼球追踪获得，然后第二终端104可以根据第一用户的瞳孔位置确定第一用户视角，第二终端104可以通过现场拍摄、视角合成(view synthesis)或者是环境模拟等方式获得第一用户视角的故障现场影像。
其中，现场拍摄是指通过机械臂带动一个双目摄像头，该机械臂由第一用户的瞳孔位置进行驱动，双目摄像头在机械臂带动下对故障现场进行拍摄，第二终端104可以从该双目摄像头获取第一用户视角的故障现场影像。
视角合成是指预先在第二用户所处的地理场景(如故障现场)中部署固定的若干摄像头，使用和第一用户视角接近的多个视角的摄像头拍摄到的视频，进行视频合成，获得第一用户视角的故障现场影像。
环境模拟是指预先对故障现场的环境进行建模得到环境模型，然后以第一用户视角(如专家视角)建立坐标系，对环境模型进行渲染，由此获得第一用户视角的故障现场影像。
上述S510至S512也可以在S502之前执行。例如，第二终端104可以先发送第二用户如操作员所处的地理场景的影像，如故障现场影像，第一终端102播放该故障现场影像，向第一用户如专家重现该故障现场影像至第一用户所处的地理场景中。
如此，第一用户可以根据该故障现场作出相应的动作，例如眼睛向左侧看，第一终端102可以根据眼睛部位图像或者是眼球追踪结果，确定包括瞳孔位置在内的关键点的位置信息，根据关键点的位置信息以及人体模型生成第一用户的动作影像。第一终端102向第二终端104发送该动作影像。对应地，第二终端104可以根据动作影像确定第一用户视角，获取第一用户视角的故障现场影像，返回至第一终端102。第一用户根据故障现场影像排查出故障原因后，还可以通过动作和语音等指示故障维修方式，第一终端102可以生成指示有故障维修方式的动作影像，发送至第二终端104。
在一些实施例中，第二用户相关的影像可以是第二用户的动作影像。例如，在线学习场景中，第一终端102还可以执行如下步骤：
S514：第一终端102接收第二终端104发送的、第二用户的动作影像。
在线学习场景中，第一用户可以是具有专业能力的教练、教师，第二用户可以是学习专业技能或专业知识的学员、学生。第一用户更关注第二用户自身，而不是第二用户所处的地理场景。基于此，第二终端104可以获取第二用户的动作影像，并向第一终端102发送第二用户的动作影像。
为了获得更加自然的交互体验，第一终端102和第二终端104可以均为VR终端，例如均为VR眼镜。当然，在本申请实施例其他可能的实现方式中，第一终端102和第二终端104可以是其他类型的终端。
S516：第一终端102向第一用户呈现第二用户的动作影像。
具体地，第一终端102通过播放第二用户的动作影像，从而向第一用户呈现第二用户的动作影像至第一用户所处的地理场景中。第一用户如教练、老师可以根据第二用户如学员、学生的动作影像判断学员和学生对专业技能、专业知识的掌握情况，并根据该情况进行针对性纠正，由此提高在线学习的效率，以及提高在线学习的用户体验。
与上述S510至S512类似，S514、S516可以在S508之后执行，或者是在S502之前执 行，本申请实施例对此不作限定。
上文结合图1至图5对本申请实施例提供的远程重现方法进行了详细介绍，下面将结合附图对本申请实施例提供的系统、装置、设备进行介绍。
参见图1或图2所示的远程重现系统100的结构示意图，该系统100包括第一终端102和第二终端104，所述第一终端102由第一用户佩戴，所述第二终端104由第二用户佩戴，所述第一用户与所述第二用户处于不同地理场景：
所述第二终端104，用于获取所述第一用户的动作影像，向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述远程重现系统100还包括服务器，例如包括中心服务器106和边缘服务器108；
所述边缘服务器108，用于获取所述第一用户的关键点的位置信息；
所述中心服务器106，用于根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹，根据所述移动轨迹以及存储在所述服务器中的第一用户的人体模型，获得所述第一用户的动作影像。
在一些可能的实现方式中，所述服务器(例如边缘服务器108)具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述服务器(例如边缘服务器108)具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述第一终端102用于：
获取所述第二用户所处的地理场景的影像；
向所述第一用户重现所述第二用户所处的地理场景的影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述第一终端为虚拟现实终端，所述第二终端为增强现实终端。
在一些可能的实现方式中，所述第一终端102用于：
获取所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
向所述第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述第一终端和所述第二终端为虚拟现实终端。
在一些可能的实现方式中，所述第一终端102用于：
获取所述第一用户的关键点的位置信息；
根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
根据所述移动轨迹以及存储在第一终端102中的第一用户的人体模型，获得所述第一 用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
根据本申请实施例的远程重现系统100可对应于执行本申请实施例中描述的方法，并且远程重现系统100的各个组成部分的上述和其它操作和/或功能分别为了实现图4、图5所示实施例中的各个方法的相应流程，为了简洁，在此不再赘述。
基于本申请实施例提供的远程重现方法，本申请实施例提供了对应的远程重现装置。该远程重现装置可以是软件装置，该远程重现装置部署在第二终端104中，所述第二终端104由第二用户佩戴，所述第二用户与第一用户处于不同地理场景。
参见图6所示的远程重现装置的结构示意图，该装置600包括：
影像获取单元602，用于获取所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
影像重现单元604，用于向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中。
在一些可能的实现方式中，所述装置600还包括：
影像发送单元606，用于向第一终端发送第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述装置600还包括：
影像生成单元608，用于获取所述第二用户的关键点的位置信息，根据所述第二用户的关键点的位置信息，确定所述第二用户的关键点的移动轨迹，根据所述移动轨迹以及存储在第二终端104中的第二用户的人体模型，获得所述第二用户的动作影像。
在一些可能的实现方式中，所述影像生成单元608具体用于：
获取所述第二用户的关键部位图像，根据所述第二用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述影像生成单元608具体用于：
从部署在所述第二用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述装置600还包括：
影像发送单元606，用于向第一终端发送第二用户所处的地理场景的影像。
在一些可能的实现方式中，所述影像获取单元602具体用于：
从服务器或者第一终端获取所述第一用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
根据本申请实施例的远程重现装置600可对应于执行本申请实施例中描述的方法，并且远程重现装置600的各个模块/单元的上述和其它操作和/或功能分别为了实现图4、图5 所示实施例中的各个方法的相应流程，为了简洁，在此不再赘述。
基于本申请实施例提供的远程重现方法，本申请实施例提供了另一种远程重现装置。该远程重现装置可以是软件装置，该远程重现装置部署在服务器中。
参见图7所示的远程重现装置的结构示意图，该装置700包括：
位置获取单元702，用于获取第一用户的关键点的位置信息；
轨迹确定单元704，用于根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
影像生成单元706，用于根据所述移动轨迹以及存储在服务器(例如是中心服务器106)中的第一用户的人体模型，获得所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
影像发送单元708，用于向第二终端发送所述第一用户的动作影像，以使所述第二终端向第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户与所述第二用户处于不同地理场景。
在一些可能的实现方式中，所述位置获取单元702具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述位置获取单元702具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述位置获取单元702还用于：
获取第二用户的关键点的位置信息；
所述轨迹确定单元704还用于：
根据所述第二用户的关键点的位置信息，确定所述第二用户的关键点的移动轨迹；
所述影像生成单元706还用于：
根据所述移动轨迹以及存储在服务器中的第二用户的人体模型，获得所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列定，所述第二用户的人体模型为预先构建的模型；
所述影像发送单元708还用于：
向第一终端发送所述第二用户的动作影像，以使所述第一终端向第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述影像生成单元706还用于：
生成第二用户所处的地理场景的影像；
所述影像发送单元708还用于：
向第一终端发送所述第二用户所处的地理场景的影像。
根据本申请实施例的远程重现装置700可对应于执行本申请实施例中描述的方法，并且远程重现装置700的各个模块/单元的上述和其它操作和/或功能分别为了实现图4、图5 所示实施例中的各个方法的相应流程，为了简洁，在此不再赘述。
基于本申请实施例提供的远程重现方法，本申请实施例提供了又一种远程重现装置。该远程重现装置可以是软件装置，该远程重现装置部署在第一终端102中。第一终端102由第一用户佩戴，所述第一用户与第二用户处于不同地理场景。
参见图8所示的远程重现装置的结构示意图，该装置800包括：
影像获取单元802，用于获取所述第二用户所处的地理场景的影像和/或所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
影像重现单元804，用于向所述第一用户重现所述第二用户所处的地理场景的影像和/或所述第二用户的动作影像至所述第一用户所处的地理场景中。
在一些可能的实现方式中，所述装置800还包括：
影像发送单元806，用于向第二终端发送第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型。
在一些可能的实现方式中，所述装置800还包括：
影像生成单元808，用于获取所述第一用户的关键点的位置信息，根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹，根据所述移动轨迹以及存储在第一终端中的第一用户的人体模型，获得所述第一用户的动作影像。
在一些可能的实现方式中，所述影像生成单元808具体用于：
获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
在一些可能的实现方式中，所述影像生成单元808具体用于：
从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
在一些可能的实现方式中，所述装置800还包括：
影像发送单元806，用于向第二终端发送第一用户所处的地理场景的影像。
在一些可能的实现方式中，所述影像获取单元802具体用于：
从服务器或者第二终端104获取所述第二用户的动作影像。
在一些可能的实现方式中，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
根据本申请实施例的远程重现装置800可对应于执行本申请实施例中描述的方法，并且远程重现装置800的各个模块/单元的上述和其它操作和/或功能分别为了实现图4、图5所示实施例中的各个方法的相应流程，为了简洁，在此不再赘述。
本申请实施例还提供了一种第二终端104。该第二终端104具体用于实现如上述图6所示实施例中远程重现装置600的功能。
图9提供了一种第二终端104的结构示意图，如图9所示，第二终端104包括总线1041、 处理器1042、显示器1043和存储器1044。处理器1042、存储器1044和显示器1043之间通过总线1041通信。
总线1041可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图9中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
处理器1042可以为中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
显示器1043是一种输入输出(input/output，I/O)设备。根据制造材料不同，显示器1043可以分为液晶显示器(liquid crystal display，LCD)、有机电激光(organic light emitting diode，OLED)显示器等。具体地，显示器1043可以向第二用户重现第一用户的动作影像至第二用户所处的地理场景中。
存储器1044可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。存储器1044还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，硬盘驱动器(hard disk drive，HDD)或固态驱动器(solid state drive，SSD)。
存储器1044中存储有可执行代码，处理器1042执行该可执行代码以执行前述远程重现方法。具体地，在实现图6所示实施例的情况下，且图6实施例中所描述的远程重现装置600的各模块或单元为通过软件实现的情况下，执行图6中的影像获取单元602、影像重现单元604、影像发送单元606以及影像生成单元608功能所需的软件或程序代码可以部分或全部存储在存储器1044中。处理器1042执行存储器1044中存储的各单元对应的程序代码，执行远程重现方法。
本申请实施例还提供了一种服务器。该服务器具体用于实现如上述图7所示实施例中远程重现装置700的功能。该服务器可以中心服务器106，也可以是边缘服务器108，本申请实施例以中心服务器106进行示例说明。
图10提供了一种中心服务器106的结构示意图，如图10所示，中心服务器106包括总线1061、处理器1062、通信接口1063和存储器1064。处理器1062、存储器1064和通信接口1063之间通过总线1061通信。
其中，总线1061、处理器1062和存储器1064的具体实现可以参见图9所示实施例相关内容描述。通信接口1063主要用于和外部通信，例如用于从位姿传感器或边缘服务器108获取第一用户的关键点的位置信息，向第二终端104发送所述第一用户的动作影像，又例如从位姿传感器或边缘服务器108获取第二用户的关键点的位置信息，向第一终端102发送所述第一用户的动作影像。
具体地，在实现图7所示实施例的情况下，且图7实施例中所描述的远程重现装置700的各模块或单元为通过软件实现的情况下，执行图7中的位置获取单元702、轨迹确定单元704、影像生成单元706以及影像发送单元708功能所需的软件或程序代码可以部分或 全部存储在存储器1064中。处理器1062执行存储器1064中存储的各单元对应的程序代码，执行远程重现方法。
本申请实施例还提供了一种第一终端102。该第一终端102具体用于实现如上述图8所示实施例中远程重现装置800的功能。
图11提供了一种第一终端102的结构示意图，如图11所示，第一终端102包括总线1021、处理器1022、显示器1023和存储器1024。处理器1022、存储器1024和显示器1023之间通过总线1021通信。
其中，总线1021、处理器1022、显示器1023和存储器1024的具体实现可以参见图9所示实施例相关内容描述。具体地，在实现图8所示实施例的情况下，且图8实施例中所描述的远程重现装置800的各模块或单元为通过软件实现的情况下，执行图8中的影像获取单元802、影像重现单元804、影像发送单元806以及影像生成单元808功能所需的软件或程序代码可以部分或全部存储在存储器1024中。处理器1022执行存储器1024中存储的各单元对应的程序代码，执行远程重现方法。
本申请实施例还提供了一种计算机可读存储介质。所述计算机可读存储介质可以是计算设备能够存储的任何可用介质或者是包含一个或多个可用介质的数据中心等数据存储设备。所述可用介质可以是磁性介质(例如软盘、硬盘、磁带)、光介质(例如DVD)、或者半导体介质(例如固态硬盘)等。该计算机可读存储介质包括指令，所述指令指示计算设备执行上述应用于远程重现装置600、远程重现装置700或者是远程重现装置800的远程重现方法。
本申请实施例还提供了一种计算机程序产品，所述计算机程序产品包括一个或多个计算机指令。在计算设备上加载和执行所述计算机指令时，全部或部分地产生按照本申请实施例所述的流程或功能。
所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机或数据中心进行传输。
所述计算机程序产品被计算机执行时，所述计算机执行前述远程重现方法的任一方法。该计算机程序产品可以为一个软件安装包，在需要使用前述远程重现方法的任一方法的情况下，可以下载该计算机程序产品并在计算机上执行该计算机程序产品。
上述各个附图对应的流程或结构的描述各有侧重，某个流程或结构中没有详述的部分，可以参见其他流程或结构的相关描述。

Claims (24)

  1. 一种远程重现方法，其特征在于，应用于远程重现系统，所述远程重现系统包括第一终端和第二终端，所述第一终端由第一用户佩戴，所述第二终端由第二用户佩戴，所述第一用户与所述第二用户处于不同地理场景，所述方法包括：
    所述第二终端获取所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
    所述第二终端向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中。
  2. 根据权利要求1所述的方法，其特征在于，所述远程重现系统还包括服务器，所述方法还包括：
    所述服务器获取所述第一用户的关键点的位置信息；
    所述服务器根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
    所述服务器根据所述移动轨迹和存储在所述服务器中的所述第一用户的人体模型，获得所述第一用户的动作影像；
    所述服务器发送所述第一用户的动作影像至所述第二终端。
  3. 根据权利要求2所述的方法，其特征在于，所述服务器获取所述第一用户的关键点的位置信息，包括：
    所述服务器获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
  4. 根据权利要求2所述的方法，其特征在于，所述服务器获取所述第一用户的关键点的位置信息，包括：
    所述服务器从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
  5. 根据权利要求1至4任一项所述的方法，其特征在于，所述方法还包括：
    所述第一终端获取所述第二用户所处的地理场景的影像；
    所述第一终端向所述第一用户重现所述第二用户所处的地理场景的影像至所述第一用户所处的地理场景中。
  6. 根据权利要求5所述的方法，其特征在于，所述第一终端为虚拟现实终端，所述第二终端为增强现实终端。
  7. 根据权利要求1至4任一项所述的方法，其特征在于，所述方法还包括：
    所述第一终端获取所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
    所述第一终端向所述第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
  8. 根据权利要求7所述的方法，其特征在于，所述第一终端和所述第二终端为虚拟现实终端。
  9. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    所述第一终端获取所述第一用户的关键点的位置信息；
    所述第一终端根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
    所述第一终端根据所述移动轨迹和存储在所述第一终端中的所述第一用户的人体模型，获得所述第一用户的动作影像。
  10. 根据权利要求1至9任一项所述的方法，其特征在于，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
  11. 一种远程重现系统，其特征在于，所述系统包括第一终端和第二终端，所述第一终端由第一用户佩戴，所述第二终端由第二用户佩戴，所述第一用户与所述第二用户处于不同地理场景；
    所述第二终端，用于获取所述第一用户的动作影像，向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型。
  12. 根据权利要求11所述的系统，其特征在于，所述远程重现系统还包括服务器；
    所述服务器，用于获取所述第一用户的关键点的位置信息，根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹，根据所述移动轨迹和存储在所述服务器中的所述第一用户的人体模型，获得所述第一用户的动作影像，发送所述第一用户的动作影像至所述第二终端。
  13. 根据权利要求12所述的系统，其特征在于，所述服务器具体用于：
    获取所述第一用户的关键部位图像，根据所述第一用户的关键部位图像获得对应的关键点的位置信息。
  14. 根据权利要求12所述的系统，其特征在于，所述服务器具体用于：
    从部署在所述第一用户的关键部位的位姿传感器获取与所述关键部位对应的关键点的位置信息。
  15. 根据权利要求11至14任一项所述的系统，其特征在于，所述第一终端用于：
    获取所述第二用户所处的地理场景的影像；
    向所述第一用户重现所述第二用户所处的地理场景的影像至所述第一用户所处的地理场景中。
  16. 根据权利要求15所述的系统，其特征在于，所述第一终端为虚拟现实终端，所述第二终端为增强现实终端。
  17. 根据权利要求11至14任一项所述的系统，其特征在于，所述第一终端用于：
    获取所述第二用户的动作影像，所述第二用户的动作影像包括所述第二用户的人体模型形成的至少一个动作的序列，所述第二用户的人体模型为预先构建的模型；
    向所述第一用户重现所述第二用户的动作影像至所述第一用户所处的地理场景中。
  18. 根据权利要求17所述的系统，其特征在于，所述第一终端和所述第二终端为虚拟现实终端。
  19. 根据权利要求11所述的系统，其特征在于，所述第一终端用于：
    获取所述第一用户的关键点的位置信息；
    根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
    根据所述移动轨迹和存储在所述第一终端中的所述第一用户的人体模型，获得所述第一用户的动作影像。
  20. 根据权利要求11至19任一项所述的系统，其特征在于，所述人体模型为三维人体模型，所述动作影像为三维动作影像。
  21. 一种远程重现装置，其特征在于，所述远程重现装置部署在第二终端中，所述第二终端由第二用户佩戴，所述第二用户与第一用户处于不同地理场景，所述装置包括：
    影像获取单元，用于获取所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
    影像重现单元，用于向所述第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中。
  22. 一种远程重现装置，其特征在于，所述远程重现装置部署在服务器中，所述装置包括：
    位置获取单元，用于获取第一用户的关键点的位置信息；
    轨迹确定单元，用于根据所述第一用户的关键点的位置信息，确定所述第一用户的关键点的移动轨迹；
    影像生成单元，用于根据所述移动轨迹和所述第一用户的人体模型，获得所述第一用户的动作影像，所述第一用户的动作影像包括所述第一用户的人体模型形成的至少一个动作的序列，所述第一用户的人体模型为预先构建的模型；
    影像发送单元，用于向第二终端发送所述第一用户的动作影像，以使所述第二终端向第二用户重现所述第一用户的动作影像至所述第二用户所处的地理场景中，所述第一用户与所述第二用户处于不同地理场景。
  23. 一种终端，其特征在于，所述终端包括处理器和存储器；
    所述处理器用于执行所述存储器中存储的指令，以使得所述终端实现如权利要求21中所述的远程重现装置的功能。
  24. 一种服务器，其特征在于，所述服务器包括处理器和存储器；
    所述处理器用于执行所述存储器中存储的指令，以使得所述服务器实现如权利要求22中所述的远程重现装置的功能。
PCT/CN2021/130532 2021-01-26 2021-11-15 远程重现方法、系统、装置、设备、介质及程序产品 WO2022160867A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21922457.3A EP4280226A4 (en) 2021-01-26 2021-11-15 REMOTE PLAYBACK METHOD, SYSTEM AND APPARATUS, EQUIPMENT, MEDIUM AND PROGRAM PRODUCT
US18/358,228 US20230368467A1 (en) 2021-01-26 2023-07-25 Remote reproduction method, system, and apparatus, device, medium, and program product

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110105037.0A CN114898132A (zh) 2021-01-26 2021-01-26 远程重现方法、系统、装置、设备、介质及程序产品
CN202110105037.0 2021-01-26

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/358,228 Continuation US20230368467A1 (en) 2021-01-26 2023-07-25 Remote reproduction method, system, and apparatus, device, medium, and program product

Publications (1)

Publication Number Publication Date
WO2022160867A1 true WO2022160867A1 (zh) 2022-08-04

Family

ID=82654152

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/130532 WO2022160867A1 (zh) 2021-01-26 2021-11-15 远程重现方法、系统、装置、设备、介质及程序产品

Country Status (4)

Country Link
US (1) US20230368467A1 (zh)
EP (1) EP4280226A4 (zh)
CN (1) CN114898132A (zh)
WO (1) WO2022160867A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150091891A1 (en) * 2013-09-30 2015-04-02 Dumedia, Inc. System and method for non-holographic teleportation
CN110600141A (zh) * 2019-09-11 2019-12-20 孙皓 一种基于全息影像技术的融合仿生机器人远程关怀系统
CN111834021A (zh) * 2020-07-20 2020-10-27 北京百度网讯科技有限公司 数据交互方法、装置、设备以及存储介质
CN112033432A (zh) * 2020-07-31 2020-12-04 东莞市易联交互信息科技有限责任公司 动作捕捉方法、系统和存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150091891A1 (en) * 2013-09-30 2015-04-02 Dumedia, Inc. System and method for non-holographic teleportation
CN110600141A (zh) * 2019-09-11 2019-12-20 孙皓 一种基于全息影像技术的融合仿生机器人远程关怀系统
CN111834021A (zh) * 2020-07-20 2020-10-27 北京百度网讯科技有限公司 数据交互方法、装置、设备以及存储介质
CN112033432A (zh) * 2020-07-31 2020-12-04 东莞市易联交互信息科技有限责任公司 动作捕捉方法、系统和存储介质

Also Published As

Publication number Publication date
EP4280226A4 (en) 2024-06-05
US20230368467A1 (en) 2023-11-16
EP4280226A1 (en) 2023-11-22
CN114898132A (zh) 2022-08-12

Similar Documents

Publication Publication Date Title
WO2023045144A1 (zh) 一种立体综合教学场系统的工作方法
Orlosky et al. Virtual and augmented reality on the 5G highway
US11410570B1 (en) Comprehensive three-dimensional teaching field system and method for operating same
TWI752502B (zh) 一種分鏡效果的實現方法、電子設備及電腦可讀儲存介質
Liang RETRACTED ARTICLE: Scene art design based on human-computer interaction and multimedia information system: an interactive perspective
US20130218542A1 (en) Method and system for driving simulated virtual environments with real data
Kurillo et al. 3D teleimmersion for collaboration and interaction of geographically distributed users
CN109035932A (zh) 一种vr全息教学系统
CN107835436A (zh) 一种基于WebGL的实时虚实融合直播系统及方法
CN109887096A (zh) 利用虚拟现实技术的教育教学信息处理系统及其教学方法
Pereira et al. Using 360-degree interactive panoramas to develop virtual representation of construction sites
CN109035415B (zh) 虚拟模型的处理方法、装置、设备和计算机可读存储介质
US11514638B2 (en) 3D asset generation from 2D images
CN109447020A (zh) 基于全景肢体动作的交互方法及系统
Zikas et al. MAGES 4.0: Accelerating the world’s transition to VR training and democratizing the authoring of the medical metaverse
US11537351B2 (en) Systems and methods for virtual and augmented reality
Song et al. An immersive VR system for sports education
Hackett et al. Volumetric video and mixed reality for healthcare training
Kamarianakis et al. Less is more: Efficient networked VR transformation handling using geometric algebra
WO2022160867A1 (zh) 远程重现方法、系统、装置、设备、介质及程序产品
Dermawan et al. Moving Mixed Reality Game to the Cloud: A Survey on Feasibility
Leite et al. Inter-Acting: Understanding interaction with performance-driven puppets using low-cost optical motion capture device
Qian et al. Overlap Makes Perfect: Designing a Meta-Tutor With Live Corrective Feedback for Ubiquitous Motion Learning
Gomide Motion capture and performance
Lala et al. Enhancing communication through distributed mixed reality

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21922457

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021922457

Country of ref document: EP

Effective date: 20230814

NENP Non-entry into the national phase

Ref country code: DE