WO2020078250A1 - 用于虚拟场景的数据处理方法以及设备 - Google Patents

用于虚拟场景的数据处理方法以及设备 Download PDF

Info

Publication number
WO2020078250A1
WO2020078250A1 PCT/CN2019/110330 CN2019110330W WO2020078250A1 WO 2020078250 A1 WO2020078250 A1 WO 2020078250A1 CN 2019110330 W CN2019110330 W CN 2019110330W WO 2020078250 A1 WO2020078250 A1 WO 2020078250A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal
data
image
coordinate system
camera coordinate
Prior art date
Application number
PCT/CN2019/110330
Other languages
English (en)
French (fr)
Inventor
姜军
刘小伟
冯蔚腾
莫运能
吴宗武
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP19874171.2A priority Critical patent/EP3855400A4/en
Publication of WO2020078250A1 publication Critical patent/WO2020078250A1/zh
Priority to US17/230,102 priority patent/US20210233272A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/292—Multi-camera tracking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29—Geographical information databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/12—Synchronisation between the display unit and other units, e.g. other display units, video-disc players
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/28—Indexing scheme for image data processing or generation, in general involving image processing hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the invention relates to the technical field of virtual scenes, in particular to a data processing method and device for virtual scenes.
  • VR Virtual reality
  • AR Augmented Reality
  • MR Mixed reality
  • VR technology is a simulation technology that can create and experience a virtual world
  • AR technology is a technology that can superimpose and interact virtual reality and the real world
  • MR technology is a comprehensive technology that creates a new visual environment by merging real and virtual worlds and builds an interactive feedback information loop between real and virtual worlds and users.
  • simultaneous positioning and mapping Simultaneous Localization and Mapping, SLAM
  • SLAM simultaneous Localization and Mapping
  • the SLAM technology can specifically realize that when a terminal (such as a mobile terminal such as a mobile phone or VR glasses) starts to move from an unknown location in the environment, it can position itself based on position estimation and maps during the movement, and build an increment based on its own positioning. Maps to facilitate subsequent positioning, systems or modules that use SLAM technology can also be referred to as spatial positioning engines.
  • different mobile terminals may have different types or models (such as mobile phones and VR glasses, or Apple phones and Huawei phones), and they have different spatial positioning engine algorithms.
  • Coordinate system synchronization content synchronization
  • virtual content in multi-user VR or AR or MR applications can be shared and interacted in the same coordinate system space.
  • One existing solution is to synchronize the coordinate system based on a regular geometric plane (for example, a square desktop).
  • the details are as follows:
  • the mobile terminal uses the coordinates of the center point of the horizontal desktop at the current location as the origin of the terminal reference coordinate system. Then use the desktop edge information to calculate the coordinate axis of the reference coordinate system. Furthermore, the terminal reference coordinate system is established.
  • the calculation process of mobile terminals of different users is the same, so that the same reference coordinate system is established, and the posture of the mobile terminal spatial positioning engine can be used to calculate the posture of the terminal relative to the reference coordinate system, and then calculate the relative position between the terminals Posture to achieve coordinate system synchronization.
  • This solution can realize multi-user VR or AR or MR application interaction on different mobile terminals on the same desktop.
  • the embodiments of the present invention provide a data processing method and device for a virtual scene, which can realize the cross-platform synchronization of the virtual scene coordinate system (content synchronization) of different terminals in any natural scene, compatible with various spatial positioning engines, and satisfy User needs.
  • an embodiment of the present invention provides a data processing method for a virtual scene, where the virtual scene is one of a virtual reality VR scene, an augmented reality AR scene, and a mixed reality MR scene.
  • the method includes: a first terminal collects a first image through a camera, the first image includes an image of any object in any natural environment (real world); and the first terminal obtains the first image according to the first image
  • the first point cloud data of the object represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the first terminal;
  • the first terminal according to the first Images to obtain the pose data of the first terminal in the first camera coordinate system;
  • the first terminal obtains two according to the pose data, the first point cloud data and the first image Dimensional feature image;
  • the first terminal generates map data based on the first point cloud data, the pose data, and the two-dimensional feature image;
  • the first terminal sends the map data to a second terminal or server .
  • the second terminal when the first terminal sends the map data to the second terminal, the second terminal may be used to synchronize the content in the virtual scene according to the map data; the space used by the first terminal and the second terminal
  • the positioning engines can be different or the same.
  • the server when the first terminal sends the map data to the server, the server is used to obtain between the first camera coordinate system of the first terminal and the second camera coordinate system of the second terminal according to the map data Transforms the coordinate system of the data and sends the coordinate system transformation data to the second terminal, so that the second terminal can synchronize the content in the virtual scene according to the coordinate system transformation data.
  • the spatial positioning engines used by the first terminal and the second terminal may be different or the same.
  • the first terminal can use the output data of the local spatial positioning engine to perform image feature extraction, and generate map data to send to the second terminal or server to facilitate the second terminal based on the map
  • the data synchronizes the coordinate system between the two terminals.
  • the spatial positioning engines used by the first terminal and the second terminal may be different. In other words, even if different terminals are installed with different spatial positioning engines, the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person AR / VR / MR development and applications between different spatial positioning engines User needs, improve user experience.
  • the embodiments of the present invention can perform feature extraction on any object in any natural scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not affected by the environment
  • the limitation is that in the process of implementing multi-user virtual scene interaction, there is no need to rely on objects of specific geometric shapes. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment, meeting the needs of various application scenarios .
  • the first terminal obtaining a two-dimensional feature image according to the pose data, the first point cloud data, and the first image includes: the first The terminal projects the first point cloud data to the first image according to the pose data, and obtains the two-dimensional image coordinates (also called the second point cloud data corresponding to the first point cloud data in the image coordinate system of the first image 2D image coordinates), and then, according to the image block or image feature point descriptor in the first image corresponding to the two-dimensional image coordinates, a two-dimensional feature image (also called a 2D feature image) is extracted.
  • the image feature point descriptor includes but is not limited to one of Brief, Brisk, ORB, Freak, SIFT, and SURF.
  • the first terminal may select a frame of image (first image) at a preset time, and the pose data and the first point cloud data of the same time stamp output by the spatial positioning engine, and the first point Cloud data is projected into the first image through the pose to obtain 2D image coordinates, and then perform 2D feature description on the 2D image coordinates.
  • the feature description may be an image feature point descriptor or an image block description, where the image feature Point descriptors include but are not limited to one of Brief, Brisk, ORB, Freak, SIFT, SURF.
  • the set of the 2D feature image, the first point cloud data, and the pose data associated with the same time stamp of the first terminal may be set as a key frame of the first terminal.
  • the spatial positioning engine of the first terminal may calculate and obtain the posture data of the first terminal in the first camera coordinate system according to the data (first image) collected by the camera.
  • an inertial measurement unit (IMU) sensor is also provided in the first terminal, then the input signal of the spatial positioning engine includes the image data collected from the camera and the data collected by the IMU sensor (it can be said Is motion data), the spatial positioning engine may calculate and obtain the pose data of the first terminal in the first camera coordinate system according to the image data and the motion data.
  • IMU inertial measurement unit
  • the first terminal in addition to the camera and the IMU, is also provided with a positioning module (GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.) related to posture or movement.
  • a positioning module GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.
  • the spatial positioning engine may further refer to the data collected by the positioning module to calculate and obtain the pose data of the first terminal in the first camera coordinate system.
  • the method further includes: The first terminal obtains positioning data, and the positioning data includes GPS positioning data of the terminal, Beidou positioning data of the first terminal, WIFI positioning data of the first terminal, and Bluetooth positioning of the first terminal At least one of data or base station positioning data of the first terminal. Then, the positioning data and the above-mentioned map data are packaged into a data package for subsequent transmission; or, the positioning data may be carried in the map data for subsequent transmission.
  • an embodiment of the present invention provides yet another data processing method for a virtual scene, where the virtual scene is one of a virtual reality VR scene, an augmented reality AR scene, and a mixed reality MR scene.
  • the method includes: a second terminal receives map data sent by a first terminal, the map data is based on first point cloud data of an arbitrary object, and the first terminal is in a first camera coordinate system of the first terminal Generated by the pose data of the first terminal and the two-dimensional feature image of the first terminal; wherein the first point cloud data represents the surface point of the object in the first camera coordinate system of the first terminal Three-dimensional coordinate values; the second terminal obtains the posture data of the second terminal in the second camera coordinate system of the second terminal; the second terminal obtains a two-dimensional feature image of the second terminal; The second terminal obtains the first camera coordinate system based on the map data, the posture data of the second terminal in the second camera coordinate system, and the two-dimensional feature image of the second terminal Coordinate system transformation data between the second camera coordinate systems; the second terminal
  • the second terminal can use the output data of the local spatial positioning engine to perform two-dimensional image feature extraction, receive the map data sent by the first terminal, and according to the map data, the The posture data of the second terminal and the two-dimensional feature image obtain coordinate system transformation data, and finally achieve coordinate system synchronization between the two terminals based on the coordinate system transformation data.
  • the spatial positioning engines used by the first terminal and the second terminal may be different. In other words, even if different terminals are installed with different spatial positioning engines, the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person AR / VR / MR development and applications between different spatial positioning engines. User needs, improve user experience.
  • the embodiments of the present invention can perform feature extraction on any object in any natural scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not affected by the environment
  • the limitation is that in the process of implementing multi-user virtual scene interaction, there is no need to rely on objects of specific geometric shapes. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment, meeting the needs of various application scenarios .
  • the second terminal is based on the map data, the pose data of the second terminal in the second camera coordinate system, and the two-dimensional Feature images, obtaining coordinate system transformation data between the first camera coordinate system and the second camera coordinate system, including: the second terminal using the two-dimensional feature image of the second terminal in the map data Perform a feature matching operation in to obtain the pose data of the second terminal in the first camera coordinate system; the second terminal according to the pose data of the second terminal in the first camera coordinate system And pose data in the second camera coordinate system to obtain coordinate system transformation data between the first camera coordinate system and the second camera coordinate system.
  • the second terminal may select a frame of image (second image) at a preset time, and the pose data and the second point cloud data of the same time stamp output by the spatial positioning engine, and the second point Cloud data is projected into the second image through the pose to obtain 2D image coordinates, and then perform 2D feature description on the 2D image coordinates.
  • the feature description may be an image feature point descriptor or an image block description, where the image feature Point descriptors include but are not limited to one of Brief, Brisk, ORB, Freak, SIFT, SURF.
  • the set of the 2D feature image of the second terminal, the second point cloud data, and the pose data of the second terminal associated with the same time stamp of the second terminal can be set as a key of the second terminal frame.
  • the second terminal can perform feature matching based on the key frame determined by itself and the received map data, extract the same image features respectively, and output a matching set of feature points, and then estimate the second terminal relative to the first
  • the pose data of a camera coordinate system where the pose estimation algorithm may be a PNP algorithm.
  • the second terminal obtains the first camera coordinate system and the second camera coordinate based on the second terminal's pose data in the first camera coordinate system and the pose data in the second camera coordinate system Coordinate system transformation data between systems (for example, a coordinate system transformation matrix).
  • the posture transformation data of the anchor points used in the virtual scene may be calculated according to the coordinate system transformation matrix.
  • the second terminal synchronizing the content in the virtual scene according to the coordinate system transformation data includes: the second terminal obtaining the anchor of the virtual scene according to the coordinate system transformation data Point posture transformation data; the second terminal synchronizes the content in the virtual scene according to the posture transformation data of the anchor point.
  • the method before the second terminal acquires the pose data of the second terminal in the second camera coordinate system of the second terminal, the method includes: A second image, the second image includes an image of the object. Since the second image is obtained by photographing the environment, and both the first terminal and the second terminal are in the environment, there is an image of the object in the environment in the second image, and the object is the first image collected by the first terminal. An object used in an image; the second terminal obtains second point cloud data of the object according to the second image; the second point cloud data indicates that the surface point of the object is in the second Three-dimensional coordinate values in the camera coordinate system;
  • acquiring the posture data of the second terminal in the second camera coordinate system of the second terminal by the second terminal includes: the second terminal obtaining the The pose data of the two terminals in the second camera coordinate system;
  • the second terminal acquiring the two-dimensional feature image of the second terminal includes: according to the pose data of the second terminal in the second camera coordinate system, the second terminal Projecting the second point cloud data onto the second image to obtain the two-dimensional image coordinates corresponding to the second point cloud data in the image coordinate system of the second image; the second terminal according to the two-dimensional Image coordinates correspond to image blocks or image feature point descriptors in the second image to obtain a two-dimensional feature image of the second terminal.
  • the image feature point descriptor includes but is not limited to one of Brief, Brisk, ORB, Freak, SIFT, and SURF.
  • the map data further carries positioning data
  • the positioning data includes GPS positioning data of the first terminal, Beidou positioning data of the first terminal, and the first At least one of WIFI positioning data of a terminal, Bluetooth positioning data of the first terminal, or base station positioning data of the first terminal; correspondingly, the second terminal obtains the two-dimensional of the second terminal Feature images, including: the second terminal determines a search range of a two-dimensional feature image based on the positioning data; and based on the search range, determines the two-dimensional feature of the second terminal from a plurality of historical two-dimensional feature images image.
  • the second terminal can first determine the two-dimensional according to the positioning data The search range of the feature image, so as to filter multiple key frames generated by the second terminal multiple times according to the search range, for example, select a key frame that is closer to or closest to the positioning data, and then use the key frame in the map data Perform feature matching to obtain the pose data of the second terminal in the first camera coordinate system of the first terminal. In this way, it is helpful to improve the matching efficiency and accuracy.
  • an embodiment of the present invention provides a data processing method for a virtual scene.
  • the virtual scene is one of a virtual reality VR scene, an augmented reality AR scene, and a mixed reality MR scene.
  • the method includes: The server receives map data sent by the first terminal, the map data is first point cloud data based on the object, posture data of the first terminal in the first camera coordinate system of the first terminal, and the Generated by a two-dimensional feature image of the first terminal; wherein the first point cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the first terminal; the server receives The posture data of the second terminal in the second camera coordinate system of the second terminal, and the two-dimensional feature image of the second terminal; the server uses the two-dimensional feature image of the second terminal in the Performing feature matching operations on the map data to obtain the pose data of the second terminal in the first camera coordinate system; the server according to the second terminal in the first camera coordinate system Posture data and posture data in the second camera coordinate system to obtain coordinate system transformation data between
  • different mobile terminals can use the output data of their respective spatial positioning engines to perform the same type of image feature extraction, where the first terminal can further generate map data and upload it to the server, and the second terminal can further Generate the key frame and upload it to the server.
  • the server then obtains the coordinate system synchronization data according to the key frame and map data and sends it to the second terminal to achieve coordinate system synchronization between the terminals.
  • the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person sharing AR / VR / MR development and applications between different spatial positioning engines, improving User experience.
  • the server since the server is used as a data interaction node between different terminals, it can efficiently meet the content synchronization requirements between a large number of terminals.
  • the embodiments of the present invention can perform feature extraction on objects in any scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not limited by the environment That is, in the process of realizing the interaction of multi-user virtual scenes, objects of specific geometric shapes are no longer needed. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment and meets the needs of various application scenarios.
  • an embodiment of the present invention provides a terminal for data processing of a virtual scene, where the virtual scene is one of a virtual reality VR scene, an augmented reality AR scene, and a mixed reality MR scene.
  • the terminal specifically includes: an image acquisition module for acquiring a first image through a camera of the terminal, the first image including an image of an object; and a SLAM module for acquiring a first object of the object according to the first image Point cloud data; the first point cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the terminal; the SLAM module is further used to obtain, based on the first image A pose data of the terminal in the first camera coordinate system; a feature extraction module, configured to obtain a two-dimensional feature image based on the pose data, the first point cloud data, and the first image; A map generation module is used to generate map data based on the first point cloud data, the pose data and the two-dimensional feature image; a communication module is used to send the map data to other terminals or
  • the feature extraction module is specifically configured to: according to the pose data, project the first point cloud data onto the first image to obtain the first point
  • the cloud data corresponds to the two-dimensional image coordinates in the image coordinate system of the first image; according to the two-dimensional image coordinates corresponding to the image block or the image feature point descriptor in the first image, the two-dimensional image is obtained Feature image.
  • the SLAM module is specifically configured to: obtain the terminal in the first camera coordinate system according to the first image and the motion data collected by the inertial measurement unit IMU Pose data.
  • the terminal further includes a positioning module; the positioning module is configured to acquire positioning data, the positioning data includes GPS positioning data of the terminal, and Beidou positioning of the terminal At least one of data, WIFI positioning data of the terminal, Bluetooth positioning data of the terminal, or base station positioning data of the terminal; correspondingly, the map data generated by the map generation module also carries the Positioning data.
  • each functional module of the terminal may be used to implement the method described in the first aspect.
  • an embodiment of the present invention provides a terminal for data processing in a virtual scene, where the virtual scene is one of a virtual reality VR scene, an augmented reality AR scene, and a mixed reality MR scene.
  • the terminal specifically includes: a communication module for receiving map data sent by other terminals, the map data is based on first point cloud data of the object, and the posture of the other terminal in the first camera coordinate system of the other terminal Generated by the data and the two-dimensional feature image of the other terminal; wherein the first point cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the other terminal;
  • SLAM The module is used to obtain the posture data of the terminal in the second camera coordinate system of the terminal; the SLAM module is also used to obtain a two-dimensional feature image of the terminal; the coordinate system transformation module is used to The map data, the posture data of the terminal in the second camera coordinate system, and the two-dimensional feature image of the terminal, to obtain the relationship between the first camera coordinate system and the second camera coordinate system Coordinat
  • the coordinate system transformation module is specifically configured to: use the two-dimensional feature image of the terminal to perform feature matching operations in the map data to obtain the terminal in the first Pose data in a camera coordinate system; obtain the first camera coordinate system according to the posture data of the terminal in the first camera coordinate system and the posture data in the second camera coordinate system The coordinate system transforms data with the second camera coordinate system.
  • the terminal further includes an image acquisition module configured to acquire a second image through a camera, the second image including an image of the object;
  • the SLAM module is also used to obtain second point cloud data of the object according to the second image;
  • the second point cloud data represents the three-dimensional coordinates of the surface point of the object in the second camera coordinate system Value;
  • the SLAM module is specifically used to obtain the posture data of the terminal in the second camera coordinate system according to the second image;
  • the SLAM module is also specifically used to The pose data in the second camera coordinate system, projecting the second point cloud data onto the second image to obtain the second point cloud data corresponding to the image coordinate system in the second image Two-dimensional image coordinates; according to the two-dimensional image coordinates corresponding to the image block or the image feature point descriptor in the second image, obtain the two-dimensional feature image of the terminal.
  • the content synchronization module is specifically configured to: obtain posture transformation data of the anchor point of the virtual scene according to the coordinate system transformation data; and transform the posture of the anchor point Data to synchronize the content in the virtual scene.
  • the map data further carries positioning data
  • the positioning data includes GPS positioning data of the other terminals, Beidou positioning data of the other terminals, and At least one of WIFI positioning data, Bluetooth positioning data of the other terminal, or base station positioning data of the other terminal;
  • the SLAM module is specifically used to obtain the two-dimensional characteristics of the terminal according to the positioning data image.
  • each functional module of the terminal may be used to implement the method described in the second aspect.
  • an embodiment of the present invention provides a server for data processing in a virtual scene.
  • the server includes: a receiving module for receiving map data sent by a first terminal, where the map data is based on an object One point cloud data, the pose data of the first terminal in the first camera coordinate system of the first terminal, and the two-dimensional feature image of the first terminal; wherein, the first point Cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the first terminal; the receiving module is further used to receive the second camera coordinate of the second terminal in the second terminal The pose data in the system and the two-dimensional feature image of the second terminal; the coordinate system transformation module is used to perform the feature matching operation in the map data using the two-dimensional feature image of the second terminal to obtain The posture data of the second terminal in the first camera coordinate system; according to the posture data of the second terminal in the first camera coordinate system and the posture in the second camera coordinate system number , Coordinate transformation data obtained between the first camera and the second camera coordinate system is the coordinate system; transmitting means
  • each functional module of the server may be used to implement the method described in the third aspect.
  • an embodiment of the present invention provides another terminal.
  • the terminal includes: a processor, a coupled memory, and a transmitter.
  • the processor can call program instructions stored in the memory to implement the method described in the first aspect.
  • an embodiment of the present invention provides yet another terminal.
  • the terminal includes: a processor and a coupled memory and receiver.
  • the processor can call program instructions stored in the memory to implement the method described in the second aspect.
  • an embodiment of the present invention provides another server.
  • the server includes a processor and a coupled memory and a communication interface.
  • the processor can call program instructions stored in the memory to implement the method described in the third aspect.
  • an embodiment of the present invention provides a non-volatile computer-readable storage medium; the computer-readable storage medium is used to store the implementation code of the method of the first aspect.
  • the program code is executed by a computing device, the computing device is used in the method of the first aspect.
  • an embodiment of the present invention provides yet another non-volatile computer-readable storage medium; the computer-readable storage medium is used to store the implementation code of the method of the second aspect.
  • the program code is executed by a computing device, the user equipment is used in the method of the second aspect.
  • an embodiment of the present invention provides yet another non-volatile computer-readable storage medium; the computer-readable storage medium is used to store the implementation code of the method of the third aspect.
  • the program code is executed by a computing device, the user equipment is used in the method of the third aspect.
  • an embodiment of the present invention provides a computer program product; the computer program product includes program instructions, and when the computer program product is executed by a computing device, the controller performs the method described in the first aspect.
  • the computer program product may be a software installation package, and in the case where it is necessary to use any of the methods provided by any of the possible designs of the first aspect, the computer program product may be downloaded and executed on the controller to Implementing the method described in the first aspect.
  • an embodiment of the present invention provides yet another computer program product.
  • the computer program product includes program instructions, and when the computer program product is executed by the user equipment, the controller executes the method provided by any one of the possible designs of the foregoing second aspect.
  • the computer program product may be a software installation package.
  • the computer program product may be downloaded and executed on the controller to Implementing the method described in the second aspect.
  • an embodiment of the present invention provides yet another computer program product.
  • the computer program product includes program instructions, and when the computer program product is executed by the user equipment, the controller executes the method provided by any one of the possible designs of the foregoing second aspect.
  • the computer program product may be a software installation package.
  • the computer program product may be downloaded and executed on the controller to Implementing the method described in the third aspect.
  • different mobile terminals can use the output data of their respective spatial positioning engines to perform the same type of image feature extraction.
  • the second terminal realizes it based on the key frames generated by itself and the map data provided by the first terminal
  • the coordinate system between the two terminals is synchronized, or the second terminal realizes the coordinate system synchronization between the two terminals according to the coordinate system transformation data delivered by the server.
  • the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person sharing AR / VR / MR development and applications between different spatial positioning engines User experience.
  • the embodiments of the present invention can perform feature extraction on objects in any scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not limited by the environment , That is, in the process of implementing multi-user virtual scene interaction, there is no need to rely on known information such as specific geometric objects. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment and meets various application scenarios Demand.
  • FIG. 1 is a schematic diagram of an application architecture provided by an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of yet another application architecture provided by an embodiment of the present invention.
  • FIG. 3 is a schematic flowchart of a data processing method for a virtual scene provided by an embodiment of the present invention
  • FIG. 4 is a schematic diagram of a map and a key frame generated by a first terminal and a second terminal provided by an embodiment of the present invention
  • FIG. 5 is a schematic diagram of obtaining coordinate system transformation data and anchor point transformation data by a second terminal according to an embodiment of the present invention
  • FIG. 6 is a schematic flowchart of yet another data processing method for a virtual scene provided by an embodiment of the present invention.
  • FIG. 7 is a schematic structural diagram of a terminal provided by an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a server according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of a system, a first terminal, and a second terminal provided by an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of yet another system provided by an embodiment of the present invention, and a first terminal, a second terminal, and a server.
  • the first application architecture provided by an embodiment of the present invention includes multiple terminals.
  • the terminal may be a mobile terminal device such as a mobile phone, a tablet computer, a notebook computer, a video recorder, or a virtual scene terminal device (such as VR glasses, MR Interactive devices, etc.).
  • Each terminal in the multiple terminals may be the same terminal (for example, a mobile phone of the same model) or a terminal of a different terminal (for example, multiple terminals include mobile phones of different models, or multiple terminals include mobile terminal devices and virtual scenes) Terminal equipment), for example, the first terminal and the second terminal are exemplarily shown in FIG. 1, and the first terminal and the second terminal are different types of mobile phones.
  • the first terminal and the second terminal may move in the same environment in the real world, and the environment may be any natural environment.
  • the first terminal and the second terminal respectively run virtual scene applications (VR or AR or MR applications), and a communication connection can be made between the first terminal and the second terminal.
  • the first terminal can collect information of any object in the environment through a camera and / or a sensor (for example, the image of the object 1 in the environment is collected in the illustration, and the object 1 may be a complete object, or may be a partial feature of an object),
  • data is sent to the second terminal (as in the sending of map data described below).
  • the second terminal may synchronize the virtual scene content moved by the second terminal with the virtual scene content run by the first terminal according to the data sent by the first terminal.
  • a second application architecture provided by an embodiment of the present invention includes multiple terminals and servers, where the terminals may be mobile terminal devices such as mobile phones, tablet computers, notebook computers, video recorders, or virtual scene terminal devices (such as VR Glasses, MR interactive equipment, etc.).
  • Each terminal in the multiple terminals may be the same terminal (for example, a mobile phone of the same model) or a terminal of a different terminal (for example, multiple terminals include mobile phones of different models, or multiple terminals include mobile terminal devices and virtual scene Terminal equipment), for example, the first terminal and the second terminal are exemplarily shown in FIG. 2, and the first terminal and the second terminal are different types of mobile phones.
  • the server may be one or more physical servers (for example, one physical server is exemplarily shown in FIG. 2), a computer cluster, or a virtual machine in a cloud computing scenario.
  • the first terminal and the second terminal can move in the same environment scene in the real world.
  • the first terminal and the second terminal can respectively run virtual scene applications (VR or AR or MR applications), and the first terminal and the server can be connected for communication, and the second terminal and the server can also be connected for communication.
  • the first terminal may collect information of any object in the environment through a local camera and / or sensor (for example, an image of the object 1 in the environment is collected in the illustration), and then upload data to the server (such as sending map data described below).
  • the second terminal may also collect the information of the same object in the environment as the first terminal through the local camera and / or sensor (for example, collect the image of the object 1 in the environment in the illustration), and then upload the data to the server (as described below) 2D feature image).
  • the server synthesizes the above data to perform calculation to obtain content synchronization related data (coordinate system conversion data as described below), and delivers the related data to the second terminal, and the second terminal implements the virtual scene content and motion of the second terminal according to the related data Synchronization of virtual scene content run by the first terminal.
  • content synchronization related data coordinate system conversion data as described below
  • FIG. 3 is a schematic flowchart of a data processing method according to an embodiment of the present invention, and is described from the perspective of the first terminal side and the second terminal side.
  • the first terminal and the second terminal may be the same terminal or different terminals.
  • the method includes but is not limited to the following steps:
  • Step 101 The first terminal collects the first image through the camera of the first terminal.
  • the first terminal is provided with a camera (or camera, for details, refer to the description about the structure of the terminal below). Therefore, the first terminal can shoot the real world environment through the camera to obtain the first image.
  • a camera or camera, for details, refer to the description about the structure of the terminal below. Therefore, the first terminal can shoot the real world environment through the camera to obtain the first image.
  • Step 102 The first terminal obtains first point cloud data of objects in the environment according to the first image.
  • the first terminal is configured with a spatial positioning engine (also called SLAM module or SLAM system, for details, refer to the description about the terminal structure below).
  • a spatial positioning engine also called SLAM module or SLAM system, for details, refer to the description about the terminal structure below.
  • the spatial positioning engine may calculate the first point cloud data of the object according to the first image, where the first point cloud data represents the three-dimensional surface points of the object in the environment in the first camera coordinate system of the first terminal Coordinate value (or 3D coordinate value).
  • the first camera coordinate system of the first terminal is a local reference coordinate system of the first terminal.
  • the first camera coordinate system is a rectangular coordinate system formed by the intersection of the optical axis and the image plane as the origin of the image coordinate system.
  • the origin of the first camera coordinate system is the optical center of the camera of the first terminal
  • the x-axis and y-axis of the first camera coordinate system are parallel to the two axes of the image coordinate system
  • the The z axis may be the optical axis of the camera, which is perpendicular to the image plane.
  • Step 103 The first terminal obtains the pose data of the first terminal in the first camera coordinate system according to the first image.
  • the posture data is used to represent the position and posture (or direction) of the first terminal in the first camera coordinate system
  • the position can be represented by three coordinate axes x, y, z
  • the direction can be ( ⁇ , ⁇ , ⁇ ) )
  • ( ⁇ , ⁇ , ⁇ ) represents the angle of rotation around the three coordinate axes.
  • the spatial positioning engine of the first terminal may calculate and obtain the posture data of the first terminal in the first camera coordinate system according to the data (first image) collected by the camera.
  • an inertial measurement unit (IMU) sensor is also provided in the first terminal, then the input signal of the spatial positioning engine includes image data collected from the camera and data collected by the IMU sensor (it can be said Is motion data), the spatial positioning engine may calculate and obtain the pose data of the first terminal in the first camera coordinate system according to the image data and the motion data.
  • IMU inertial measurement unit
  • the first terminal in addition to the camera and the IMU, is also provided with a positioning module (GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.) related to posture or movement.
  • a positioning module GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.
  • the spatial positioning engine may further refer to the data collected by the positioning module to calculate and obtain the pose data of the first terminal in the first camera coordinate system.
  • Step 104 The first terminal obtains a two-dimensional feature image according to the pose data, the first point cloud data, and the first image corresponding to the same time stamp.
  • the first terminal projects the first point cloud data to the first image according to the pose data, and obtains the second point cloud data corresponding to the second in the image coordinate system of the first image Dimensional image coordinates (also called 2D image coordinates), and then, according to the image block or image feature point descriptor in the first image corresponding to the two-dimensional image coordinates, extract the two-dimensional feature image (also called 2D feature image) ).
  • the first image projects the first point cloud data to the first image according to the pose data, and obtains the second point cloud data corresponding to the second in the image coordinate system of the first image Dimensional image coordinates (also called 2D image coordinates), and then, according to the image block or image feature point descriptor in the first image corresponding to the two-dimensional image coordinates, extract the two-dimensional feature image (also called 2D feature image) ).
  • the first terminal may select a frame of image (first image) at a preset time, and the pose data of the same time stamp output by the spatial positioning engine And the first point cloud data, project the first point cloud data into the first image through the pose, obtain 2D image coordinates, and then perform a 2D feature description on the 2D image coordinates
  • the feature description may be an image feature point description Sub or image block description, wherein the image feature point descriptor includes but is not limited to one of Brief, Brisk, ORB, Freak, SIFT, SURF.
  • the set of the 2D feature image, the first point cloud data, and the pose data associated with the same time stamp of the first terminal may be set as a key frame of the first terminal.
  • Step 105 The first terminal generates map data according to the first point cloud data, the pose data, and the two-dimensional feature image.
  • the first terminal generates first point cloud data and posture data according to the spatial positioning engine, and the 2D feature image associated with the first point cloud data extracted in step 104 above to generate map data, which is packaged into a data package to facilitate Follow-up outgoing.
  • first point cloud data, and pose data associated with the same time stamp can be set as a key frame of the first terminal, if the first terminal Steps 101 to 104 described above are repeated at a preset time. Then, in step 105, the final packaged map data may be a set of one or more key frames in the local area where the first terminal is located.
  • the first terminal when the first terminal is provided with a positioning module (such as GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc., please refer to the description about the terminal structure below), then the first terminal also The positioning data of the first terminal when generating the map data may be obtained by the positioning module, that is, the positioning data includes at least one of GPS positioning data, Beidou positioning data, WIFI positioning data, Bluetooth positioning data, or base station positioning data. Then, the positioning data and the above-mentioned map data are packaged into a data package (or the positioning data is carried in the map data), so as to facilitate subsequent outgoing distribution.
  • a positioning module such as GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc., please refer to the description about the terminal structure below.
  • Step 106 The first terminal sends map data to the second terminal, and correspondingly, the second terminal receives the map data.
  • Step 107 The second terminal collects the second image through the camera of the second terminal.
  • the second terminal may also be provided with a camera (or camera, for details, refer to the description about the structure of the terminal below). Therefore, the second terminal can shoot the real world environment through the camera to obtain the second image.
  • a camera or camera, for details, refer to the description about the structure of the terminal below. Therefore, the second terminal can shoot the real world environment through the camera to obtain the second image.
  • Step 108 The second terminal obtains second point cloud data of the object in the environment according to the second image.
  • the second point cloud data represents a three-dimensional coordinate value of the surface point of the object in the second camera coordinate system of the second terminal.
  • the second terminal is also equipped with a spatial positioning engine (also called SLAM module or SLAM system, for details, refer to the description of the terminal structure below), and the spatial positioning engine of the second terminal may be different from that of the first terminal
  • the spatial positioning engine may also be the same as the spatial positioning engine of the first terminal. Since the second image is obtained by photographing the environment, there is an image of an object in the environment in the second image, and the object is the object adopted by the first terminal in step 102.
  • the spatial positioning engine can calculate the second point cloud data of the object according to the second image, where the second point cloud data represents the second camera coordinates of the surface point of the object in the environment at the second terminal
  • the second camera coordinate system of the second terminal is a local reference coordinate system of the second terminal.
  • the second camera coordinate system is a rectangular coordinate system formed by the intersection of the optical axis and the image plane as the origin of the image coordinate system.
  • the origin of the second camera coordinate system is the optical center of the camera of the second terminal
  • the x-axis and y-axis of the second camera coordinate system are parallel to the two axes of the image coordinate system
  • the The z axis may be the optical axis of the camera, which is perpendicular to the image plane.
  • Step 109 The second terminal obtains pose data of the second terminal in the second camera coordinate system according to the second image.
  • the posture data is used to indicate the position and posture (or direction) of the second terminal in the second camera coordinate system.
  • the position can be represented by three coordinate axes x, y, and z, and the direction can be passed ( ⁇ , ⁇ , ⁇ ) to represent, ( ⁇ , ⁇ , ⁇ ) represents the angle of rotation around the three coordinate axes.
  • the spatial positioning engine of the second terminal may calculate and obtain the pose data of the second terminal in the second camera coordinate system according to the data (second image) collected by the camera.
  • an IMU sensor is also provided in the second terminal, then the input signal of the spatial positioning engine includes image data collected from the camera and data collected by the IMU sensor (may be referred to as motion data), and the spatial positioning engine may Based on the image data and the motion data, the posture data of the second terminal in the second camera coordinate system is calculated and obtained.
  • the second terminal in addition to the camera and the IMU, is also provided with a positioning module (GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.) related to posture or movement.
  • a positioning module GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.
  • the spatial positioning engine may further refer to the data collected by the positioning module to calculate and obtain the pose data of the second terminal in the second camera coordinate system.
  • Step 110 The second terminal obtains a two-dimensional feature image according to the pose data, the second point cloud data, and the second image.
  • the second terminal projects the second point cloud data onto the second image according to the second terminal's pose data in the second camera coordinate system to obtain the second point cloud data Corresponding to the two-dimensional image coordinates (also called 2D image coordinates) in the image coordinate system of the second image, and then corresponding to the image block or image feature point description in the second image according to the two-dimensional image coordinates Then, obtain a two-dimensional feature image (also called a 2D feature image) of the second terminal.
  • a two-dimensional feature image also called a 2D feature image
  • the second terminal may select a frame of image (second image) at a preset time, and the pose data of the same time stamp output by the spatial positioning engine And the second point cloud data, project the second point cloud data into the second image through the pose, obtain the 2D image coordinates, and then perform a 2D feature description on the 2D image coordinates
  • the feature description may be an image feature point description Sub or image block description, wherein the image feature point descriptor includes but is not limited to one of Brief, Brisk, ORB, Freak, SIFT, SURF.
  • the set of the 2D feature image of the second terminal, the second point cloud data, and the pose data of the second terminal associated with the same time stamp of the second terminal can be set as a key of the second terminal frame.
  • step 106 may be implemented before step 107, and step 106 may also be implemented between step 107-step 110, and step 106 also It may be implemented after step 110, and the invention is not specifically limited.
  • Step 111 The second terminal obtains the first camera according to the map data, the posture data of the second terminal in the second camera coordinate system, and the two-dimensional feature image of the second terminal
  • the coordinate system transforms data between the coordinate system and the second camera coordinate system.
  • the second terminal may use key frames of the second terminal to perform feature matching in the map data sent from the first terminal, thereby obtaining the first camera coordinate system of the second terminal at the first terminal (here also May be referred to as the pose data in the map coordinate system); then, the second terminal according to the pose data of the second terminal in the first camera coordinate system and the pose in the second camera coordinate system Data to obtain coordinate system transformation data between the first camera coordinate system and the second camera coordinate system.
  • the second terminal can perform feature matching based on the key frame determined by itself and the received map data, extract the same image features of the two, and output a matching set of feature points, and then estimate The pose data of the second terminal relative to the first camera coordinate system, where the pose estimation algorithm may be a PNP algorithm. Then, the second terminal obtains the first camera coordinate system and the second camera coordinate based on the second terminal's pose data in the first camera coordinate system and the pose data in the second camera coordinate system Coordinate system transformation data between systems (for example, a coordinate system transformation matrix).
  • the posture transformation data of the anchor points used in the virtual scene may be calculated according to the coordinate system transformation matrix.
  • the second terminal may First, determine the search range of the two-dimensional feature image based on the positioning data, so as to filter multiple key frames generated through step 110 multiple times according to the search range, for example, select a key frame that is closer to or closest to the positioning data, and then Use the key frame to perform feature matching in the map data to obtain the pose data of the second terminal in the first camera coordinate system of the first terminal. In this way, it is helpful to improve the matching efficiency and accuracy.
  • Step 112 The second terminal performs synchronization operation on the content in the virtual scene (such as a virtual object) according to the coordinate system transformation data.
  • the second terminal may synchronize the posture of the virtual object in the virtual scene according to the posture transformation data of the anchor point, that is to say Through the synchronization operation, the second terminal can correctly display the posture of the virtual object shared by the first terminal.
  • different mobile terminals can use the output data of their respective spatial positioning engines to perform the same type of image feature extraction, and the second terminal can then use the key frames generated by itself and the map data provided by the first terminal
  • the coordinate system between the two terminals is synchronized.
  • the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person sharing AR / VR / MR development and applications between different spatial positioning engines, improving User experience.
  • the embodiments of the present invention can perform feature extraction on objects in any scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not limited by the environment , That is, in the process of implementing multi-user virtual scene interaction, there is no longer need to rely on known information such as Marker images, objects with specific geometric shapes, etc. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment. Requirements of various application scenarios.
  • FIG. 6 is a schematic flowchart of still another data processing method according to an embodiment of the present invention, and is described from the perspective of the first terminal side, the server side, and the second terminal side.
  • the first terminal and the second terminal may be the same terminal or different terminals.
  • the method includes but is not limited to the following steps:
  • Step 201 The first terminal collects the first image through the camera of the first terminal.
  • the first terminal collects the first image through the camera of the first terminal.
  • Step 202 The first terminal obtains first point cloud data of objects in the environment according to the first image.
  • first point cloud data For specific implementation, reference may be made to the description of step 102 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 203 The first terminal obtains pose data of the first terminal in the first camera coordinate system according to the first image.
  • step 103 For specific implementation, reference may be made to the description of step 103 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 204 The first terminal obtains a two-dimensional feature image according to the pose data, the first point cloud data, and the first image.
  • step 104 For specific implementation, reference may be made to the description of step 104 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 205 The first terminal generates map data according to the first point cloud data, the pose data, and the two-dimensional feature image.
  • map data For specific implementation, reference may be made to the description of step 105 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 206 The first terminal sends map data to the server, and accordingly, the server receives the map data.
  • Step 207 The second terminal collects the second image through the camera of the second terminal.
  • the second terminal collects the second image through the camera of the second terminal.
  • Step 208 The second terminal obtains second point cloud data of the object in the environment according to the second image.
  • step 108 For specific implementation, reference may be made to the description of step 108 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 209 The second terminal obtains pose data of the second terminal in the second camera coordinate system according to the second image.
  • step 109 For specific implementation, reference may be made to the description of step 109 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 210 The second terminal obtains a two-dimensional feature image according to the pose data, the second point cloud data, and the second image.
  • the second terminal obtains a two-dimensional feature image according to the pose data, the second point cloud data, and the second image.
  • Step 211 The second terminal sends the two-dimensional feature image and the posture data of the second terminal to the server.
  • the second terminal may set the set of the 2D feature image of the second terminal, the second point cloud data, and the pose data of the second terminal in the second camera coordinate system associated with the same time stamp as the first A key frame for two terminals. Then, the second terminal may also directly send the key frame to the server.
  • Step 212 The server obtains coordinate system transformation data according to the map data, the posture data of the second terminal in the second camera coordinate system, and the two-dimensional feature image of the second terminal.
  • the server may use key frames of the second terminal to perform feature matching in the map data sent from the first terminal, thereby obtaining the first camera coordinate system of the second terminal at the first terminal (here also May be referred to as the pose data in the map coordinate system); then, the server according to the pose data of the second terminal in the first camera coordinate system and the pose data in the second camera coordinate system, Obtain coordinate system transformation data between the first camera coordinate system and the second camera coordinate system.
  • the subsequent server may also calculate the posture transformation data of the anchor point used in the virtual scene according to the coordinate system transformation data.
  • step 212 for the specific implementation process of this step 212, reference may be made to the description of step 111 in the embodiment of FIG. 3, and details are not described herein again.
  • Step 213 The server sends the coordinate system transformation data to the second terminal. It should be noted that, in step 212, if the server has calculated the posture transformation data of the anchor point used in the virtual scene according to the coordinate system transformation data, the server may also send the posture transformation data of the anchor point to the second terminal.
  • Step 214 The second terminal performs synchronization operation on the content in the virtual scene according to the coordinate system transformation data.
  • step 112 the description of step 112 in the embodiment of FIG. 3, and details are not repeated here.
  • different mobile terminals can use the output data of their respective spatial positioning engines to perform the same type of image feature extraction, where the first terminal can further generate map data and upload it to the server, and the second terminal can further Generate the key frame and upload it to the server.
  • the server then obtains the coordinate system synchronization data according to the key frame and map data and sends it to the second terminal to achieve coordinate system synchronization between the terminals.
  • the content in the virtual scene can still be synchronized, which is conducive to cross-platform multi-person sharing AR / VR / MR development and applications between different spatial positioning engines User experience.
  • the server since the server is used as a data interaction node between different terminals, it can efficiently meet the content synchronization requirements between a large number of terminals.
  • the embodiments of the present invention can perform feature extraction on objects in any scene through an effective feature extraction method, thereby achieving coordinate system synchronization between different terminals, so the AR application scene shared by multiple people is not limited by the environment , That is, in the process of implementing multi-user virtual scene interaction, there is no longer need to rely on known information such as Marker images, objects with specific geometric shapes, etc. Therefore, the realization of multi-person sharing AR / VR / MR applications is no longer affected by the environment. Requirements of various application scenarios.
  • FIG. 7 is a structural block diagram of an implementation manner of a terminal according to an embodiment of the present invention.
  • the terminal 300 may include a chip 310, a memory 315 (one or more computer-readable storage media), a peripheral system 317, a positioning module 331, and a communication module 332. These components can communicate on one or more communication buses 314.
  • the peripheral system 317 is mainly used to realize the interactive function between the terminal 300 and the user / external environment.
  • the peripheral system 317 may include: a touch screen controller 318, a camera controller 319, and several components in the sensor management module 320. Each controller may be coupled with respective corresponding peripheral devices such as touch screen 323, camera 324, and sensor 325.
  • the touch screen 323 can be obtained by coupling a touch sensing element and a display screen.
  • the touch sensing element can be disposed below the display screen. The touch sensing element can be used to detect when a user inputs a touch operation through the display screen.
  • the touch pressure acting on the display screen (such as the user's touch and sliding operations), and the display screen is used for content display (such as displaying virtual objects for virtual scene applications);
  • the camera 324 may be a monocular camera Or a binocular camera or a depth camera;
  • the sensor 325 may include an inertial measurement unit IMU, and may further include other sensors (for example, a proximity sensor).
  • the peripheral system 317 may also include other I / O peripherals.
  • the chip 310 may integrate one or more processors 311, a clock module 312, and possibly a power management module 313.
  • the clock module 312 integrated in the chip 310 is mainly used to generate a clock required for data transmission and timing control for the processor 311.
  • the power management module 313 integrated in the baseband chip 310 is mainly used to provide a stable and highly accurate voltage for the processor 311 and peripheral systems.
  • the memory 315 is coupled to the processor 311 and is used to store data (eg, image data, point cloud data, map data, key frame data, coordinate system conversion data, etc.), various software programs, and / or multiple sets of program instructions.
  • the memory 315 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.
  • the memory 315 may also store one or more application programs. As shown in the figure, these applications may include: virtual scene applications such as AR / VR / MR, map applications, image management applications, and so on.
  • the memory 115 may also be used to store an operating system, such as an embedded operating system such as ANDROID, IOS, WINDOWS, or LINUX.
  • the memory 115 may also store a network communication program that can be used to communicate with one or more additional devices, one or more terminal devices, and one or more network devices.
  • the memory 115 can also store a user interface program, which can vividly display the contents of the application program (such as virtual objects in a virtual scene) through a graphical operation interface, and input through menus, dialog boxes, and keys.
  • the control receives the user's control operations on the application.
  • the positioning module 331 is used to realize the physical positioning of the terminal 300, and the movement data of the terminal 300 can be collected through the positioning module 331.
  • the positioning module 331 may include, for example, one or more of a WIFI positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module (such as a GPS positioning module and / or a Beidou positioning module).
  • the communication module 332 integrates a transmitter and a receiver for receiving and sending radio frequency signals, so that the terminal 300 establishes a communication connection with other terminals, or the terminal 300 establishes a communication connection with a server.
  • the communication module 332 mainly integrates the receiver and transmitter of the terminal 300.
  • the communication module 332 may include but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card, and a storage medium, etc. .
  • the communication module 332 may also be implemented on a separate chip.
  • terminal 300 is only an example provided by the embodiment of the present invention, and the terminal 300 may have more or less components than those shown, may combine two or more components, or may have components Different configurations.
  • the terminal 300 may be the first terminal and / or the second terminal in the embodiment of FIG. 3 or FIG. 6 described above.
  • the camera 324 can be used to collect images of objects in the environment, and the processor 311 can be used to call program instructions in the memory 315 to implement the first of the data processing methods described in the embodiment of FIG. 3 or FIG.
  • the functions of the terminal or the second terminal, so the functions of the various components of the terminal 300 can be referred to the above description, which will not be repeated here.
  • FIG. 8 is a structural block diagram of an implementation manner of a server according to an embodiment of the present invention.
  • the server 400 includes a processor 403, a memory 401 (one or more computer-readable storage media), and a communication interface 402. These components can communicate on one or more communication buses 404. among them:
  • the processor 403 may be one or more central processing units (Central Processing Unit, CPU).
  • CPU Central Processing Unit
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the memory 401 is coupled to the processor 403 and is used to store various software programs and / or multiple sets of instructions and data (such as map data, key frame data, coordinate system conversion data, etc.).
  • the memory 401 includes, but is not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), and erasable programmable read-only memory (Erasable Programmable Read Only Only Memory, EPROM) or portable read-only memory (Compact Disc Read-Only Memory, CD-ROM).
  • the memory 401 can also be used to store algorithm programs for calculating coordinate system conversion data.
  • the communication interface 402 mainly integrates a receiver and a transmitter, where the receiver is used to receive data sent by the terminal, and the transmitter is used to send data to the terminal.
  • the transmitter may be used to deliver coordinate system conversion data or anchor point gesture conversion data to the terminal, and the receiver may be used to receive map data or key frame data uploaded by the terminal.
  • server 400 is only an example provided by the embodiment of the present invention. In a specific implementation, the server 400 may have more components than shown.
  • the above-mentioned server 400 may be the server in the above-mentioned embodiment of FIG. 6.
  • the processor 403 may be used to call program instructions in the memory 401 to implement the functions of the server in the data processing method described in the embodiment of FIG. 6 above in the embodiment of the present invention. The description of the article will not be repeated here.
  • FIG. 9 is a structural block diagram of a multi-user interaction system provided by an embodiment of the present invention.
  • the system includes a terminal 51 and a terminal 52, and a communication connection can be implemented between the terminal 51 and the terminal 52 through respective communication modules.
  • the terminal 51 includes an image acquisition module 511, a SLAM module 512, a feature extraction module 513, a map generation module 514, and a communication module 515.
  • the data / programs of these functional modules can be stored in the above-mentioned memory 315, and These functional modules may be executed on the processor 311 described above. among them:
  • the image acquisition module 511 is used to acquire the first image
  • the SLAM module 512 is used to obtain first point cloud data of objects in the environment according to the first image; the first point cloud data indicates that the surface points of the objects in the environment are in the first camera coordinate system of the terminal 51 The three-dimensional coordinate value of; also used to obtain the posture data of the terminal 51 in the first camera coordinate system according to the first image;
  • the feature extraction module 513 is configured to obtain a two-dimensional feature image based on the pose data, the first point cloud data, and the first image;
  • a map generation module 514 configured to generate map data based on the first point cloud data, the pose data, and the two-dimensional feature image
  • the communication module 515 is configured to send the map data to the terminal 52.
  • the terminal 52 includes an image acquisition module 521, a SLAM module 522, a feature extraction module 523, a coordinate system transformation module 524, a communication module 525, and a content synchronization module 526.
  • the data / programs of these functional modules can be stored
  • the above-mentioned memory 315, and these functional modules can be run on the above-mentioned processor 311. among them:
  • the communication module 525 is configured to receive map data sent by the terminal 51, the map data is based on first point cloud data of objects in the environment, posture data of the terminal 51 in the first camera coordinate system of the terminal 51, and Generated by a two-dimensional feature image of the terminal 51; wherein the first point cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the terminal 51;
  • the image acquisition module 521 is used to acquire a second image
  • the SLAM module 522 is used to obtain the posture data of the terminal 52 in the second camera coordinate system of the terminal 52;
  • the feature extraction module 523 is used to obtain a two-dimensional feature image of the terminal 52;
  • the coordinate system transformation module 524 is configured to obtain the first camera coordinate system based on the map data, the posture data of the terminal 52 in the second camera coordinate system, and the two-dimensional feature image of the terminal 52 Coordinate system transformation data with the second camera coordinate system;
  • the content synchronization module 526 is configured to synchronize the content in the virtual scene according to the coordinate system transformation data.
  • the terminal 51 may be the first terminal in the embodiment of FIG. 3 or FIG. 6 above, and the terminal 52 may be the second terminal in the embodiment of FIG. 3 or FIG. 6 above. That is to say, in a specific implementation, the relevant modules in the terminal 51 and the relevant modules in the terminal 52 can be used to implement the first terminal, the first The relevant functional steps of the second terminal will not be repeated here.
  • FIG. 10 is a structural block diagram of yet another multi-user interaction system provided by an embodiment of the present invention.
  • the system includes a terminal 61, a terminal 62, and a server 63.
  • a communication connection can be performed between the terminal 61 and the server 63, and a communication connection can be performed between the terminal 62 and the server 63.
  • the terminal 61 includes an image acquisition module 611, a SLAM module 612, a feature extraction module 613, a map generation module 614, and a communication module 615.
  • the data / programs of these functional modules can be stored in the above-mentioned memory 315, and These functional modules may be executed on the processor 311 described above. among them:
  • the image acquisition module 611 is used to acquire the first image
  • the SLAM module 612 is used to obtain first point cloud data of objects in the environment according to the first image; the first point cloud data indicates that the surface points of the objects in the environment are in the first camera coordinate system of the terminal 61 The three-dimensional coordinate value of; also used to obtain the posture data of the terminal 61 in the first camera coordinate system according to the first image;
  • the feature extraction module 613 is used to obtain a two-dimensional feature image based on the pose data, the first point cloud data, and the first image;
  • a map generation module 614 configured to generate map data based on the first point cloud data, the pose data, and the two-dimensional feature image
  • the communication module 615 is configured to send the map data to the server 63.
  • the terminal 62 includes an image acquisition module 621, a SLAM module 622, a feature extraction module 623, a communication module 624, and a content synchronization module 625.
  • the data / programs of these functional modules can be stored in the above-mentioned memory 315, and These functional modules may be executed on the processor 311 described above. among them:
  • the image acquisition module 621 is used to acquire a second image
  • the SLAM module 622 is used to obtain the posture data of the terminal 62 in the second camera coordinate system of the terminal 62;
  • the feature extraction module 623 is used to obtain a two-dimensional feature image of the terminal 62;
  • the communication module 624 is configured to send a key frame of the terminal 62 to the server.
  • the key frame of the terminal 62 includes posture data of the terminal 62 in the second camera coordinate system of the terminal 62 and a two-dimensional feature image of the terminal 62.
  • the communication module 624 is also used to receive coordinate system transformation data sent by the server 63.
  • the content synchronization module 625 is configured to synchronize the content in the virtual scene according to the coordinate system transformation data.
  • the server includes a receiving module 631, a coordinate system transformation module 632 and a sending module 633.
  • the receiving module 631 and the sending module 633 can also be integrated into a communication module 634, that is, the receiving module 631 and the sending module 633 are used as communication Two sub-functions of module 634.
  • the data / programs of these functional modules can be stored in the aforementioned memory 401, and these functional modules can be executed on the aforementioned processor 403. among them:
  • the receiving module 631 is configured to receive map data sent by the terminal 61, the map data is based on first point cloud data of objects in the environment, posture data of the terminal 61 in the first camera coordinate system of the terminal 61, and Generated by a two-dimensional feature image of the terminal 61; wherein the first point cloud data represents the three-dimensional coordinate value of the surface point of the object in the first camera coordinate system of the terminal 61; also used for, Receiving the posture data of the terminal 62 in the second camera coordinate system of the terminal 62, and the two-dimensional feature image of the terminal 62;
  • the coordinate system transformation module 632 is used to perform a feature matching operation on the map data using the two-dimensional feature image of the terminal 62 to obtain posture data of the terminal 62 in the first camera coordinate system;
  • the sending module 633 is configured to send the coordinate system transformation data to the terminal 62.
  • the terminal 61 may be the first terminal in the embodiment of FIG. 6 above
  • the terminal 62 may be the second terminal in the embodiment of FIG. 6 above
  • the server 63 may be the above FIG. 6.
  • the server in the embodiment. That is to say, in a specific implementation, the relevant modules in the terminal 61, the relevant modules in the terminal 62, and the relevant modules in the server 63 can be used to implement the first of the data processing methods described in the embodiment of FIG. The relevant functional steps of the terminal, the second terminal, and the server will not be repeated here.
  • the feature extraction module, the map generation module, and the coordinate system transformation module proposed in the embodiment of FIG. 9 or FIG. 10 of the present invention can provide a unified external interface. Through the unified external interface, it can greatly reduce the number of different application scenarios. Multi-person sharing of AR application development work improves user experience.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices.
  • the computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be from a network site, computer, server or data center Transmission to another network site, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or may be a data storage device such as a server, a data center, or the like that includes one or more available medium integrations.
  • the usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD, etc.), or a semiconductor medium (such as a solid-state hard disk), or the like.
  • a magnetic medium such as a floppy disk, a hard disk, a magnetic tape, etc.
  • an optical medium such as a DVD, etc.
  • a semiconductor medium such as a solid-state hard disk

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Remote Sensing (AREA)
  • Data Mining & Analysis (AREA)
  • Processing Or Creating Images (AREA)

Abstract

用于虚拟场景的数据处理方法以及设备，该方法包括：第一终端通过相机采集第一图像，第一图像包括物体的图像；根据第一图像，获得所述物体的第一点云数据；第一点云数据表示所述物体的表面点在第一终端的第一相机坐标系中的三维坐标值；根据第一图像，获得第一终端在第一相机坐标系中的位姿数据；根据位姿数据、第一点云数据和第一图像，获得二维特征图像；根据第一点云数据、位姿数据和二维特征图像生成地图数据；向第二终端或服务器发送所述地图数据。该方法能够实现相同款或不同款的终端在任意自然场景中跨平台实现虚拟场景的坐标系同步，兼容各种空间定位引擎，满足用户需求。

Description

用于虚拟场景的数据处理方法以及设备 技术领域
本发明涉及虚拟场景技术领域，尤其涉及用于虚拟场景的数据处理方法以及设备。
背景技术
虚拟现实(Virtual Reality，VR)、增强现实(Augmented Reality，AR)和混合现实(Mixed Reality，MR)技术是近年来新兴的多媒体虚拟场景技术。其中，VR技术是一种可以创建和体验虚拟世界的仿真技术，AR技术是一种可以将虚拟现实和真实世界叠加并进行互动的技术。MR技术是通过合并真实世界和虚拟世界而产生的新的可视化环境，并且在真实世界、虚拟世界和用户之间搭起一个交互反馈的信息回路的综合技术。
在上述虚拟场景技术中，通常采用同时定位与建图(Simultaneous Localization and Mapping，SLAM)技术进行终端自身在环境中的定位。SLAM技术具体可实现终端(例如手机、VR眼镜等移动终端)在环境中从一个未知位置开始移动时,在移动过程中根据位置估计和地图进行自身定位,同时在自身定位的基础上建造增量式地图，以便于后续的定位，采用SLAM技术的系统或模块又可称为空间定位引擎。
在多用户交互的虚拟场景中，不同移动终端的类型或型号可能不同(如手机与VR眼镜、或者苹果手机与华为手机)，它们具有的空间定位引擎算法也各有差异。不同移动终端之间的坐标系同步(内容同步)是核心技术，通过坐标系同步可实现多用户VR或AR或MR应用中的虚拟内容在同一个坐标系空间共享和交互。
一种现有方案是基于规则几何形状平面(例如正方形桌面)实现坐标系同步。具体如下：移动终端利用计算当前所在位置水平桌面的中心点坐标，作为终端参考坐标系原点。然后利用桌面边缘信息计算参考坐标系的坐标轴。进而，建立终端参考坐标系。不同用户的移动终端计算过程皆如此，从而建立相同的参考坐标系，利用移动终端空间定位引擎输出的位姿，即可计算终端相对参考坐标系的位姿，进而计算出终端之间的相对位姿，从而实现坐标系同步。该方案可以实现不同款的移动终端在同一桌面上实现多用户VR或AR或MR应用交互。
由于这种方案依赖于规则几何形状平面，应用场景受到局限，在任意平面或非平面场景，此方案将无法实现坐标系同步。
发明内容
本发明实施例提供了用于虚拟场景的数据处理方法以及设备，能够实现不同款的终端在任意自然场景中跨平台实现虚拟场景的坐标系同步(内容同步)，兼容各种空间定位引擎，满足用户需求。
第一方面，本发明实施例提供了一种用于虚拟场景的数据处理方法，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。所述方法包括：第一终端通过相机采集第一图像，所述第一图像包括任意自然环境(真实世界)中的任意物体的图像；所述第一终端根据所述第一图像，获得所述物体的第一点云数据；所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；所述第一终 端根据所述第一图像，获得所述第一终端在所述第一相机坐标系中的位姿数据；所述第一终端根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；所述第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；所述第一终端向第二终端或服务器发送所述地图数据。
其中，当第一终端向第二终端发送所述地图数据时，所述第二终端可用于根据所述地图数据对虚拟场景中的内容进行同步操作；第一终端和第二终端所采用的空间定位引擎可以是不同的，也可以是相同的。
其中，当第一终端向服务器发送所述地图数据时，所述服务器用于根据所述地图数据，得到所述第一终端的第一相机坐标系和第二终端的第二相机坐标系之间的坐标系变换数据，并将所述坐标系变换数据发送至第二终端，以便于所述第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步操作。其中，第一终端和第二终端所采用的空间定位引擎可以是不同的，也可以是相同的。
可以看到，实施本发明实施例，第一终端皆能够利用本地的空间定位引擎的输出数据，进行图像特征提取，并生成地图数据发送给第二终端或服务器，以便于实现第二终端基于地图数据实现两终端间的坐标系同步。其中，第一终端和第二终端所采用的空间定位引擎可以是不同的。也就是说，即使不同终端安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，满足用户需求，提高用户使用体验。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意自然场景中的任意物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要依赖特定几何形状物体，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
基于第一方面，在可能的实施例中，所述第一终端根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像，包括：所述第一终端依据所述位姿数据将第一点云数据投影到所述第一图像，获得所述第一点云数据对应在所述第一图像的图像坐标系中的二维图像坐标(又称为2D图像坐标)，然后，根据二维图像坐标对应在所述第一图像中的图像块或者图像特征点描述子，提取出二维特征图像(又可称2D特征图像)。其中所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。
在一具体实施方式中，第一终端可隔一段预设时间选取一帧图像(第一图像)，以及空间定位引擎输出的同一时间戳的位姿数据和第一点云数据，将第一点云数据通过该位姿投影到该第一图像中，获取2D图像坐标，再对2D图像坐标进行2D特征描述，所述特征描述可以是图像特征点描述子或图像块描述，其中所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。这样，就可获得图像对应的同一时间戳的2D特征图像。本发明实施例中，可将第一终端的同一时间戳相关联的2D特征图像、第一点云数据、以及位姿数据的集合设置为第一终端的一个关键帧。
基于第一方面，在可能的实施例中，第一终端的空间定位引擎可根据相机采集的数据(第一图像)，计算获得第一终端在所述第一相机坐标系中的位姿数据。
在又一可能实施例中，第一终端中还设置有惯性测量单元(Inertial measurement  unit，IMU)传感器，那么空间定位引擎的输入信号包括来自相机采集的图像数据和IMU传感器采集的数据(可称为运动数据)，空间定位引擎可根据该图像数据和运动数据，计算获得第一终端在所述第一相机坐标系中的位姿数据。
在又一可能实施例中，第一终端中除了相机和IMU外，还设置有与位姿或运动相关的定位模块(GPS定位、北斗定位、WIFI定位、或基站定位等，具体可参考下文中关于终端结构的描述)，那么，那么空间定位引擎还可进一步参考定位模块所采集的数据来计算获得第一终端在所述第一相机坐标系中的位姿数据。
基于第一方面，在可能的实施例中，在所述第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据之前，所述方法还包括：所述第一终端获取定位数据，所述定位数据包括所述终端的GPS定位数据、所述第一终端的北斗定位数据、所述第一终端的WIFI定位数据、所述第一终端的蓝牙定位数据、或所述第一终端的基站定位数据中的至少一种。然后将该定位数据与上述地图数据一起打包成数据包以便于后续发送；或者，也可将定位数据携带于地图数据中，以便于后续发送。
第二方面，本发明实施例提供了又一种用于虚拟场景的数据处理方法，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。所述方法包括：第二终端接收第一终端发送的地图数据，所述地图数据是基于任意物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据；所述第二终端获取所述第二终端的二维特征图像；所述第二终端根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；所述第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步。
可以看到，实施本发明实施例，第二终端皆能够利用本地的空间定位引擎的输出数据，进行二维图像特征提取，接收第一终端发送的地图数据，并根据所述地图数据、所述第二终端的位姿数据和该二维特征图像，获得坐标系变换数据，最终基于坐标系变换数据实现两终端间的坐标系同步。其中，第一终端和第二终端所采用的空间定位引擎可以是不同的。也就是说，即使不同终端安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，满足用户需求，提高用户使用体验。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意自然场景中的任意物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要依赖特定几何形状物体，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
基于第二方面，在可能的实施例中，所述第二终端根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据，包括：所述第二终端使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一 相机坐标系中的位姿数据；所述第二终端根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
在一具体实施方式中，第二终端可隔一段预设时间选取一帧图像(第二图像)，以及空间定位引擎输出的同一时间戳的位姿数据和第二点云数据，将第二点云数据通过该位姿投影到该第二图像中，获取2D图像坐标，再对2D图像坐标进行2D特征描述，所述特征描述可以是图像特征点描述子或图像块描述，其中所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。这样，就可获得图像对应的同一时间戳的2D特征图像。本发明实施例中，可将第二终端的同一时间戳相关联的第二终端的2D特征图像、第二点云数据、以及第二终端的位姿数据的集合设置为第二终端的一个关键帧。
在一实现中，第二终端可根据自身所确定的关键帧以及所接收的地图数据进行特征匹配，分别提取出两者相同的图像特征，输出特征点匹配集合，进而估计第二终端相对于第一相机坐标系的位姿数据，其中位姿估计算法可以是PNP算法。然后，第二终端根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得第一相机坐标系与第二相机坐标系之间的坐标系变换数据(例如坐标系变换矩阵)。在具体的实现中，后续还可根据该坐标系变换矩阵计算出虚拟场景所采用锚点的姿态变换数据。
在可能的实施例中，所述第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步，包括：所述第二终端根据所述坐标系变换数据，获得所述虚拟场景的锚点的姿态变换数据；所述第二终端根据所述锚点的姿态变换数据，对所述虚拟场景中的内容进行同步。
基于第二方面，在可能的实施例中，所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据之前，包括：所述第二终端采集第二图像，所述第二图像包括所述物体的图像。由于第二图像是对环境进行拍摄而获得的，而第一终端和第二终端都处于该环境中，所以第二图像中有环境中该物体的图像，该物体即为第一终端采集的第一图像中所采用的物体；所述第二终端根据所述第二图像，获得所述物体的第二点云数据；所述第二点云数据表示所述物体的表面点在所述第二相机坐标系中的三维坐标值；
相应的，所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据，包括：所述第二终端根据所述第二图像，获得所述第二终端在所述第二相机坐标系中的位姿数据；
相应的，所述第二终端获取所述第二终端的二维特征图像，包括：所述第二终端根据所述第二终端在所述第二相机坐标系中的位姿数据，将所述第二点云数据投影到所述第二图像，以得到所述第二点云数据对应在所述第二图像的图像坐标系中的二维图像坐标；所述第二终端根据所述二维图像坐标对应在所述第二图像中的图像块或者图像特征点描述子，获得所述第二终端的二维特征图像。所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。
基于第二方面，在可能的实施例中，所述地图数据还携带有定位数据，所述定位数据包括所述第一终端的GPS定位数据、所述第一终端的北斗定位数据、所述第一终端的WIFI定位数据、所述第一终端的蓝牙定位数据、或所述第一终端的基站定位数据中的至少一种；相应的，所述第二终端获取所述第二终端的二维特征图像，包括：所述第二终端根据所述 定位数据，确定二维特征图像的搜索范围；根据所述搜索范围，从多个历史二维特征图像中确定所述第二终端的二维特征图像。
也就是说，如果第一终端向第二终端发送过来的数据包或地图数据中携带有第一种终端的定位数据，那么在特征匹配过程中，第二终端可先根据定位数据，确定二维特征图像的搜索范围，从而根据所述搜索范围对第二终端多次生成的多个关键帧进行筛选，例如选择出离定位数据较近或最近的关键帧，再使用该关键帧在地图数据中进行特征匹配，获得所述第二终端在第一终端的第一相机坐标系中的位姿数据。这样，有利于提高匹配效率和准确性。
第三方面，本发明实施例提供了一种用于虚拟场景的数据处理方法，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种，所述方法包括：服务器接收第一终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；所述服务器接收第二终端在所述第二终端的第二相机坐标系中的位姿数据，以及所述第二终端的二维特征图像；所述服务器使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一相机坐标系中的位姿数据；所述服务器根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；所述服务器将所述坐标系变换数据发送至所述第二终端。
可以看到，实施本发明实施例，不同移动终端皆能够利用各自空间定位引擎的输出数据，进行相同类型的图像特征提取，其中第一终端可进一步生成地图数据上传至服务器，第二终端可进一步生成关键帧上传至服务器，服务器进而根据关键帧及地图数据获得坐标系同步数据并下发至第二终端，实现终端间的坐标系同步。也就是说，即使不同移动安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，提高用户使用体验。同时，由于采用服务器作为不同终端之间的数据交互节点，所以能够高效率地满足大量终端之间的内容同步需求。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意场景中的物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要特定几何形状物体，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
第四方面，本发明实施例提供了一种用于虚拟场景的数据处理的终端，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。终端具体包括：图像获取模块，用于通过所述终端的相机采集第一图像，所述第一图像包括物体的图像；SLAM模块，用于根据所述第一图像，获得所述物体的第一点云数据；所述第一点云数据表示所述物体的表面点在所述终端的第一相机坐标系中的三维坐标值；所述SLAM模块还用于，根据所述第一图像，获得所述终端在所述第一相机坐标系中的位姿数据；特征提取模块，用于根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；地图生成 模块，用于根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；通信模块，用于向其他终端或服务器发送所述地图数据。
基于第四方面，在可能的实施例中，所述特征提取模块具体用于：根据所述位姿数据，将所述第一点云数据投影到所述第一图像，获得所述第一点云数据对应在所述第一图像的图像坐标系中的二维图像坐标；根据所述二维图像坐标对应在所述第一图像中的图像块或者图像特征点描述子，获得所述二维特征图像。
基于第四方面，在可能的实施例中，所述SLAM模块具体用于：根据所述第一图像以及惯性测量单元IMU采集的运动数据，获得所述终端在所述第一相机坐标系中的位姿数据。
基于第四方面，在可能的实施例中，所述终端还包括定位模块；所述定位模块用于，获取定位数据，所述定位数据包括所述终端的GPS定位数据、所述终端的北斗定位数据、所述终端的WIFI定位数据、所述终端的蓝牙定位数据、或所述终端的基站定位数据中的至少一种；相应的，所述地图生成模块所生成的地图数据还携带有所述定位数据。
具体实现中，该终端的各个功能模块可用于实现第一方面描述的方法。
第五方面，本发明实施例提供了一种用于虚拟场景的数据处理的终端，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。终端具体包括：通信模块，用于接收其他终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述其他终端在所述其他终端的第一相机坐标系中的位姿数据、以及所述其他终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述其他终端的第一相机坐标系中的三维坐标值；SLAM模块，用于获取所述终端在所述终端的第二相机坐标系中的位姿数据；所述SLAM模块还用于，获取所述终端的二维特征图像；坐标系变换模块，用于根据所述地图数据、所述终端在所述第二相机坐标系中的位姿数据和所述终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；内容同步模块，用于根据所述坐标系变换数据对虚拟场景中的内容进行同步。
基于第五方面，在可能的实施例中，所述坐标系变换模块具体用于：使用所述终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述终端在所述第一相机坐标系中的位姿数据；根据所述终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
基于第五方面，在可能的实施例中，所述终端还包括图像获取模块，所述图像获取模块用于，通过相机采集第二图像，所述第二图像包括所述物体的图像；所述SLAM模块还用于，根据所述第二图像，获得所述物体的第二点云数据；所述第二点云数据表示所述物体的表面点在所述第二相机坐标系中的三维坐标值；所述SLAM模块具体用于，根据所述第二图像，获得所述终端在所述第二相机坐标系中的位姿数据；所述SLAM模块还具体用于，根据所述终端在所述第二相机坐标系中的位姿数据，将所述第二点云数据投影到所述第二图像，以得到所述第二点云数据对应在所述第二图像的图像坐标系中的二维图像坐标；根据所述二维图像坐标对应在所述第二图像中的图像块或者图像特征点描述子，获得所述终端的二维特征图像。
基于第五方面，在可能的实施例中，所述内容同步模块具体用于：根据所述坐标系变换数据，获得所述虚拟场景的锚点的姿态变换数据；根据所述锚点的姿态变换数据，对所 述虚拟场景中的内容进行同步。
基于第五方面，在可能的实施例中，所述地图数据还携带有定位数据，所述定位数据包括所述其他终端的GPS定位数据、所述其他终端的北斗定位数据、所述其他终端的WIFI定位数据、所述其他终端的蓝牙定位数据、或所述其他终端的基站定位数据中的至少一种；所述SLAM模块具体用于，根据所述定位数据，获取所述终端的二维特征图像。
具体实现中，该终端的各个功能模块可用于实现第二方面描述的方法。
第六方面，本发明实施例提供了一种用于虚拟场景的数据处理的服务器，所述服务器包括：接收模块，用于接收第一终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；所述接收模块还用于，接收第二终端在所述第二终端的第二相机坐标系中的位姿数据，以及所述第二终端的二维特征图像；坐标系变换模块，用于使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一相机坐标系中的位姿数据；根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；发送模块，用于将所述坐标系变换数据发送至所述第二终端。
具体实现中，该服务器的各个功能模块可用于实现第三方面描述的方法。
第七方面，本发明实施例提供了又一种终端，终端包括：处理器以及耦合的存储器、发射器，该处理器可调用存储器存储的程序指令，实现第一方面描述的方法。
第八方面，本发明实施例提供了又一种终端，终端包括：处理器以及耦合的存储器、接收器，该处理器可调用存储器存储的程序指令，实现第二方面描述的方法。
第九方面，本发明实施例提供了又一种服务器，服务器包括：处理器以及耦合的存储器、通信接口，该处理器可调用存储器存储的程序指令，实现第三方面描述的方法。
第十方面，本发明实施例提供了一种非易失性计算机可读存储介质；所述计算机可读存储介质用于存储第一方面所述方法的实现代码。所述程序代码被计算设备执行时，所述计算设备用于第一方面所述方法。
第十一方面，本发明实施例提供了又一种非易失性计算机可读存储介质；所述计算机可读存储介质用于存储第二方面所述方法的实现代码。所述程序代码被计算设备执行时，所述用户设备用于第二方面所述方法。
第十二方面，本发明实施例提供了又一种非易失性计算机可读存储介质；所述计算机可读存储介质用于存储第三方面所述方法的实现代码。所述程序代码被计算设备执行时，所述用户设备用于第三方面所述方法。
第十三方面，本发明实施例提供了一种计算机程序产品；该计算机程序产品包括程序指令，当该计算机程序产品被计算设备执行时，该控制器执行前述第一方面所述方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第一方面的任一种可能的设计提供的方法的情况下，可以下载该计算机程序产品并在控制器上执行该计算机程序产品，以实现第一方面所述方法。
第十四方面，本发明实施例提供了又一种计算机程序产品。该计算机程序产品包括程序指令，当该计算机程序产品被用户设备执行时，该控制器执行前述第二方面的任一种可能的设计提供的方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第二方面的任一种可能的设计提供的方法的情况下，可以下载该计算机程序产品并在控制器上执行该计算机程序产品，以实现第二方面所述方法。
第十五方面，本发明实施例提供了又一种计算机程序产品。该计算机程序产品包括程序指令，当该计算机程序产品被用户设备执行时，该控制器执行前述第二方面的任一种可能的设计提供的方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第二方面的任一种可能的设计提供的方法的情况下，可以下载该计算机程序产品并在控制器上执行该计算机程序产品，以实现第三方面所述方法。
可以看到，实施本发明实施例，不同移动终端皆能够利用各自空间定位引擎的输出数据，进行相同类型的图像特征提取，第二终端根据自身生成的关键帧以及第一终端提供的地图数据实现两终端间的坐标系同步，或者第二终端根据服务器下发的坐标系变换数据实现两终端间的坐标系同步。也就是说，即使不同移动安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，提高用户使用体验。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意场景中的物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要依赖特定几何形状物体等已知信息，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
附图说明
为了更清楚地说明本发明实施例或背景技术中的技术方案，下面将对本发明实施例或背景技术中所需要使用的附图进行说明。
图1是本发明实施例提供的一种应用架构的示意图；
图2是本发明实施例提供的又一种应用架构的示意图；
图3是本发明实施例提供的一种用于虚拟场景的数据处理方法的流程示意图；
图4是本发明实施例提供的第一终端、第二终端分别生成地图和关键帧的示意图；
图5是本发明实施例提供的第二终端获得坐标系变换数据、锚点姿态变换数据的示意图；
图6是本发明实施例提供的又一种用于虚拟场景的数据处理方法的流程示意图；
图7是本发明实施例提供的终端的结构示意图；
图8是本发明实施例提供的一种服务器的结构示意图；
图9是本发明实施例提供的一种系统、以及第一终端与第二终端的结构示意图；
图10是本发明实施例提供的又一种系统，以及第一终端、第二终端和服务器的结构示意图。
具体实施方式
下面结合本发明实施例中的附图对本发明实施例进行描述。本发明的实施方式部分使用的术语仅用于对本发明的具体实施例进行解释，而非旨在限定本发明。
首先描述本发明实施例所涉及的两种应用架构。
参见图1，本发明实施例提供的第一种应用架构包括多个终端，终端可以为手机、平板电脑、笔记本电脑、录像机等移动终端设备，也可以是虚拟场景终端设备(如VR眼镜、MR互动设备等)。多个终端中的各个终端可以是相同款的终端(例如同型号的手机)，也可以是不同款的终端(例如多个终端包括不同型号的手机，或者多个终端包括移动终端设备与虚拟场景终端设备)，例如图1中示例性地示出了第一终端和第二终端，且第一终端与第二终端是不同型号的手机。
第一终端与第二终端可在真实世界的同一环境中运动，所述环境可以是任意自然环境。第一终端与第二终端分别运行虚拟场景应用(VR或AR或MR应用)，且第一终端与第二终端之间可进行通信连接。第一终端可通过相机和/或传感器采集环境中任意物体的信息(例如图示中采集环境中的物体1的图像，物体1可以是一个完整的物体，也可以是一个物体的部分特征)，进而向第二终端发送数据(如发送下文所描述的地图数据)。第二终端可根据第一终端发送的数据，实现第二终端所运动的虚拟场景内容与第一终端所运行的虚拟场景内容的同步。
参见图2，本发明实施例提供的第二种应用架构包括多个终端和服务器，其中终端可以为手机、平板电脑、笔记本电脑、录像机等移动终端设备，也可以是虚拟场景终端设备(如VR眼镜、MR互动设备等)。多个终端中的各个终端可以是相同款的终端(例如同型号的手机)，也可以是不同款的终端(例如多个终端包括不同型号的手机，或者多个终端包括移动终端设备与虚拟场景终端设备)，例如图2中示例性地示出了第一终端和第二终端，且第一终端与第二终端是不同型号的手机。服务器可以是一台或多台物理服务器(例如图2中示例性地示出了一台物理服务器)，也可以是计算机集群，还可以是云计算场景的虚拟机。
第一终端与第二终端可在真实世界的同一环境场景中运动。第一终端与第二终端可分别运行虚拟场景应用(VR或AR或MR应用)，且第一终端与服务器之间可进行通信连接，第二终端与服务器之间也可进行通信连接。第一终端可通过本地相机和/或传感器采集环境中任意物体的信息(例如图示中采集环境中的物体1的图像)，进而向服务器上传数据(如发送下文所描述的地图数据)。第二终端也可通过本地相机和/或传感器采集环境中与第一终端相同的物体的信息(例如图示中采集环境中的物体1的图像)，进而向服务器上传数据(如发送下文所描述的二维特征图像)。服务器综合上述数据进行计算获得内容同步相关数据(如下文描述的坐标系转换数据)，并将相关数据下发至第二终端，第二终端根据相关数据实现第二终端所运动的虚拟场景内容与第一终端所运行的虚拟场景内容的同步。
基于上述第一种应用架构，下面描述本发明实施例提供的应用于虚拟场景的数据处理方法，所述虚拟场景即为VR场景、AR场景、MR场景中的一种，请参见图3，图3是本发明实施例提供的一种数据处理方法的流程示意图，从第一终端侧和第二终端侧的角度进行描述。其中第一终端和第二终端可以是相同款的终端，也可以是不同款的终端。该方法包括但不限于如下步骤：
步骤101：第一终端通过所述第一终端的相机采集第一图像。
具体地，第一终端中设置有相机(或称摄像头，具体可参考下文中关于终端结构的描述)。所以第一终端可通过该相机对真实世界的环境进行拍摄，从而获得第一图像。
步骤102：第一终端根据所述第一图像，获得环境中物体的第一点云数据。
具体的，第一终端中配置有空间定位引擎(又可称SLAM模块或SLAM系统，具体可参考下文中关于终端结构的描述)。由于第一图像是对环境进行拍摄而获得的，所以第一图像中有环境中物体的图像，该物体可以是环境中具有任意形状的任意物体的全部或部分特征，即该物体不必是特定物体，也不必具有规则形状。
空间定位引擎可根据所述第一图像计算获得该物体的第一点云数据，其中，第一点云数据表示所述环境中物体的表面点在第一终端的第一相机坐标系中的三维坐标值(或称3D坐标值)。所述第一终端的第一相机坐标系即为第一终端的局部参考坐标系，第一相机坐标系是以光轴与图像平面的交点为图像坐标系的原点所构成的直角坐标系。在一具体实现中，第一相机坐标系的原点为第一终端的相机的光心，第一相机坐标系的x轴与y轴与图像坐标系的两个轴平行，第一相机坐标系的z轴可为相机光轴，其与图像平面垂直。
步骤103：第一终端根据所述第一图像，获得所述第一终端在所述第一相机坐标系中的位姿数据。
其中，位姿数据用于表示第一终端在第一相机坐标系中的位置和姿态(或方向)，位置可以通过三个坐标轴x、y、z表示，方向可以通过(α,β,γ)来表示，(α,β,γ)表示围绕三个坐标轴旋转的角度。
在一可能实施例中，第一终端的空间定位引擎可根据相机采集的数据(第一图像)，计算获得第一终端在所述第一相机坐标系中的位姿数据。
在又一可能实施例中，第一终端中还设置有惯性测量单元(Inertial measurement unit，IMU)传感器，那么空间定位引擎的输入信号包括来自相机采集的图像数据和IMU传感器采集的数据(可称为运动数据)，空间定位引擎可根据该图像数据和运动数据，计算获得第一终端在所述第一相机坐标系中的位姿数据。
在又一可能实施例中，第一终端中除了相机和IMU外，还设置有与位姿或运动相关的定位模块(GPS定位、北斗定位、WIFI定位、或基站定位等，具体可参考下文中关于终端结构的描述)，那么，那么空间定位引擎还可进一步参考定位模块所采集的数据来计算获得第一终端在所述第一相机坐标系中的位姿数据。
步骤104：第一终端根据同一时间戳对应的位姿数据、第一点云数据和第一图像，获得二维特征图像。
具体的，所述第一终端依据所述位姿数据将第一点云数据投影到所述第一图像，获得所述第一点云数据对应在所述第一图像的图像坐标系中的二维图像坐标(又称为2D图像坐标)，然后，根据二维图像坐标对应在所述第一图像中的图像块或者图像特征点描述子，提取出二维特征图像(又可称2D特征图像)。
在一具体实施方式中，如图4所示的第一终端中，第一终端可隔一段预设时间选取一帧图像(第一图像)，以及空间定位引擎输出的同一时间戳的位姿数据和第一点云数据，将第一点云数据通过该位姿投影到该第一图像中，获取2D图像坐标，再对2D图像坐标进行 2D特征描述，所述特征描述可以是图像特征点描述子或图像块描述，其中所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。这样，就可获得图像对应的同一时间戳的2D特征图像。本发明实施例中，可将第一终端的同一时间戳相关联的2D特征图像、第一点云数据、以及位姿数据的集合设置为第一终端的一个关键帧。
步骤105：第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据。
具体的，第一终端根据空间定位引擎输出第一点云数据和位姿数据、以及上述步骤104提取出的与上述第一点云数据关联的2D特征图像生成地图数据，打包成数据包以便于后续的外发。
本发明实施例中，如果可将同一时间戳相关联的2D特征图像、第一点云数据、以及位姿数据的集合设置为第一终端的一个关键帧，那么，如果第一终端每隔一段预设时间重复上文所描述的步骤101-步骤104，那么，在步骤105中，最终所打包的地图数据可以是第一终端所在局部区域的一个或多个关键帧的集合。
另外，在可能的实施例中，如果第一终端设置有定位模块(如GPS定位、北斗定位、WIFI定位、或基站定位等，具体可参考下文中关于终端结构的描述)，那么第一终端还可以通过定位模块获取生成地图数据时所述第一终端的定位数据，即定位数据包括GPS定位数据、北斗定位数据、WIFI定位数据、蓝牙定位数据、或基站定位数据中的至少一种。然后将该定位数据与上述地图数据一起打包成数据包(或将定位数据携带于地图数据中)，以便于后续的外发。
步骤106：所述第一终端发送地图数据至第二终端，相应的，第二终端接收所述地图数据。
步骤107：第二终端通过所述第二终端的相机采集第二图像。
具体地，第二终端中同样可设置有相机(或称摄像头，具体可参考下文中关于终端结构的描述)。所以第二终端可通过该相机对真实世界的环境进行拍摄，从而获得第二图像。
步骤108：所述第二终端根据所述第二图像，获得环境中所述物体的第二点云数据。所述第二点云数据表示所述物体的表面点在第二终端的第二相机坐标系中的三维坐标值。
具体的，第二终端中同样配置有空间定位引擎(又可称SLAM模块或SLAM系统，具体可参考下文中关于终端结构的描述)，并且第二终端的空间定位引擎可以不同于第一终端的空间定位引擎，也可以与第一终端的空间定位引擎相同。由于第二图像是对环境进行拍摄而获得的，所以第二图像中有环境中物体的图像，该物体即为第一终端在步骤102所采用的物体。
同理，该空间定位引擎可根据所述第二图像计算获得该物体的第二点云数据，其中，第二点云数据表示所述环境中物体的表面点在第二终端的第二相机坐标系中的三维坐标值(或称3D坐标)。所述第二终端的第二相机坐标系即为第二终端的局部参考坐标系，第二相机坐标系是以光轴与图像平面的交点为图像坐标系的原点所构成的直角坐标系。在一具体实现中，第二相机坐标系的原点为第二终端的相机的光心，第二相机坐标系的x轴与y轴与图像坐标系的两个轴平行，第二相机坐标系的z轴可为相机光轴，其与图像平面垂直。
步骤109：所述第二终端根据所述第二图像，获得所述第二终端在所述第二相机坐标 系中的位姿数据。
同理，该位姿数据用于表示第二终端在第二相机坐标系中的位置和姿态(或方向)，位置可以通过三个坐标轴x、y、z表示，方向可以通过(α,β,γ)来表示，(α,β,γ)表示围绕三个坐标轴旋转的角度。
在一可能实施例中，第二终端的空间定位引擎可根据相机采集的数据(第二图像)，计算获得第二终端在所述第二相机坐标系中的位姿数据。
在又一可能实施例中，第二终端中还设置有IMU传感器，那么空间定位引擎的输入信号包括来自相机采集的图像数据和IMU传感器采集的数据(可称为运动数据)，空间定位引擎可根据该图像数据和运动数据，计算获得第二终端在所述第二相机坐标系中的位姿数据。
在又一可能实施例中，第二终端中除了相机和IMU外，还设置有与位姿或运动相关的定位模块(GPS定位、北斗定位、WIFI定位、或基站定位等，具体可参考下文中关于终端结构的描述)，那么，那么该空间定位引擎还可进一步参考定位模块所采集的数据来计算获得第二终端在所述第二相机坐标系中的位姿数据。
步骤110：所述第二终端根据所述位姿数据、所述第二点云数据和所述第二图像，获得二维特征图像。
同理，第二终端依据所述第二终端在所述第二相机坐标系中的位姿数据，将所述第二点云数据投影到所述第二图像，获得所述第二点云数据对应在所述第二图像的图像坐标系中的二维图像坐标(又称为2D图像坐标)，然后根据所述二维图像坐标对应在所述第二图像中的图像块或者图像特征点描述子，获得所述第二终端的二维特征图像(又可称2D特征图像)。
在一具体实施方式中，如图4所示的第二终端中，第二终端可隔一段预设时间选取一帧图像(第二图像)，以及空间定位引擎输出的同一时间戳的位姿数据和第二点云数据，将第二点云数据通过该位姿投影到该第二图像中，获取2D图像坐标，再对2D图像坐标进行2D特征描述，所述特征描述可以是图像特征点描述子或图像块描述，其中所述图像特征点描述子包括但不限于Brief、Brisk、ORB、Freak、SIFT、SURF中的一种。这样，就可获得图像对应的同一时间戳的2D特征图像。本发明实施例中，可将第二终端的同一时间戳相关联的第二终端的2D特征图像、第二点云数据、以及第二终端的位姿数据的集合设置为第二终端的一个关键帧。
需要说明的是，上述步骤107-步骤110与步骤106之间没有必然的先后顺序，也就是步骤106可能在步骤107之前实施，步骤106也可能在步骤107-步骤110之间实施，步骤106也可能在步骤110之后实施，本发明不作具体限定。
步骤111：所述第二终端根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
具体地，第二终端可使用第二终端的关键帧，在第一终端所发送过来的地图数据中进行特征匹配，从而获得所述第二终端在第一终端的第一相机坐标系(这里也可称为地图坐标系)中的位姿数据；然后，第二终端根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐 标系之间的坐标系变换数据。
在一实现中，如图5所示，第二终端可根据自身所确定的关键帧以及所接收的地图数据进行特征匹配，分别提取出两者相同的图像特征，输出特征点匹配集合，进而估计第二终端相对于第一相机坐标系的位姿数据，其中位姿估计算法可以是PNP算法。然后，第二终端根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得第一相机坐标系与第二相机坐标系之间的坐标系变换数据(例如坐标系变换矩阵)。在具体的实现中，后续还可根据该坐标系变换矩阵计算出虚拟场景所采用锚点的姿态变换数据。
需要说明的是，在可能的实施例中，如果第一终端向第二终端发送过来的数据包或地图数据中携带有第一种终端的定位数据，那么在特征匹配过程中，第二终端可先根据定位数据，确定二维特征图像的搜索范围，从而根据所述搜索范围对通过步骤110多次生成的多个关键帧进行筛选，例如选择出离定位数据较近或最近的关键帧，再使用该关键帧在地图数据中进行特征匹配，获得所述第二终端在第一终端的第一相机坐标系中的位姿数据。这样，有利于提高匹配效率和准确性。
步骤112：所述第二终端根据所述坐标系变换数据对虚拟场景中的内容(如虚拟物体)进行同步操作。
具体的，在根据坐标系变换数据获得所述虚拟场景的锚点的姿态变换数据后，第二终端可根据所述锚点的姿态变换数据，实现虚拟场景中的虚拟物体姿态同步，也就是说通过同步操作能够使得第二终端正确显示第一终端所共享过来的虚拟物体的位姿。
可以看到，实施本发明实施例，不同移动终端皆能够利用各自空间定位引擎的输出数据，进行相同类型的图像特征提取，第二终端进而根据自身生成的关键帧以及第一终端提供的地图数据实现两终端间的坐标系同步。也就是说，即使不同移动安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，提高用户使用体验。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意场景中的物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要依赖Marker图像、特定几何形状物体等已知信息，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
基于上述第二种应用架构，下面描述本发明实施例提供的应用于虚拟场景的又一种数据处理方法，所述虚拟场景即为VR场景、AR场景、MR场景中的一种，请参见图6，图6是本发明实施例提供的又一种数据处理方法的流程示意图，从第一终端侧、服务器侧和第二终端侧的角度进行描述。其中第一终端和第二终端可以是相同款的终端，也可以是不同款的终端。该方法包括但不限于如下步骤：
步骤201：第一终端通过所述第一终端的相机采集第一图像。具体实现可参考图3实施例步骤101的描述，这里不再赘述。
步骤202：所述第一终端根据所述第一图像，获得环境中物体的第一点云数据。具体 实现可参考图3实施例步骤102的描述，这里不再赘述。
步骤203：所述第一终端根据所述第一图像，获得所述第一终端在所述第一相机坐标系中的位姿数据。具体实现可参考图3实施例步骤103的描述，这里不再赘述。
步骤204：所述第一终端根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像。具体实现可参考图3实施例步骤104的描述，这里不再赘述。
步骤205：所述第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据。具体实现可参考图3实施例步骤105的描述，这里不再赘述。
步骤206：所述第一终端发送地图数据至服务器，相应的，服务器接收所述地图数据。
步骤207：第二终端通过所述第二终端的相机采集第二图像。具体实现可参考图3实施例步骤107的描述，这里不再赘述。
步骤208：所述第二终端根据所述第二图像，获得环境中所述物体的第二点云数据。具体实现可参考图3实施例步骤108的描述，这里不再赘述。
步骤209：所述第二终端根据所述第二图像，获得所述第二终端在所述第二相机坐标系中的位姿数据。具体实现可参考图3实施例步骤109的描述，这里不再赘述。
步骤210：所述第二终端根据所述位姿数据、所述第二点云数据和所述第二图像，获得二维特征图像。具体实现可参考图3实施例步骤110的描述，这里不再赘述。
步骤211：所述第二终端发送该二维特征图像、第二终端的位姿数据至服务器。
具体实施例中，第二终端可将同一时间戳相关联的第二终端的2D特征图像、第二点云数据、以及第二终端在第二相机坐标系中的位姿数据的集合设置为第二终端的一个关键帧。那么，第二终端也可直接将该关键帧发送至服务器。
步骤212：服务器根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得坐标系变换数据。
具体实施例中，服务器可使用第二终端的关键帧，在第一终端所发送过来的地图数据中进行特征匹配，从而获得所述第二终端在第一终端的第一相机坐标系(这里也可称为地图坐标系)中的位姿数据；然后，服务器根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
另外，在可能的实施例中，后续服务器还可根据该坐标系变换数据计算出虚拟场景所采用锚点的姿态变换数据。
需要说明的是，本步骤212的具体实现过程可参考图3实施例步骤111的描述，这里不再赘述。
步骤213：服务器发送该坐标系变换数据至第二终端。需要说明的是，如果上述步骤212中，服务器已根据该坐标系变换数据计算出虚拟场景所采用锚点的姿态变换数据，那么服务器也可发送锚点的姿态变换数据至第二终端。
步骤214：第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步操作。具体实现过程可参考图3实施例步骤112的描述，这里不再赘
可以看到，实施本发明实施例，不同移动终端皆能够利用各自空间定位引擎的输出数据，进行相同类型的图像特征提取，其中第一终端可进一步生成地图数据上传至服务器，第二 终端可进一步生成关键帧上传至服务器，服务器进而根据关键帧及地图数据获得坐标系同步数据并下发至第二终端，实现终端间的坐标系同步。也就是说，即使不同移动安装不同空间定位引擎，依然可实现虚拟场景中的内容同步，这样有利于实现不同空间定位引擎之间的跨平台的多人共享AR/VR/MR开发和应用，提高用户使用体验。同时，由于采用服务器作为不同终端之间的数据交互节点，所以能够高效率地满足大量终端之间的内容同步需求。
还可以看到，本发明实施例通过有效的特征提取方式，能够对任意场景中的物体进行特征提取，进而实现不同终端之间的坐标系同步，所以多人共享AR应用场景不受环境的限制，即在实现多用户虚拟场景交互的过程中，不再需要依赖Marker图像、特定几何形状物体等已知信息，所以，实现多人共享AR/VR/MR应用不再受环境影响，满足了各种应用场景的需求。
上文详细描述了本发明实施例提供的相关方法，基于相同的发明构思，下面进一步描述本发明实施例的相关设备。
参见图7，图7是本发明实施例提供的一种终端的实现方式的结构框图。如图7所示，终端300可包括：芯片310、存储器315(一个或多个计算机可读存储介质)、外围系统317、定位模块331以及通信模块332。这些部件可在一个或多个通信总线314上通信。
外围系统317主要用于实现终端300和用户/外部环境之间的交互功能，具体实现中，外围系统317可包括：触摸屏控制器318、相机控制器319以及传感器管理模块320中的若干个部件。各个控制器可与各自对应的外围设备如触摸屏323、相机324以及传感器325等耦合。其中，在一些实施例中，触摸屏323可以通过耦合触控感应元件和显示屏而获得，触控感应元件可设置于显示屏下方，触控感应元件可用于检测用户通过显示屏输入触控操作时作用于显示屏上的触控压力(如用户的触摸、滑动操作)，显示屏用于进行内容显示(如显示虚拟场景应用的虚拟物体)；在一些实施例中，相机324可以是单目摄像头或双目摄像头或深度摄像头；在一些实施例中，传感器325可包括惯性测量单元IMU，此外还可能包括其他传感器(例如，近距感应传感器)。需要说明的，外围系统317还可以包括其他I/O外设。
芯片310可集成包括：一个或多个处理器311、时钟模块312以及可能的电源管理模块313。集成于芯片310中的时钟模块312主要用于为处理器311产生数据传输和时序控制所需要的时钟。集成于基带芯片310中的电源管理模块313主要用于为处理器311以及外围系统提供稳定的、高精确度的电压。
存储器315与处理器311耦合，用于存储数据(例如图像数据、点云数据、地图数据、关键帧数据、坐标系转换数据等)、各种软件程序和/或多组程序指令。具体实现中，存储器315可包括高速随机存取的存储器，并且也可包括非易失性存储器，例如一个或多个磁盘存储设备、闪存设备或其他非易失性固态存储设备。存储器315还可以存储一个或多个应用程序。如图示中这些应用程序可包括：AR/VR/MR等虚拟场景应用程序、地图类应用程序、图像管理类应用程序等等。存储器115还可以用于存储操作系统，例如ANDROID，IOS，WINDOWS，或者LINUX等嵌入式操作系统。存储器115还可以存储网络通信程序，该网络通信程序可用于与一个或多个附加设备，一个或多个终端设备，一个或多个网络设备 进行通信。存储器115还可以存储用户界面程序，该用户界面程序可以通过图形化的操作界面将应用程序的内容(如虚拟场景中的虚拟物体)形象逼真的显示出来，并通过菜单、对话框以及按键等输入控件接收用户对应用程序的控制操作。
定位模块331用于实现对终端300的物理定位，通过定位模块331可采集终端300的运动数据。定位模块331例如可包括WIFI定位模块、蓝牙定位模块、基站定位模块、卫星定位模块(如GPS定位模块和/或北斗定位模块)中的一个或多个。
通信模块332集成了发射器和接收器，用于接收和发送射频信号，以便于终端300与其他终端建立通信连接，或者终端300与服务器建立通信连接。通信模块332主要集成了终端300的接收器和发射器。具体实现中，通信模块332可包括但不限于：天线系统、RF收发器、一个或多个放大器、调谐器、一个或多个振荡器、数字信号处理器、CODEC芯片、SIM卡和存储介质等。在一些实施例中，还可在单独的芯片上实现通信模块332。
应当理解，上述终端300仅为本发明实施例提供的一个例子，并且，终端300可具有比示出的部件更多或更少的部件，可以组合两个或更多个部件，或者可具有部件的不同配置实现。
本发明具体实施例中，上述终端300可以是上述图3或图6实施例中第一终端和/或第二终端。具体实现中，相机324可用于采集环境中物体的图像，处理器311可用于调用存储器315中的程序指令，实现本发明实施例上文图3或图6实施例描述的数据处理方法中第一终端或第二终端的功能，故终端300的各个部件的功能实现可参考上文的描述，这里不再赘述。
参见图8，图8是本发明实施例提供的一种服务器的实现方式的结构框图。如图8所示，服务器400包括处理器403、存储器401(一个或多个计算机可读存储介质)、通信接口402。这些部件可在一个或多个通信总线404上通信。其中：
处理器403可以是一个或多个中央处理器(Central Processing Unit，CPU)，在处理器403是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
存储器401与处理器403耦合，用于存储各种软件程序和/或多组指令、以及数据(例如地图数据、关键帧数据、坐标系转换数据等)。具体实现中，存储器401包括但不限于是随机存储记忆体(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦除可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、或便携式只读存储器(Compact Disc Read-Only Memory，CD-ROM)，具体的，该存储器401还可用于存储用于计算坐标系转换数据的算法程序。
通信接口402主要集成了接收器和发射器，其中接收器用于接收终端发送的数据，发射器用于向终端发送数据。具体实现中，发射器可用于向终端下发坐标系转换数据或锚点的姿态转换数据，接收器可用于接收终端上传的地图数据或关键帧数据。
应当理解，上述服务器400仅为本发明实施例提供的一个例子，具体实现中，服务器400可具有比图示更多的部件。
本发明具体实施例中，上述服务器400可以是上述图6实施例中服务器。具体实现中，处理器403可用于调用存储器401中的程序指令，实现本发明实施例上文图6实施例描述 的数据处理方法中服务器的功能，故服务器400的各个部件的功能实现可参考上文的描述，这里不再赘述。
参见图9，图9是本发明实施例提供的一种多用户交互系统的结构框图。该系统包括终端51和终端52，终端51和终端52之间可通过各自的通信模块实现通信连接。
其中，终端51包括图像获取模块511、SLAM模块512、特征提取模块513、地图生成模块514、通信模块515，在一具体实现中，这些功能模块的数据/程序可被存储于上述存储器315，且这些功能模块可被运行于上述处理器311。其中：
图像获取模块511，用于获取第一图像；
SLAM模块512，用于根据所述第一图像，获得环境中物体的第一点云数据；所述第一点云数据表示所述环境中物体的表面点在终端51的第一相机坐标系中的三维坐标值；还用于，根据所述第一图像，获得终端51在所述第一相机坐标系中的位姿数据；
特征提取模块513，用于根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；
地图生成模块514，用于根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；
通信模块515，用于将所述地图数据发送至终端52。
其中，终端52包括图像获取模块521、SLAM模块522、特征提取模块523、坐标系变换模块524、通信模块525、内容同步模块526，在一具体实现中，这些功能模块的数据/程序可被存储于上述存储器315，且这些功能模块可被运行于上述处理器311。其中：
通信模块525，用于接收终端51发送的地图数据，所述地图数据是基于环境中物体的第一点云数据、终端51在所述终端51的第一相机坐标系中的位姿数据、以及所述终端51的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述终端51的第一相机坐标系中的三维坐标值；
图像获取模块521，用于获取第二图像；
SLAM模块522，用于获取终端52在所述终端52的第二相机坐标系中的位姿数据；
特征提取模块523，用于获取所述终端52的二维特征图像；
坐标系变换模块524，用于根据所述地图数据、所述终端52在所述第二相机坐标系中的位姿数据和所述终端52的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
内容同步模块526，用于根据所述坐标系变换数据对虚拟场景中的内容进行同步操作。
需要说明的是，本发明具体实施例中，上述终端51可以是上述图3或图6实施例中第一终端，上述终端52可以是上述图3或图6实施例中第二终端。也就是说，具体实现中，终端51中的相关模块和终端52中的相关模块可分别用于实现本发明实施例上文图3或图6实施例描述的数据处理方法中第一终端、第二终端的相关功能步骤，这里不再赘述。
参见图10，图10是本发明实施例提供的又一种多用户交互系统的结构框图。该系统包括终端61、终端62和服务器63，终端61和服务器63之间可进行通信连接，终端62和 服务器63之间可进行通信连接。
其中，终端61包括图像获取模块611、SLAM模块612、特征提取模块613、地图生成模块614、通信模块615，在一具体实现中，这些功能模块的数据/程序可被存储于上述存储器315，且这些功能模块可被运行于上述处理器311。其中：
图像获取模块611，用于获取第一图像；
SLAM模块612，用于根据所述第一图像，获得环境中物体的第一点云数据；所述第一点云数据表示所述环境中物体的表面点在终端61的第一相机坐标系中的三维坐标值；还用于，根据所述第一图像，获得终端61在所述第一相机坐标系中的位姿数据；
特征提取模块613，用于根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；
地图生成模块614，用于根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；
通信模块615，用于将所述地图数据发送至服务器63。
其中，终端62包括图像获取模块621、SLAM模块622、特征提取模块623、通信模块624、内容同步模块625，在一具体实现中，这些功能模块的数据/程序可被存储于上述存储器315，且这些功能模块可被运行于上述处理器311。其中：
图像获取模块621，用于获取第二图像；
SLAM模块622，用于获取终端62在所述终端62的第二相机坐标系中的位姿数据；
特征提取模块623，用于获取所述终端62的二维特征图像；
通信模块624，用于向服务器发送终端62的关键帧，终端62的关键帧包括终端62在所述终端62的第二相机坐标系中的位姿数据以及所述终端62的二维特征图像。
通信模块624还用于，接收服务器63发送的坐标系变换数据。
内容同步模块625，用于根据所述坐标系变换数据对虚拟场景中的内容进行同步操作。
其中，服务器包括接收模块631、坐标系变换模块632和发送模块633，在具体实现中，还可将接收模块631和发送模块633整合成通信模块634，即接收模块631、发送模块633分别作为通信模块634的两个子功能。在一具体实现中，这些功能模块的数据/程序可存储于上述存储器401，且这些功能模块可被运行于上述处理器403。其中：
接收模块631，用于接收终端61发送的地图数据，所述地图数据是基于环境中物体的第一点云数据、终端61在所述终端61的第一相机坐标系中的位姿数据、以及所述终端61的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述终端61的第一相机坐标系中的三维坐标值；还用于，接收终端62在所述终端62的第二相机坐标系中的位姿数据，以及所述终端62的二维特征图像；
坐标系变换模块632，用于使用所述终端62的二维特征图像在所述地图数据中进行特征匹配操作，获得所述终端62在所述第一相机坐标系中的位姿数据；根据所述终端62在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
发送模块633，用于将所述坐标系变换数据发送至所述终端62。
需要说明的是，本发明具体实施例中，上述终端61可以是上述图6实施例中第一终端， 上述终端62可以是上述图6实施例中第二终端，上述服务器63可以是上述图6实施例中的服务器。也就是说，具体实现中，终端61中的相关模块、终端62中的相关模块、服务器63的相关模块可分别用于实现本发明实施例上文图6实施例描述的数据处理方法中第一终端、第二终端、服务器的相关功能步骤，这里不再赘述。
需要说明的是，本发明上述图9或图10实施例提出的特征提取模块、地图生成模块和坐标系变换模块，可提供统一对外接口，通过统一的对外接口，能够大量减少针对不同应用场景的多人共享AR应用的开发工作，提升用户使用体验。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者任意组合来实现。当使用软件实现时，可以全部或者部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令，在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络或其他可编程装置。所述计算机指令可存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网络站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、微波等)方式向另一个网络站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质，也可以是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如软盘、硬盘、磁带等)、光介质(例如DVD等)、或者半导体介质(例如固态硬盘)等等。
在上述实施例中，对各个实施例的描述各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。

Claims (24)

  1. 一种用于虚拟场景的数据处理方法，其特征在于，所述方法包括：
    第一终端通过相机采集第一图像，所述第一图像包括物体的图像；
    所述第一终端根据所述第一图像，获得所述物体的第一点云数据；所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；
    所述第一终端根据所述第一图像，获得所述第一终端在所述第一相机坐标系中的位姿数据；
    所述第一终端根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；
    所述第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；
    所述第一终端向第二终端或服务器发送所述地图数据。
  2. 根据权利要求1所述的方法，其特征在于，所述第一终端根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像，包括：
    所述第一终端根据所述位姿数据，将所述第一点云数据投影到所述第一图像，以得到所述第一点云数据对应在所述第一图像的图像坐标系中的二维图像坐标；
    所述第一终端根据所述二维图像坐标对应在所述第一图像中的图像块或者图像特征点描述子，获得所述二维特征图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一终端根据所述第一图像，获得所述第一终端在所述第一相机坐标系中的位姿数据，具体包括：
    所述第一终端根据所述第一图像以及惯性测量单元IMU采集的运动数据，获得所述第一终端在所述第一相机坐标系中的位姿数据。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，在所述第一终端根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据之前，所述方法还包括：
    所述第一终端获取定位数据，所述定位数据包括所述终端的GPS定位数据、所述第一终端的北斗定位数据、所述第一终端的WIFI定位数据、所述第一终端的蓝牙定位数据、或所述第一终端的基站定位数据中的至少一种；
    相应的，所述地图数据还携带有所述定位数据。
  5. 根据权利要求1-4所述的方法，其特征在于，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。
  6. 一种用于虚拟场景的数据处理方法，其特征在于，所述方法包括：
    第二终端接收第一终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维 特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；
    所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据；
    所述第二终端获取所述第二终端的二维特征图像；
    所述第二终端根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
    所述第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步。
  7. 根据权利要求6所述的方法，其特征在于，所述第二终端根据所述地图数据、所述第二终端在所述第二相机坐标系中的位姿数据和所述第二终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据，包括：
    所述第二终端使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一相机坐标系中的位姿数据；
    所述第二终端根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
  8. 根据权利要求6或7所述的方法，其特征在于，所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据之前，包括：
    所述第二终端采集第二图像，所述第二图像包括所述物体的图像；以及，
    所述第二终端根据所述第二图像，获得所述物体的第二点云数据；所述第二点云数据表示所述物体的表面点在所述第二相机坐标系中的三维坐标值；
    相应的，所述第二终端获取所述第二终端在所述第二终端的第二相机坐标系中的位姿数据，包括：所述第二终端根据所述第二图像，获得所述第二终端在所述第二相机坐标系中的位姿数据；
    相应的，所述第二终端获取所述第二终端的二维特征图像，包括：所述第二终端根据所述第二终端在所述第二相机坐标系中的位姿数据，将所述第二点云数据投影到所述第二图像，以得到所述第二点云数据对应在所述第二图像的图像坐标系中的二维图像坐标；所述第二终端根据所述二维图像坐标对应在所述第二图像中的图像块或者图像特征点描述子，获得所述第二终端的二维特征图像。
  9. 根据权利要求6-8任一项所述的方法，其特征在于，所述第二终端根据所述坐标系变换数据对虚拟场景中的内容进行同步，包括：
    所述第二终端根据所述坐标系变换数据，获得所述虚拟场景的锚点的姿态变换数据；
    所述第二终端根据所述锚点的姿态变换数据，对所述虚拟场景中的内容进行同步。
  10. 根据权利要求6-9任一项所述的方法，其特征在于，所述地图数据还携带有定位 数据，所述定位数据包括所述第一终端的GPS定位数据、所述第一终端的北斗定位数据、所述第一终端的WIFI定位数据、所述第一终端的蓝牙定位数据、或所述第一终端的基站定位数据中的至少一种；
    相应的，所述第二终端获取所述第二终端的二维特征图像，包括：
    所述第二终端根据所述定位数据，确定二维特征图像的搜索范围；
    根据所述搜索范围，从多个历史二维特征图像中确定所述第二终端的二维特征图像。
  11. 根据权利要求6-10任一项所述的方法，其特征在于，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。
  12. 一种用于虚拟场景的数据处理方法，其特征在于，所述方法包括：
    服务器接收第一终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；
    所述服务器接收第二终端在所述第二终端的第二相机坐标系中的位姿数据，以及所述第二终端的二维特征图像；
    所述服务器使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一相机坐标系中的位姿数据；
    所述服务器根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
    所述服务器将所述坐标系变换数据发送至所述第二终端。
  13. 一种用于虚拟场景的数据处理的终端，其特征在于，具体包括：
    图像获取模块，用于通过所述终端的相机采集第一图像，所述第一图像包括物体的图像；
    SLAM模块，用于根据所述第一图像，获得所述物体的第一点云数据；所述第一点云数据表示所述物体的表面点在所述终端的第一相机坐标系中的三维坐标值；
    所述SLAM模块还用于，根据所述第一图像，获得所述终端在所述第一相机坐标系中的位姿数据；
    特征提取模块，用于根据所述位姿数据、所述第一点云数据和所述第一图像，获得二维特征图像；
    地图生成模块，用于根据所述第一点云数据、所述位姿数据和所述二维特征图像生成地图数据；
    通信模块，用于向其他终端或服务器发送所述地图数据。
  14. 根据权利要求13所述的终端，其特征在于，所述特征提取模块具体用于：
    根据所述位姿数据，将所述第一点云数据投影到所述第一图像，获得所述第一点云数据对应在所述第一图像的图像坐标系中的二维图像坐标；
    根据所述二维图像坐标对应在所述第一图像中的图像块或者图像特征点描述子，获得所述二维特征图像。
  15. 根据权利要求13或14所述的终端，其特征在于，所述SLAM模块具体用于：
    根据所述第一图像以及惯性测量单元IMU采集的运动数据，获得所述终端在所述第一相机坐标系中的位姿数据。
  16. 根据权利要求13-15任一项所述的终端，其特征在于，所述终端还包括定位模块；所述定位模块用于，获取定位数据，所述定位数据包括所述终端的GPS定位数据、所述终端的北斗定位数据、所述终端的WIFI定位数据、所述终端的蓝牙定位数据、或所述终端的基站定位数据中的至少一种；
    相应的，所述地图生成模块所生成的地图数据还携带有所述定位数据。
  17. 根据权利要求13-16所述的终端，其特征在于，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。
  18. 一种用于虚拟场景的数据处理的终端，其特征在于，具体包括：
    通信模块，用于接收其他终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述其他终端在所述其他终端的第一相机坐标系中的位姿数据、以及所述其他终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述其他终端的第一相机坐标系中的三维坐标值；
    SLAM模块，用于获取所述终端在所述终端的第二相机坐标系中的位姿数据；
    所述SLAM模块还用于，获取所述终端的二维特征图像；
    坐标系变换模块，用于根据所述地图数据、所述终端在所述第二相机坐标系中的位姿数据和所述终端的二维特征图像，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
    内容同步模块，用于根据所述坐标系变换数据对虚拟场景中的内容进行同步。
  19. 根据权利要求18所述的终端，其特征在于，所述坐标系变换模块具体用于：
    使用所述终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述终端在所述第一相机坐标系中的位姿数据；
    根据所述终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据。
  20. 根据权利要求18或19所述的终端，其特征在于，所述终端还包括图像获取模块，所述图像获取模块用于，通过相机采集第二图像，所述第二图像包括所述物体的图像；
    所述SLAM模块还用于，根据所述第二图像，获得所述物体的第二点云数据；所述第二点云数据表示所述物体的表面点在所述第二相机坐标系中的三维坐标值；
    所述SLAM模块具体用于，根据所述第二图像，获得所述终端在所述第二相机坐标系中的位姿数据；
    所述SLAM模块还具体用于，根据所述终端在所述第二相机坐标系中的位姿数据，将所述第二点云数据投影到所述第二图像，以得到所述第二点云数据对应在所述第二图像的图像坐标系中的二维图像坐标；根据所述二维图像坐标对应在所述第二图像中的图像块或者图像特征点描述子，获得所述终端的二维特征图像。
  21. 根据权利要求18-20任一项所述的终端，其特征在于，所述内容同步模块具体用于：
    根据所述坐标系变换数据，获得所述虚拟场景的锚点的姿态变换数据；
    根据所述锚点的姿态变换数据，对所述虚拟场景中的内容进行同步。
  22. 根据权利要求18-21任一项所述的终端，其特征在于，所述地图数据还携带有定位数据，所述定位数据包括所述其他终端的GPS定位数据、所述其他终端的北斗定位数据、所述其他终端的WIFI定位数据、所述其他终端的蓝牙定位数据、或所述其他终端的基站定位数据中的至少一种；
    所述SLAM模块具体用于，根据所述定位数据，获取所述终端的二维特征图像。
  23. 根据权利要求18-22任一项所述的终端，其特征在于，所述虚拟场景为虚拟现实VR场景、增强现实AR场景、混合现实MR场景中的一种。
  24. 一种用于虚拟场景的数据处理的服务器，其特征在于，所述服务器包括：
    接收模块，用于接收第一终端发送的地图数据，所述地图数据是基于物体的第一点云数据、所述第一终端在所述第一终端的第一相机坐标系中的位姿数据、以及所述第一终端的二维特征图像而生成的；其中，所述第一点云数据表示所述物体的表面点在所述第一终端的第一相机坐标系中的三维坐标值；
    所述接收模块还用于，接收第二终端在所述第二终端的第二相机坐标系中的位姿数据，以及所述第二终端的二维特征图像；
    坐标系变换模块，用于使用所述第二终端的二维特征图像在所述地图数据中进行特征匹配操作，获得所述第二终端在所述第一相机坐标系中的位姿数据；根据所述第二终端在所述第一相机坐标系中的位姿数据和在所述第二相机坐标系中的位姿数据，获得所述第一相机坐标系与所述第二相机坐标系之间的坐标系变换数据；
    发送模块，用于将所述坐标系变换数据发送至所述第二终端。
PCT/CN2019/110330 2018-10-15 2019-10-10 用于虚拟场景的数据处理方法以及设备 WO2020078250A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19874171.2A EP3855400A4 (en) 2018-10-15 2019-10-10 DATA PROCESSING METHOD AND DEVICE FOR VIRTUAL SCENE
US17/230,102 US20210233272A1 (en) 2018-10-15 2021-04-14 Data processing method and device used in virtual scenario

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811201612.1 2018-10-15
CN201811201612.1A CN109949422B (zh) 2018-10-15 2018-10-15 用于虚拟场景的数据处理方法以及设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/230,102 Continuation US20210233272A1 (en) 2018-10-15 2021-04-14 Data processing method and device used in virtual scenario

Publications (1)

Publication Number Publication Date
WO2020078250A1 true WO2020078250A1 (zh) 2020-04-23

Family

ID=67005928

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/110330 WO2020078250A1 (zh) 2018-10-15 2019-10-10 用于虚拟场景的数据处理方法以及设备

Country Status (4)

Country Link
US (1) US20210233272A1 (zh)
EP (1) EP3855400A4 (zh)
CN (2) CN109949422B (zh)
WO (1) WO2020078250A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634439A (zh) * 2020-12-25 2021-04-09 北京奇艺世纪科技有限公司 一种3d信息展示方法及装置

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109949422B (zh) * 2018-10-15 2020-12-15 华为技术有限公司 用于虚拟场景的数据处理方法以及设备
KR102608127B1 (ko) * 2019-04-08 2023-12-01 삼성전자주식회사 이미지 프로세싱을 수행하는 전자 장치 및 방법
CN112348887A (zh) * 2019-08-09 2021-02-09 华为技术有限公司 终端位姿确定方法以及相关装置
CN110989825B (zh) * 2019-09-10 2020-12-01 中兴通讯股份有限公司 增强现实互动实现方法及系统、增强现实设备、存储介质
CN110673115B (zh) * 2019-09-25 2021-11-23 杭州飞步科技有限公司 雷达与组合导航系统的联合标定方法、装置、设备及介质
CN110866977B (zh) * 2019-10-31 2023-06-16 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备
CN112785715A (zh) 2019-11-08 2021-05-11 华为技术有限公司 虚拟物体显示方法以及电子设备
WO2021089006A1 (zh) * 2019-11-08 2021-05-14 华为技术有限公司 数字空间的管理方法、装置与设备
CN111078003B (zh) * 2019-11-27 2021-10-22 Oppo广东移动通信有限公司 数据处理方法、装置、电子设备及存储介质
CN111060118B (zh) * 2019-12-27 2022-01-07 炬星科技（深圳）有限公司 场景地图建立方法、设备及存储介质
CN111338474B (zh) * 2020-02-19 2022-11-08 Oppo广东移动通信有限公司 虚拟对象位姿校准方法及装置、存储介质和电子设备
CN111415388B (zh) * 2020-03-17 2023-10-24 Oppo广东移动通信有限公司 一种视觉定位方法及终端
CN111506323B (zh) * 2020-04-20 2023-06-27 武汉灏存科技有限公司 基于虚拟场景的数据处理方法、装置、设备及存储介质
CN114812381B (zh) * 2021-01-28 2023-07-18 华为技术有限公司 电子设备的定位方法及电子设备
CN112950711A (zh) * 2021-02-25 2021-06-11 深圳市慧鲤科技有限公司 一种对象的控制方法、装置、电子设备及存储介质
US11409359B1 (en) * 2021-08-06 2022-08-09 Kinoo, Inc. Systems and methods for collective control of virtual objects
CN113784049B (zh) * 2021-09-17 2023-08-01 西安万像电子科技有限公司 安卓系统虚拟机的摄像头调用方法、电子设备和存储介质
CN114794667B (zh) * 2022-03-31 2023-04-14 深圳市如本科技有限公司 工具标定方法、系统、装置、电子设备以及可读存储介质
CN115220636B (zh) * 2022-07-14 2024-04-26 维沃移动通信有限公司 虚拟操作方法、装置、电子设备及可读存储介质
CN115222899B (zh) * 2022-09-21 2023-02-21 湖南草根文化传媒有限公司 虚拟数字人生成方法、系统、计算机设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105528082A (zh) * 2016-01-08 2016-04-27 北京暴风魔镜科技有限公司 三维空间及手势识别追踪交互方法、装置和系统
CN106355647A (zh) * 2016-08-25 2017-01-25 北京暴风魔镜科技有限公司 增强现实系统和方法
US9858669B2 (en) * 2015-10-23 2018-01-02 The Boeing Company Optimized camera pose estimation system
CN107748569A (zh) * 2017-09-04 2018-03-02 中国兵器工业计算机应用技术研究所 用于无人机的运动控制方法、装置及无人机系统
CN109949422A (zh) * 2018-10-15 2019-06-28 华为技术有限公司 用于虚拟场景的数据处理方法以及设备

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140267234A1 (en) * 2013-03-15 2014-09-18 Anselm Hook Generation and Sharing Coordinate System Between Users on Mobile
US9773313B1 (en) * 2014-01-03 2017-09-26 Google Inc. Image registration with device data
WO2015161307A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US9928656B2 (en) * 2015-09-11 2018-03-27 Futurewei Technologies, Inc. Markerless multi-user, multi-object augmented reality on mobile devices
US9630619B1 (en) * 2015-11-04 2017-04-25 Zoox, Inc. Robotic vehicle active safety systems and methods
CN107223269B (zh) * 2016-12-29 2021-09-28 达闼机器人有限公司 三维场景定位方法和装置
GB201705767D0 (en) * 2017-04-10 2017-05-24 Blue Vision Labs Uk Ltd Co-localisation
CN107230225B (zh) * 2017-04-25 2020-06-09 华为技术有限公司 三维重建的方法和装置
CN110799804A (zh) * 2017-06-30 2020-02-14 深圳市大疆创新科技有限公司 地图生成系统和方法
CN107820593B (zh) * 2017-07-28 2020-04-17 深圳市瑞立视多媒体科技有限公司 一种虚拟现实交互方法、装置及系统
CN107885871A (zh) * 2017-11-24 2018-04-06 南京华捷艾米软件科技有限公司 基于云计算的同步定位与地图构建方法、系统、交互系统
CN108389264B (zh) * 2018-02-07 2022-03-29 杭州易现先进科技有限公司 坐标系统确定方法、装置、存储介质及电子设备
WO2019204800A1 (en) * 2018-04-20 2019-10-24 WeRide Corp. Method and system for generating high definition map

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9858669B2 (en) * 2015-10-23 2018-01-02 The Boeing Company Optimized camera pose estimation system
CN105528082A (zh) * 2016-01-08 2016-04-27 北京暴风魔镜科技有限公司 三维空间及手势识别追踪交互方法、装置和系统
CN106355647A (zh) * 2016-08-25 2017-01-25 北京暴风魔镜科技有限公司 增强现实系统和方法
CN107748569A (zh) * 2017-09-04 2018-03-02 中国兵器工业计算机应用技术研究所 用于无人机的运动控制方法、装置及无人机系统
CN109949422A (zh) * 2018-10-15 2019-06-28 华为技术有限公司 用于虚拟场景的数据处理方法以及设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3855400A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634439A (zh) * 2020-12-25 2021-04-09 北京奇艺世纪科技有限公司 一种3d信息展示方法及装置
CN112634439B (zh) * 2020-12-25 2023-10-31 北京奇艺世纪科技有限公司 一种3d信息展示方法及装置

Also Published As

Publication number Publication date
CN109949422A (zh) 2019-06-28
EP3855400A4 (en) 2021-11-10
CN109949422B (zh) 2020-12-15
EP3855400A1 (en) 2021-07-28
CN112530024A (zh) 2021-03-19
US20210233272A1 (en) 2021-07-29

Similar Documents

Publication Publication Date Title
WO2020078250A1 (zh) 用于虚拟场景的数据处理方法以及设备
US11145083B2 (en) Image-based localization
US10055879B2 (en) 3D human face reconstruction method, apparatus and server
CN110140099B (zh) 用于跟踪控制器的系统和方法
WO2019184889A1 (zh) 增强现实模型的调整方法、装置、存储介质和电子设备
US11893702B2 (en) Virtual object processing method and apparatus, and storage medium and electronic device
US11776209B2 (en) Image processing method and apparatus, electronic device, and storage medium
WO2021088498A1 (zh) 虚拟物体显示方法以及电子设备
US20220319050A1 (en) Calibration method and apparatus, processor, electronic device, and storage medium
KR102197615B1 (ko) 증강 현실 서비스를 제공하는 방법 및 증강 현실 서비스를 제공하기 위한 서버
CN111542128B (zh) 一种基于uwb的设备交互方法、装置及设备
CN109754464B (zh) 用于生成信息的方法和装置
CN109992111B (zh) 增强现实扩展方法和电子设备
US11869195B2 (en) Target object controlling method, apparatus, electronic device, and storage medium
CN112506465B (zh) 全景漫游中场景切换的方法和装置
US11385856B2 (en) Synchronizing positioning systems and content sharing between multiple devices
WO2021088497A1 (zh) 虚拟物体显示方法、全局地图更新方法以及设备
WO2023109564A1 (zh) 视频图像处理方法、装置、电子设备及存储介质
CN114529452A (zh) 用于显示图像的方法、装置和电子设备
KR102525583B1 (ko) 경량형 증강현실장치의 스테레오 프레임을 이용한 3차원 맵을 생성하기 위한 장치 및 이를 위한 방법
CN112270242B (zh) 轨迹的显示方法、装置、可读介质和电子设备
CN117906634A (zh) 一种设备检测方法、装置、设备及介质
CN116137666A (zh) 视频匀速显示方法及装置、计算机可读介质和电子设备
CN114078083A (zh) 头发变换模型生成方法和装置、头发变换方法和装置
CN113497944A (zh) 多视角三维直播方法、系统、装置、终端和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19874171

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019874171

Country of ref document: EP

Effective date: 20210419