WO2021164712A1 - Pose tracking method, wearable device, mobile device, and storage medium - Google Patents

Abstract

Disclosed in embodiments of the present application are a pose tracking method, a wearable device, a mobile device, and a storage medium. The method comprises: acquiring an image to be processed; performing feature processing on said image, and determining a group of feature information of said image; and determining pose information of the wearable device according to the group of feature information. Thus, regarding pose tracking of the wearable device, since simultaneous localization and mapping (SLAM) is operated on the mobile device, and the wearable device does not perform mapping and merely executes a localization algorithm, the power consumption of the wearable device is reduced, and system latency is also reduced, thereby improving the visual quality of the wearable device.

Description

Pose tracking method, wearable device, mobile device and storage medium

Cross-references to related applications

This application claims priority to the prior U.S. provisional patent application entitled "Method for Pose Tracking on Wearable Visual Enhancement Devices" filed on February 19, 2020 with application number 62/978,704, the entire content of which is incorporated by reference In this application.

Technical field

The embodiments of the present application relate to the field of vision enhancement technology, and in particular, to a pose tracking method, a wearable device, a mobile device, and a storage medium.

Background technique

In recent years, with the development of visual enhancement technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), a virtual three-dimensional world can be simulated through a computer system, so that The user can interact with the virtual scene and give the user an immersive experience.

In the split vision enhancement system, a wearable device and a mobile device are included, and the wearable device and the mobile device are connected through wired or wireless communication. At present, in order to implement 6-degree-of-freedom (DOF) tracking for wearable devices (such as split AR glasses) that work with mobile devices, there are usually two methods to achieve this. One method is to use the wearable device itself The computing power of the mobile device performs 6DOF tracking. Another way is to use the computing power of the mobile device itself to perform 6DOF tracking. However, both of these methods have some drawbacks. For example, the former leads to high hardware complexity and power consumption of wearable devices. High; the latter causes a large system delay, which may damage the visual quality of wearable devices.

Summary of the invention

The embodiments of the present application provide a pose tracking method, a wearable device, a mobile device, and a storage medium, which can not only reduce the power consumption of the wearable device, but also reduce the system delay to improve the visual quality of the wearable device.

The technical solutions of the embodiments of the present application can be implemented as follows:

In the first aspect, an embodiment of the present application provides a pose tracking method, which is applied to a wearable device, and the method includes:

Obtain the image to be processed;

Performing feature processing on the image to be processed, and determining a set of feature information of the image to be processed;

According to the set of characteristic information, the pose information of the wearable device is determined.

In some embodiments, the determining the pose information of the wearable device according to the set of characteristic information includes:

Sending the set of characteristic information to a mobile device;

Based on the response of the mobile device, receiving the pose information of the wearable device returned by the mobile device.

In some embodiments, the determining the pose information of the wearable device according to the set of characteristic information includes:

Sending the set of feature information to a mobile device, and receiving the three-dimensional coordinate information of the matching feature included in the target feature subset returned by the mobile device;

Acquiring two-dimensional projection information of the matching feature included in the target feature subset on the image to be processed;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

In some embodiments, the determining the pose information of the wearable device according to the set of characteristic information includes:

Receive a set of scene characteristic information sent by the mobile device;

Perform feature matching between the set of scene feature information and the set of feature information, determine a target feature subset, and obtain the three-dimensional coordinate information of the matching feature included in the target feature subset and the matching feature in the waiting list. Process the two-dimensional projection information on the image;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

In the second aspect, an embodiment of the present application provides a pose tracking method, which is applied to a mobile device, and the method includes:

Acquiring sensor data of the mobile device;

Performing synchronous positioning and map construction SLAM processing on the sensor data to construct a scene map;

Based on the scene map, the pose information of the mobile device is determined.

In some embodiments, the method further includes:

Receive a set of characteristic information sent by the wearable device;

Calculating the pose information of the wearable device based on the scene map and the set of feature information;

Sending the pose information of the wearable device to the wearable device.

In some embodiments, the calculating the pose information of the wearable device based on the scene map and the set of feature information includes:

Determine a set of scene feature information in the scene map according to the sensor data;

Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

Acquiring three-dimensional coordinate information of the matching feature included in the target feature subset and two-dimensional projection information on the image to be processed of the wearable device;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

In some embodiments, the method further includes:

Receive a set of characteristic information sent by the wearable device;

Determine a set of scene feature information in the scene map according to the sensor data;

Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

The three-dimensional coordinate information of the matching feature included in the target feature subset is sent to the wearable device.

In some embodiments, the method further includes:

Determine a set of scene feature information in the scene map according to the sensor data;

Sending the set of scene characteristic information to the wearable device.

In a third aspect, an embodiment of the present application provides a wearable device, the wearable device including a first acquiring unit, a first processing unit, and a first determining unit; wherein,

The first acquiring unit is configured to acquire an image to be processed;

The first processing unit is configured to perform feature processing on the image to be processed, and determine a set of feature information of the image to be processed;

The first determining unit is configured to determine the pose information of the wearable device according to the set of characteristic information.

In a fourth aspect, an embodiment of the present application provides a wearable device, the wearable device including a first memory and a first processor; wherein,

The first memory is configured to store a computer program that can run on the first processor;

The first processor is configured to execute the method according to any one of the first aspects when running the computer program.

In a fifth aspect, an embodiment of the present application provides a mobile device, the mobile device including a second acquiring unit, a second processing unit, and a second determining unit; wherein,

The second acquiring unit is configured to acquire sensor data of the mobile device;

The second processing unit is configured to perform SLAM processing on the sensor data to construct a scene map;

The second determining unit is configured to determine the pose information of the mobile device based on the scene map.

In a sixth aspect, an embodiment of the present application provides a mobile device, the mobile device including a second memory and a second processor; wherein,

The second memory is configured to store a computer program that can run on the second processor;

The second processor is configured to execute the method according to any one of the second aspects when running the computer program.

In a seventh aspect, an embodiment of the present application provides a computer storage medium that stores a computer program, and when the computer program is executed by a first processor, the method as described in any one of the first aspect is implemented, Or when executed by the second processor, the method according to any one of the second aspects is implemented.

The embodiments of the application provide a pose tracking method, wearable devices, mobile devices, and storage media, which are applied to wearable devices, by acquiring images to be processed; performing feature processing on the images to be processed to determine the to-be-processed images A set of characteristic information of the image; according to the set of characteristic information, the pose information of the wearable device is determined. Applied to a mobile device, by acquiring sensor data of the mobile device; performing SLAM processing on the sensor data to construct a scene map; and determining the pose information of the mobile device based on the scene map. In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

Description of the drawings

FIG. 1 is a schematic diagram of an application scenario of a vision enhancement system provided by an embodiment of this application;

FIG. 2 is a schematic flowchart of a pose tracking method provided by an embodiment of the application;

FIG. 3 is a schematic flowchart of another pose tracking method provided by an embodiment of the application;

FIG. 4 is a detailed flowchart of a pose tracking method provided by an embodiment of the application;

FIG. 5 is a schematic diagram of an application scenario of a pose tracking method provided by an embodiment of the application;

6 is a schematic diagram of the composition structure of a wearable device provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a specific hardware structure of a wearable device provided by an embodiment of the application;

FIG. 8 is a schematic diagram of the composition structure of a mobile device according to an embodiment of the application;

FIG. 9 is a schematic diagram of a specific hardware structure of a mobile device provided by an embodiment of the application.

Detailed ways

In order to have a more detailed understanding of the characteristics and technical content of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The attached drawings are for reference and explanation purposes only, and are not used to limit the embodiments of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terminology used herein is only for the purpose of describing the embodiments of the application, and is not intended to limit the application.

In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and Can be combined with each other without conflict. It should also be pointed out that the term "first\second\third" referred to in the embodiments of this application only distinguishes similar objects, and does not represent a specific order for the objects. Understandably, "first\second\third" "Three" may be interchanged in specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein.

Refer to FIG. 1, which shows a schematic diagram of an application scenario of a vision enhancement system provided by an embodiment of the present application. As shown in FIG. 1, the vision enhancement system 10 may include a wearable device 110 and a mobile device 120. Among them, the wearable device 110 and the mobile device 120 are in a wired or wireless communication connection.

Here, the wearable device 110 may refer to a monocular or binocular head-mounted display (Head-Mounted Display, HMD), such as AR glasses. In FIG. 1, the wearable device 110 may include one or more display modules 111 placed near the position of the user's single eye or both eyes. Among them, through the display module 111 of the wearable device 110, the content displayed therein can be presented in front of the user's eyes, and the displayed content can fill or partially fill the user's field of vision. It should also be noted that the display module 111 may refer to one or more organic light-emitting diode (OLED) modules, liquid crystal display (LCD) modules, laser display modules, and the like.

In addition, in some embodiments, the wearable device 110 may also include one or more sensors and one or more cameras. For example, the wearable device 110 may include one or more sensors such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, a proximity sensor, and a depth camera.

The mobile device 120 may be wirelessly connected to the wearable device 110 according to one or more wireless communication protocols (for example, Bluetooth, WIFI, etc.). Alternatively, the mobile device 120 may also be wired to the wearable device 110 via a data cable (such as a USB cable) according to one or more data transmission protocols such as Universal Serial Bus (USB). Here, the mobile device 120 may be implemented in various forms. For example, the mobile devices described in the embodiments of the present application may include smart phones, tablet computers, notebook computers, laptop computers, palmtop computers, personal digital assistants (Personal Digital Assistant, PDA), smart watches, and so on.

In some embodiments, a user operating on the mobile device 120 can control the operation at the wearable device 110 via the mobile device 120. In addition, the data collected by the sensors in the wearable device 110 may also be sent back to the mobile device 120 for further processing or storage.

It should also be noted that, in the embodiment of the present application, the wearable device 110 may specifically refer to a wearable vision enhancement device, such as AR glasses, MR glasses, and VR glasses. That is to say, 6DOF tracking for wearable devices, especially for those visual enhancement devices that work with the mobile device 120 (or "mobile computing devices", such as smart phones, mobile phones, computing boxes, etc.) 6DOF tracking, here, "wearable device" and "AR glasses" can be used interchangeably. Mobile devices refer to smart phones that work with AR glasses. AR glasses can be connected to mobile devices via USB cables, WiFi, Bluetooth, etc. . Among them, the 6DOF tracking (that is, the 3DOF position and the 3DOF attitude) of the AR glasses is usually completed by a simultaneous localization and mapping (Simultaneous Localization and Mapping, SLAM) algorithm. The SLAM algorithm is the process of constructing or updating an unknown environment map, while tracking the location and posture of the device in the environment. Today's mobile devices are equipped with SLAM functions that match the AR framework, such as Apple's ARKit and Google's ARCore; and independent AR glasses can also have SLAM functions, such as Microsoft Hololens and Magic Leap One.

In the related art, in order to achieve 6DOF tracking of AR glasses that work with mobile devices, there are usually two methods. One method is to use the computing power on the AR glasses to perform SLAM. At this time, the sensor data will never leave the AR glasses. Another method is to use the computing power on the mobile device to perform SLAM. At this time, the sensor data is collected by the AR glasses and then sent to the mobile device for further processing.

In this way, the advantage of performing 6DOF tracking on AR glasses is to minimize the system delay, which refers to the delay between the movement of an object and the change in response to the display module. Since a large delay will destroy the time continuity and cause the display in the AR glasses to jitter, and then cause the user to faint; therefore, the data can be directly processed on the AR glasses to minimize the data transmission, thereby reducing the system delay. However, this solution has the following shortcomings: (1) AR glasses require certain hardware (such as chips and memory, etc.) to process sensor data and perform SLAM. This may result in the need for larger hardware components, lower industrial design feasibility, and higher prices; (2) The SLAM process requires a very large amount of calculation, and may also lead to greater power consumption and heat accumulation on AR glasses. On the other hand, 6DOF tracking can also be performed on a separate mobile device, which can reduce power consumption and heat accumulation on AR glasses, and the hardware requirements for AR glasses are not so strong, and provide greater flexibility to industrial design sex. However, the added system delay of this solution transmitting sensor data to the processing unit via a USB cable or WiFi often damages the visual quality of AR glasses.

The embodiment of the application provides a pose tracking method, which is applied to a wearable device. The basic idea of the method is: obtain an image to be processed; perform feature processing on the image to be processed, and determine a group of the image to be processed Characteristic information; determine the pose information of the wearable device according to the set of characteristic information.

The embodiment of the application also provides a pose tracking method, which is applied to a mobile device. The basic idea of the method is: acquiring sensor data of the mobile device; performing SLAM processing on the sensor data to construct a scene map; The scene map determines the pose information of the mobile device.

In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

Hereinafter, each embodiment of the present application will be described in detail with reference to the accompanying drawings.

In an embodiment of the present application, refer to FIG. 2, which shows a schematic flowchart of a pose tracking method provided by an embodiment of the present application. As shown in Figure 2, the method may include:

S201: Obtain an image to be processed.

It should be noted that this method is applied to wearable devices. Here, the wearable device may also be referred to as a "head-mounted device" or a "visual enhancement device", such as AR glasses. In the embodiments of the present application, the wearable device here generally refers to a split wearable device, so the AR glasses may also be referred to as split AR glasses.

It should also be noted that in the split vision enhancement system, for pose tracking, it is also necessary to include a mobile device that works in conjunction with a wearable device, and a wired communication connection can be established between the mobile device and the wearable device through a data cable. It is also possible to establish a wireless communication connection through a wireless communication protocol.

In this way, for the processing algorithm of pose tracking, part of the processing algorithm can be executed in the mobile device, and part of the processing algorithm can be executed in the wearable device. In the embodiments of this application, in order to optimize the system delay and the power consumption of the wearable device, the simultaneous positioning and mapping (Simultaneous Localization and Mapping, SLAM) algorithm in the pose tracking can be placed in the mobile device, and in the wearable device. Only the positioning algorithm runs on the device (there is no need to build a scene map).

In this way, for a wearable device, after the positioning module in the wearable device is started, an image needs to be collected first. In some embodiments, the acquiring the image to be processed may include:

Image acquisition is performed through the camera of the wearable device to obtain the to-be-processed image.

That is to say, the wearable device includes a first camera. After the image is collected by the first camera, the image to be processed can be obtained, and the image to be processed can be subsequently used for feature extraction/detection.

S202: Perform feature processing on the image to be processed, and determine a set of feature information of the image to be processed.

It should be noted that the feature processing here may include feature detection and feature description. In the embodiment of the present application, the feature processing of the image to be processed can be implemented according to a preset feature algorithm. The preset feature algorithm may be a feature extraction and description (Oriented FAST and Rotated Brief, ORB) algorithm, or a Scale Invariant Feature Transform (SIFT) algorithm, etc., which are not specifically limited in the embodiment of the application.

It should also be noted that, considering the subsequent feature matching and pose calculation, usually at least four feature points are required for pose calculation; therefore, a set of feature information here includes at least four feature information.

In some embodiments, the performing feature processing on the image to be processed and determining a set of feature information of the image to be processed may include:

Performing feature detection on the image to be processed, and acquiring a set of detection features of the image to be processed;

Perform feature description on the set of detection features, and obtain descriptors associated with the set of detection features;

The set of detection features and the descriptor associated with the set of detection features are determined as a set of feature information of the image to be processed.

Here, taking the ORB algorithm as an example, the ORB algorithm can be divided into two parts, namely feature extraction and feature description. Among them, the feature extraction is developed by the FAST (Features from Accelerated Segment Test) algorithm, and the feature description is improved based on the BRIEF (Binary Robust Independent Elementary Features) feature description algorithm. The ORB feature combines the FAST feature detection method with the BRIEF feature descriptor, and improves and optimizes them on the original basis. Here, the feature points of the image can be simply understood as the more prominent points in the image, such as contour points, bright spots in a darker area, and dark spots in a brighter area. In other words, ORB uses FAST algorithm to detect feature points. This definition is based on the gray value of the image around the feature point, and the pixel value of a circle around the candidate feature point is detected. If there are enough pixels in the surrounding area of the candidate point and the gray value difference of the candidate point is large enough, the candidate point is considered to be a feature point. After the feature points are obtained, the attributes of these feature points need to be described in some way. The output of these attributes is called the Feature Descriptor of the feature point. ORB uses the BRIEF algorithm to calculate the descriptor of a feature point. The core idea of the BRIEF algorithm is: around the key points, select N point pairs in a certain pattern, and combine the comparison results of these N point pairs as a descriptor.

Taking the SIFT algorithm as an example, the goal of the SIFT algorithm is to solve many practical problems in low-level feature extraction and its image matching applications. The SIFT algorithm mainly includes two stages. The first stage is the generation of SIFT features, that is, the feature vectors that are irrelevant to scale scaling, rotation, and brightness changes are extracted from multiple images; the second stage is the matching of SIFT feature vectors. The low-level feature extraction in the SIFT algorithm is to select those obvious features, which have image scale (feature size) and rotation invariance, and also have a certain degree of invariance to illumination changes. In addition, the SIFT algorithm can also reduce the low extraction probability caused by occlusion, clutter and noise. Specifically, SIFT is an algorithm for detecting local features. The algorithm obtains features and performs image feature matching by obtaining feature points (interest points, or corner points) in an image and descriptors related to scale and orientation. .

In contrast, the biggest feature of the ORB algorithm is its fast calculation speed. This firstly benefits from the use of FAST to detect features; secondly, the use of the BRIEF algorithm to calculate the descriptor. The unique binary string representation of the descriptor not only saves storage space, but also greatly shortens the matching time.

In this way, after obtaining a set of feature information of the image to be processed, the scene map constructed by the mobile device can be combined to determine the pose information of the wearable device.

S203: Determine the pose information of the wearable device according to the set of characteristic information.

It should be noted that the pose information in the embodiments of this application may be 6 degrees of freedom (Degree of Freedom, DOF) information, which may specifically include 3DOF position information and 3DOF pose information; according to the pose information of the wearable device, the Tracking the position and posture of the wearable device.

Taking into account the different architectures of wearable devices, mobile devices can execute most of the processing algorithms, or part of the processing algorithms, or a small part of the processing algorithms, so that the wearable device can perform a large amount of processing calculations, and can also perform a small amount of processing. Calculations, etc., will be described in detail in several situations below.

In a possible implementation manner, most of the processing algorithms (such as feature matching and pose calculation, etc.) are performed on the mobile device. At this time, in some embodiments, the determining the pose information of the wearable device according to the set of characteristic information may include:

Sending the set of characteristic information to a mobile device;

Based on the response of the mobile device, receiving the pose information of the wearable device returned by the mobile device.

It should be noted that in this way, after the wearable device sends a set of feature information to the mobile device, the mobile device can calculate the pose information of the wearable device according to the scene map built by itself and this set of feature information , And then send the posture information of the wearable device to the wearable device to track the position and posture of the wearable device.

In another possible implementation manner, part of the processing algorithm (such as feature matching) is performed on the mobile device, and part of the processing algorithm (such as the pose calculation) is performed on the wearable device. At this time, in some embodiments, the determining the pose information of the wearable device according to the set of characteristic information may include:

Sending the set of feature information to a mobile device, and receiving the three-dimensional coordinate information of the matching feature included in the target feature subset returned by the mobile device;

Acquiring two-dimensional projection information of the matching feature included in the target feature subset on the image to be processed;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

It should be noted that in this way, after the wearable device sends a set of feature information to the mobile device, the mobile device can perform feature matching according to the scene map built by itself and this set of feature information to obtain a set of matching features. The target feature subset and the three-dimensional coordinate information of these matching features, and then after sending it to the wearable device, the wearable device can use the three-dimensional coordinate information of these matching features and the two-dimensional projection information of these matching features on the image to be processed. Perform pose calculation to obtain the pose information of the wearable device, which is used to track the position and posture of the wearable device.

In yet another possible implementation manner, most of the processing algorithms (such as feature matching, pose calculation, etc.) are performed on the wearable device. At this time, in some embodiments, the determining the pose information of the wearable device according to the set of characteristic information may include:

Receive a set of scene characteristic information sent by the mobile device;

Perform feature matching between the set of scene feature information and the set of feature information, determine a target feature subset, and obtain the three-dimensional coordinate information of the matching feature included in the target feature subset and the matching feature in the waiting list. Process the two-dimensional projection information on the image;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

It should be noted that in this way, after the mobile device constructs the scene map, a set of scene feature information can be sent to the wearable device, and then the wearable device performs feature matching to obtain the target feature subset and the target The three-dimensional coordinate information of the matching feature included in the feature subset and its two-dimensional projection information on the image to be processed, and then use the preset algorithm to calculate the pose, you can get the pose information of the wearable device to track the wearable The location and attitude of the device. It should be noted that under normal circumstances, the target feature subset includes at least four matching features for subsequent pose calculations.

Illustratively, assuming that a set of scene feature information can include 1000, and a set of feature information can include 100, then the 1000 scene feature information is matched with 100 feature information, and finally there are 10 features that can be matched. At this time, these 10 matching features form a subset of the target feature for subsequent pose calculations to obtain pose information of the wearable device.

It should also be noted that the preset algorithm described in the embodiment of the present application may be a perspective-n-point (PNP) algorithm. Here, the PNP algorithm is a method for solving three-dimensional (Three Dimension, 3D) to (Two Dimension, 2D) point pair motion. In layman's terms, it describes the problem of estimating the pose of the calibration camera given a set of n 3D points in the world reference coordinate system and their corresponding 2D projections in the image. In the embodiments of this application, there are multiple solving algorithms for the PNP problem: P3P that can use three pairs of points to estimate the pose, direct linear transformation (DLT) algorithm, and efficient PNP (Efficient Perspective-n-Point, EPNP) ) Algorithms, etc. In addition, the embodiment of the present application may also use a non-linear optimization method to construct a least squares problem and iteratively solve it, that is, the beam adjustment (Bundle Adjustment, BA) method, which is not specifically limited here.

In short, in the embodiment of the present application, the mobile device (such as a smart phone) in the split vision enhancement system runs the SLAM algorithm, and at the same time runs the positioning algorithm on the wearable device (such as AR glasses) (the map is no longer built). In addition, as an alternative to executing the SLAM algorithm on the mobile device and the positioning algorithm on the wearable device at the same time, the embodiments of the present application may not need to execute the SLAM algorithm on the mobile device and the positioning algorithm on the wearable device at the same time. It is also possible to execute the two algorithms separately in order. For example, you can run the SLAM algorithm to build a scene map of the entire room, and then use the positioning algorithm to track the pose information of the wearable device.

This embodiment provides a pose tracking method, applied to a wearable device, by acquiring an image to be processed; performing feature processing on the image to be processed to determine a set of feature information of the image to be processed; The group feature information determines the pose information of the wearable device. In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

In another embodiment of the present application, refer to FIG. 3, which shows a schematic flowchart of another pose tracking method provided in an embodiment of the present application. As shown in Figure 3, the method may include:

S301: Acquire sensor data of the mobile device.

It should be noted that this method is applied to mobile devices. Here, the mobile device may be, for example, a smart phone, a tablet computer, a notebook computer, a laptop computer, a palmtop computer, a personal digital assistant (PDA), a smart watch, etc., which are not specifically limited in the embodiment of the present application.

It should also be noted that in the split vision enhancement system, it is also necessary to include a wearable device that cooperates with the mobile device, and the mobile device and the wearable device can establish a wired communication connection through a data cable or wireless communication The protocol establishes a wireless communication connection.

In this way, for the processing algorithm of pose tracking, part of the processing algorithm can be executed in the mobile device, and part of the processing algorithm can be executed in the wearable device. In the embodiment of this application, in order to optimize the system delay and the power consumption of the wearable device, the SLAM algorithm in the pose tracking can be placed in the mobile device, and only the positioning algorithm is run on the wearable device (there is no need to build the scene map).

In addition, after the entire system is started, the SLAM algorithm on the mobile device starts to run based on the sensor data on the mobile device, so the sensor data of the mobile device needs to be acquired first. In the embodiments of the present application, the sensor data is usually realized by using a visual inertial odometer. In a specific example, it can be obtained by using a camera and an IMU sensor on a mobile device.

S302: Perform SLAM processing on the sensor data to construct a scene map.

S303: Determine the pose information of the mobile device based on the scene map.

It should be noted that the mobile device may include a SLAM module, and the SLAM module is used to execute SLAM algorithms and processing. In this way, when the entire system starts, the SLAM module on the mobile device starts to run using the sensor data on the mobile device. When the system sees enough features in the scene and initializes the SLAM module, it will start 6DOF tracking of the mobile device. Under the coordinate system established by the SLAM algorithm, the position and posture of the mobile device can be continuously tracked. At the same time, the scene map and the 3D representation of the environment will also be constructed. In addition, if the SLAM module sees an invisible part of the scene, the scene map will also be updated.

It should also be noted that under normal circumstances, after the SLAM module on the mobile device is initialized, the positioning module on the wearable device is started, and the wearable device no longer needs to build a scene map; thus not only can the system delay be reduced, but also because of the Wearable devices no longer need some hardware to perform SLAM, and it can also reduce the power consumption of wearable devices.

In the scene map constructed by the mobile device, for calculating the pose information of the wearable device, in a possible implementation manner, most of the processing algorithms (such as feature matching and pose calculation) are performed on the mobile device. At this time, in some embodiments, the method may further include:

Receive a set of characteristic information sent by the wearable device;

Calculating the pose information of the wearable device based on the scene map and the set of feature information;

Sending the pose information of the wearable device to the wearable device.

It should be noted that in this way, the wearable device only needs to send a set of characteristic information to the mobile device, and then the mobile device calculates the position of the wearable device according to the scene map constructed by itself and this set of characteristic information. At last, the mobile device sends the calculated pose information to the wearable device.

In the embodiment of this application, the scene map is actually a data structure that stores the feature points that have been seen in the scene so far; then at any one time, the camera on the wearable device sees only a partial scene The feature points of, need to be compared with the feature points in the scene map to find the corresponding points (ie matching features). Therefore, the role of the scene map is the same as the map in real life.

It should also be noted that the calculation of the pose information of the wearable device in the mobile device requires not only feature matching, but also pose calculation based on the three-dimensional coordinate information and two-dimensional projection information of the matching feature. Specifically, in some embodiments, the calculating the pose information of the wearable device based on the scene map and the set of feature information may include:

Determine a set of scene feature information in the scene map according to the sensor data;

Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

Acquiring three-dimensional coordinate information of the matching feature included in the target feature subset and two-dimensional projection information on the image to be processed of the wearable device;

A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.

Further, a set of characteristic information sent by the wearable device may include a set of detection characteristics and a set of descriptors associated with the detection characteristics. In some embodiments, the feature matching the set of scene feature information with the set of feature information to determine the target feature subset may include:

Based on the calculated distance between the descriptor and the descriptor, feature matching is performed on the set of detected features and the scene features in the set of scene feature information to determine the target feature subset.

In other words, the mobile device can construct a scene map based on the sensor data on the one hand, and on the other hand, it can also determine the scene feature information. Then, the received set of feature information and the set of scene feature information are feature-matched to determine the target feature subset composed of matching features. Here, a set of feature information includes at least four feature information, and a set of scene feature information includes at least four scene feature information. The distance between the descriptor and the descriptor can be calculated to compare this set of detected features with this set of scenes. Feature matching is performed on the scene features in the feature information. It should be noted that each feature can be understood as a vector. To determine whether two features match, or how close the two features are, can be determined by using the distance between the feature vectors.

In this way, after the target feature subset is determined, the 3D coordinate information of the matching feature included in the target feature subset and its 2D projection information on the image to be processed on the wearable device can be further obtained, and then the preset algorithm is used to target the target The 3D coordinate information and 2D projection information of the matching feature included in the feature subset are subjected to pose calculation to obtain the pose information of the wearable device.

In the embodiment of the present application, the preset algorithm may be a PNP algorithm. Here, the PNP algorithm specifically estimates the pose information of the wearable device based on a set of 3D points and their 2D projection points on the image to be processed of the wearable device.

In another possible implementation manner, part of the processing algorithm (such as feature matching) is performed on the mobile device, and part of the processing algorithm (such as the pose calculation) is performed on the wearable device. At this time, in some embodiments, the method may further include:

Receive a set of characteristic information sent by the wearable device;

Determine a set of scene feature information in the scene map according to the sensor data;

Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

The three-dimensional coordinate information of the matching feature included in the target feature subset is sent to the wearable device.

It should be noted that in this manner, after the wearable device sends a set of feature information to the mobile device, the mobile device will perform feature matching to determine the target feature subset composed of the matching features. Here, the specific method of feature matching is the same as the above, and will not be repeated here.

In this way, the mobile device can send the target feature subset and the three-dimensional coordinate information of the matching features included in the target feature subset to the wearable device, and then use the PNP algorithm to calculate the pose in the wearable device to determine the pose of the wearable device information.

In yet another possible implementation manner, most of the processing algorithms (such as feature matching, pose calculation, etc.) are performed on the wearable device. At this time, in some embodiments, the method may further include:

Determine a set of scene feature information in the scene map according to the sensor data;

Sending the set of scene characteristic information to the wearable device.

It should be noted that in this way, the mobile device mainly uses SLAM to construct a scene map, and at the same time, it can also track the position and posture of the mobile device. In addition, after the mobile device constructs the scene map, it can send a set of scene feature information to the wearable device, and then the wearable device performs feature matching and pose calculation to determine the pose information of the wearable device.

In short, in the embodiment of the present application, the mobile device (such as a smart phone) in the split vision enhancement system runs the SLAM algorithm, and at the same time runs the positioning algorithm on the wearable device (such as AR glasses) (the map is no longer built). In addition, as an alternative to executing the SLAM algorithm on the mobile device and the positioning algorithm on the wearable device at the same time, the embodiments of the present application may not need to execute the SLAM algorithm on the mobile device and the positioning algorithm on the wearable device at the same time. It is also possible to execute the two algorithms separately in order. For example, you can run the SLAM algorithm to build a scene map of the entire room, and then use the positioning algorithm to track the pose information of the wearable device.

This embodiment provides a pose tracking method, applied to a mobile device, by acquiring sensor data of the mobile device; performing SLAM processing on the sensor data to construct a scene map; based on the scene map, determining the The pose information of the mobile device. In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

In another embodiment of the present application, refer to FIG. 4, which shows a detailed flowchart of a pose tracking method provided in an embodiment of the present application. As shown in Figure 4, the detailed process may include:

S401: Acquire an image to be processed through the camera collection on the wearable device.

S402: The wearable device performs feature processing on the image to be processed, and determines a set of detection features of the image to be processed and a descriptor associated with the set of detection features.

S403: The wearable device sends a set of detection features and associated descriptors of the image to be processed to the mobile device.

S404: The mobile device acquires sensor data of the mobile device.

S405: The mobile device performs SLAM processing on the sensor data, constructs a scene map, and determines a set of scene feature information.

S406: The mobile device performs feature matching by calculating the distance between the descriptors, and determines the target feature subset.

S407: The mobile device uses the PNP algorithm to perform pose calculation on the 3D coordinate information of the matching feature included in the target feature subset and its 2D projection information on the image to be processed by the wearable device to obtain the position of the wearable device. Posture information.

S408: The mobile device sends the calculated pose information to the wearable device.

It should be noted that in the embodiments of the present application, most of the processing algorithms (such as feature matching and pose calculation, etc.) are performed on the mobile device. Among them, the execution subject of steps S401 to S403 is a wearable device, and the execution subject of steps S404 to S408 is a mobile device.

It should also be noted that in the embodiment of the present application, the mobile device (such as a smart phone) in the split vision enhancement system runs the SLAM algorithm, and the wearable device (such as AR glasses) runs the positioning algorithm (the map is no longer built) .

Refer to FIG. 5, which shows a schematic diagram of an application scenario of a pose tracking method according to an embodiment of the present application. In Figure 5, the wearable device is AR glasses, which runs a positioning algorithm to track the position and posture of the AR glasses; the mobile device is a smartphone, which runs a SLAM algorithm to build a scene map and track the position and posture of the smartphone. Taking FIG. 5 as an example, the working principle of the embodiment of the present application is as follows:

When the entire system starts, the SLAM module on the mobile device starts to run using the sensor data on the mobile device. When the system sees enough features in the scene and initializes SLAM, it will start 6DOF tracking of the mobile device. Under the coordinate system established by the SLAM algorithm, the position and direction of the mobile device are continuously tracked. At the same time, the scene map and the 3D representation of the environment will be constructed. If the SLAM module sees the invisible part of the scene, the scene map will also be updated.

After the SLAM module on the mobile device is initialized, the positioning module on the AR glasses starts. The positioning module can use sensor data from the AR glasses as input and use some computing resources on the AR glasses. At this time, the following steps can be used to implement the positioning module: 1) Use the camera on the AR glasses to collect images; 2) Detect features from the image and calculate the descriptors of these detected features on the AR glasses (for example, ORB algorithm, SIFT algorithm, etc.) ); 3) Transmit the detection features and descriptors to the mobile device; 3) By calculating the distance between the descriptors, compare the received detection features with the scene features stored on the mobile device by the SLAM module to obtain a match Features; 4) Use the 3D coordinate information of the matching feature from the mobile device and its 2D projection information on the image of the AR glasses, and then use the PNP algorithm to calculate the 6DOF pose information of the camera on the AR glasses; 5) Calculate The obtained pose information is transmitted from the mobile device to the AR glasses. In this way, the embodiment of the present application only needs to transmit the data between the AR glasses and the mobile device to include detection features, associated descriptors, and pose information. The bandwidth required for these data is much less than the bandwidth required for sensor data, thus minimizing system latency. It should also be noted that when the AR glasses send the detected features to the mobile device, they have actually sent the 2D projection information of these detected features to the mobile device so that the mobile device can use the PNP algorithm for pose calculation.

Understandably, the key idea of this application is that the system runs the SLAM algorithm on a mobile device (for example, a smart phone), and at the same time runs a positioning algorithm on a wearable device (for example, AR glasses) (no more maps are built). In this way, the technical solution of the present application has the following advantages: 1) The system delay of the SLAM algorithm is low, because the SLAM runs based on the data of the mobile device, and the sensor data is no longer required to be transmitted from the AR glasses to the mobile computing device. 2) The power consumption of the AR glasses is low, because the positioning algorithm is only run on the AR glasses instead of the complete SLAM algorithm. 3) The most important point is that the mobile device and the AR glasses are simultaneously tracked in the same coordinate system. Therefore, it is possible to interact between the smart phone and the virtual objects of the AR glasses. For example, a mobile device can be used as a 6DOF controller to interact with virtual objects displayed on AR glasses (for example, using a mobile phone as a boxing glove to play a boxing game).

In addition, it should be noted that, as an alternative to executing the SLAM algorithm on the mobile device and the positioning algorithm on the wearable device at the same time, the embodiment of the present application may not need to execute the SLAM algorithm on the mobile device and the wearable device at the same time. Execute the positioning algorithm on the above, at this time, you can also execute the two algorithms separately in order. For example, you can run the SLAM algorithm to build a scene map of the entire room, and then use the positioning algorithm to track the pose information of the wearable device.

This embodiment provides a pose tracking method. The specific implementation of the foregoing embodiment is described in detail through the foregoing embodiment. It can be seen from this that not only the power consumption of the wearable device is reduced, but the system delay is also reduced, thereby improving The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

In yet another embodiment of the present application, based on the same inventive concept as the foregoing embodiment, refer to FIG. 6, which shows a schematic diagram of the composition structure of a wearable device 60 provided by an embodiment of the present application. As shown in FIG. 6, the wearable device 60 may include: a first acquiring unit 601, a first processing unit 602, and a first determining unit 603; wherein,

The first obtaining unit 601 is configured to obtain an image to be processed;

The first processing unit 602 is configured to perform feature processing on the image to be processed, and determine a set of feature information of the image to be processed;

The first determining unit 603 is configured to determine the pose information of the wearable device according to the set of characteristic information.

In some embodiments, referring to FIG. 6, the wearable device 60 may further include a first sending unit 604 and a first receiving unit 605; wherein,

The first sending unit 604 is configured to send the set of characteristic information to a mobile device;

The first receiving unit 605 is configured to receive the pose information of the wearable device returned by the mobile device based on the response of the mobile device.

In some embodiments, the first sending unit 604 is further configured to send the set of characteristic information to the mobile device;

The first receiving unit 605 is further configured to receive the three-dimensional coordinate information of the matching feature included in the target feature subset returned by the mobile device;

The first obtaining unit 601 is further configured to obtain two-dimensional projection information of the matching feature included in the target feature subset on the image to be processed;

The first processing unit 602 is further configured to use a preset algorithm to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the position of the wearable device. Posture information.

In some embodiments, the first receiving unit 605 is further configured to receive a set of scene characteristic information sent by the mobile device;

The first processing unit 602 is further configured to perform feature matching between the set of scene feature information and the set of feature information, determine a target feature subset, and obtain three-dimensional coordinate information of the matching feature included in the target feature subset And the two-dimensional projection information of the matching feature on the image to be processed; and using a preset algorithm to perform pose to the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset Calculate to obtain the pose information of the wearable device.

In some embodiments, the preset algorithm is a PNP algorithm.

In some embodiments, the first acquisition unit 601 is specifically configured to perform image acquisition through a camera of the wearable device to obtain the image to be processed.

In some embodiments, the first processing unit 602 is further configured to perform feature detection on the image to be processed to obtain a set of detection features of the image to be processed; and perform feature description on the set of detection features to obtain The set of descriptors associated with the detection feature;

The first determining unit 603 is further configured to determine the set of detection features and the descriptor associated with the set of detection features as a set of feature information of the image to be processed.

It is understandable that, in the embodiments of the present application, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, may also be a module, or may be non-modular. Moreover, the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or It is said that the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, the embodiments of the present application provide a computer storage medium, which is applied to the wearable device 60, and the computer storage medium stores a computer program that, when executed by a first processor, implements any of the foregoing embodiments. The method described.

Based on the composition of the above-mentioned wearable device 60 and the computer storage medium, refer to FIG. 7, which shows a schematic diagram of a specific hardware structure of a wearable device 60 provided by an embodiment of the present application. As shown in FIG. 7, it may include: a first communication interface 701, a first memory 702, and a first processor 703; various components are coupled together through a first bus system 704. It can be understood that the first bus system 704 is used to implement connection and communication between these components. In addition to the data bus, the first bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity of description, various buses are marked as the first bus system 704 in FIG. 7. in,

The first communication interface 701 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

The first memory 702 is configured to store a computer program that can run on the first processor 703;

The first processor 703 is configured to execute: when the computer program is running:

Obtain the image to be processed;

Performing feature processing on the image to be processed, and determining a set of feature information of the image to be processed;

According to the set of characteristic information, the pose information of the wearable device is determined.

It can be understood that the first memory 702 in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically available Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be a random access memory (Random Access Memory, RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous Link Dynamic Random Access Memory (Synchlink DRAM, SLDRAM) And Direct Rambus RAM (DRRAM). The first memory 702 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

The first processor 703 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the first processor 703 or instructions in the form of software. The above-mentioned first processor 703 may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the first memory 702, and the first processor 703 reads the information in the first memory 702, and completes the steps of the foregoing method in combination with its hardware.

It can be understood that the embodiments described in this application can be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, and others for performing the functions described in this application Electronic unit or its combination. For software implementation, the technology described in this application can be implemented through modules (for example, procedures, functions, etc.) that perform the functions described in this application. The software codes can be stored in the memory and executed by the processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the first processor 703 is further configured to execute the method described in any one of the foregoing embodiments when the computer program is running.

This embodiment provides a wearable device, which may include a first acquiring unit, a first processing unit, and a first determining unit. In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the virtual object of the mobile device and the wearable device.

In another embodiment of the present application, based on the same inventive concept as the foregoing embodiment, refer to FIG. 8, which shows a schematic diagram of the composition structure of a mobile device 80 provided by an embodiment of the present application. As shown in FIG. 8, the mobile device 80 may include: a second acquiring unit 801, a second processing unit 802, and a second determining unit 803; wherein,

The second acquiring unit 801 is configured to acquire sensor data of the mobile device;

The second processing unit 802 is configured to perform SLAM processing on the sensor data to construct a scene map;

The second determining unit 803 is configured to determine the pose information of the mobile device based on the scene map.

In some embodiments, referring to FIG. 8, the mobile device 80 may further include a second sending unit 804 and a second receiving unit 805; wherein,

The second receiving unit 805 is configured to receive the set of characteristic information sent by the wearable device;

The second processing unit 802 is further configured to calculate the pose information of the wearable device based on the scene map and the set of feature information;

The second sending unit 804 is configured to send the pose information of the wearable device to the wearable device.

In some embodiments, the second determining unit 803 is further configured to determine a set of scene feature information in the scene map according to the sensor data;

The second processing unit 802 is further configured to perform feature matching between the set of scene feature information and the set of feature information to determine a target feature subset; and is also configured to obtain matching features included in the target feature subset Three-dimensional coordinate information and two-dimensional projection information on the image to be processed of the wearable device; and using a preset algorithm to match the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset Perform pose calculation to obtain pose information of the wearable device.

In some embodiments, the set of feature information includes a set of detection features and a descriptor associated with the set of detection features;

The second processing unit 802 is specifically configured to perform feature matching between the set of detected features and the scene features in the set of scene feature information based on the calculated distance between the descriptor and the descriptor, and determine the target feature set.

In some embodiments, the preset algorithm is a PNP algorithm.

In some embodiments, the second receiving unit 805 is further configured to receive the set of characteristic information sent by the wearable device;

The second determining unit 803 is further configured to determine a set of scene feature information in the scene map according to the sensor data;

The second processing unit 802 is further configured to perform feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

The second sending unit 804 is further configured to send the three-dimensional coordinate information of the matching feature included in the target feature subset to the wearable device.

In some embodiments, the second determining unit 803 is further configured to determine a set of scene feature information in the scene map according to the sensor data;

The second sending unit 804 is further configured to send the set of scene characteristic information to the wearable device.

It can be understood that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, or it may also be non-modular. Moreover, the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, this embodiment provides a computer storage medium, which is applied to the mobile device 80, and the computer storage medium stores a computer program. When the computer program is executed by the second processor, any one of the foregoing embodiments is implemented. The method described.

Based on the composition of the mobile device 80 and the computer storage medium described above, refer to FIG. 9, which shows a schematic diagram of a specific hardware structure of the mobile device 80 provided in an embodiment of the present application. As shown in FIG. 9, it may include: a second communication interface 901, a second memory 902, and a second processor 903; various components are coupled together through a second bus system 904. It can be understood that the second bus system 904 is used to implement connection and communication between these components. In addition to the data bus, the second bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear description, various buses are marked as the second bus system 904 in FIG. 9. in,

The second communication interface 901 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

The second memory 902 is configured to store a computer program that can run on the second processor 903;

The second processor 903 is configured to execute: when the computer program is running:

Acquiring sensor data of the mobile device;

Performing SLAM processing on the sensor data to construct a scene map;

Based on the scene map, the pose information of the mobile device is determined.

Optionally, as another embodiment, the second processor 903 is further configured to execute the method described in any one of the foregoing embodiments when the computer program is running.

It can be understood that the hardware functions of the second memory 902 and the first memory 702 are similar, and the hardware functions of the second processor 903 and the first processor 703 are similar; the details are not described herein again.

This embodiment provides a mobile device, which includes a second acquiring unit, a second processing unit, and a second determining unit. In this way, since SLAM runs on mobile devices and builds scene maps, wearable devices no longer build maps but only execute positioning algorithms; this not only reduces the power consumption of wearable devices, but also reduces system latency, thereby increasing The visual quality of the wearable device is improved; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the interaction effect between the mobile device and the virtual object of the wearable device.

It should be noted that in this application, the terms "including", "including" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements , And also include other elements not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority or inferiority of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of this application, for the pose tracking of the wearable device, since SLAM runs on the mobile device and builds the scene map, the wearable device no longer builds the map, only executes the positioning algorithm; this not only reduces the wearable device The power consumption also reduces the system delay, thereby improving the visual quality of the wearable device; in addition, the mobile device and the wearable device are located in the same coordinate system to perform pose tracking, which can also improve the virtual objects of the mobile device and the wearable device The interaction effect between.

Claims (19)

1. A pose tracking method, applied to a wearable device, the method includes:

   Obtain the image to be processed;

   Performing feature processing on the image to be processed, and determining a set of feature information of the image to be processed;

   According to the set of characteristic information, the pose information of the wearable device is determined.
2. The method according to claim 1, wherein the determining the pose information of the wearable device according to the set of characteristic information comprises:

   Sending the set of characteristic information to a mobile device;

   Based on the response of the mobile device, receiving the pose information of the wearable device returned by the mobile device.
3. The method according to claim 1, wherein the determining the pose information of the wearable device according to the set of characteristic information comprises:

   Sending the set of feature information to a mobile device, and receiving the three-dimensional coordinate information of the matching feature included in the target feature subset returned by the mobile device;

   Acquiring two-dimensional projection information of the matching feature included in the target feature subset on the image to be processed;

   A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.
4. The method according to claim 1, wherein the determining the pose information of the wearable device according to the set of characteristic information comprises:

   Receive a set of scene characteristic information sent by the mobile device;

   Perform feature matching between the set of scene feature information and the set of feature information, determine a target feature subset, and obtain the three-dimensional coordinate information of the matching feature included in the target feature subset and the matching feature in the waiting list. Process the two-dimensional projection information on the image;

   A preset algorithm is used to perform pose calculation on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset to obtain the pose information of the wearable device.
5. The method according to claim 3 or 4, wherein the preset algorithm is a multi-point perspective PNP algorithm.
6. The method according to claim 1, wherein said obtaining the image to be processed comprises:

   Image acquisition is performed through the camera of the wearable device to obtain the to-be-processed image.
7. The method according to claim 1, wherein said performing feature processing on the image to be processed and determining a set of feature information of the image to be processed comprises:

   Performing feature detection on the image to be processed, and acquiring a set of detection features of the image to be processed;

   Perform feature description on the set of detection features, and obtain descriptors associated with the set of detection features;

   The set of detection features and the descriptor associated with the set of detection features are determined as a set of feature information of the image to be processed.
8. A pose tracking method, applied to a mobile device, and the method includes:

   Acquiring sensor data of the mobile device;

   Performing synchronous positioning and map construction SLAM processing on the sensor data to construct a scene map;

   Based on the scene map, the pose information of the mobile device is determined.
9. The method according to claim 8, wherein the method further comprises:

   Receiving the set of characteristic information sent by the wearable device;

   Calculating the pose information of the wearable device based on the scene map and the set of feature information;

   Sending the pose information of the wearable device to the wearable device.
10. The method according to claim 9, wherein the calculating the pose information of the wearable device based on the scene map and the set of feature information comprises:

    Determine a set of scene feature information in the scene map according to the sensor data;

    Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

    Acquiring three-dimensional coordinate information of the matching feature included in the target feature subset and two-dimensional projection information on the image to be processed of the wearable device;

    The pose calculation is performed on the three-dimensional coordinate information and the two-dimensional projection information of the matching feature included in the target feature subset by using a preset algorithm to obtain the pose information of the wearable device.
11. The method according to claim 10, wherein the set of feature information includes a set of detection features and a descriptor associated with the set of detection features;

    Correspondingly, the feature matching the set of scene feature information with the set of feature information to determine the target feature subset includes:

    Based on the calculated distance between the descriptor and the descriptor, feature matching is performed on the set of detected features and the scene features in the set of scene feature information to determine the target feature subset.
12. The method according to claim 10 or 11, wherein the preset algorithm is a multi-point perspective PNP algorithm.
13. The method according to claim 8, wherein the method further comprises:

    Receiving the set of characteristic information sent by the wearable device;

    Determine a set of scene feature information in the scene map according to the sensor data;

    Performing feature matching between the set of scene feature information and the set of feature information to determine a target feature subset;

    The three-dimensional coordinate information of the matching feature included in the target feature subset is sent to the wearable device.
14. The method according to claim 8, wherein the method further comprises:

    Determine a set of scene feature information in the scene map according to the sensor data;

    Sending the set of scene characteristic information to the wearable device.
15. A wearable device includes a first acquisition unit, a first processing unit, and a first determination unit; wherein,

    The first acquiring unit is configured to acquire an image to be processed;

    The first processing unit is configured to perform feature processing on the image to be processed, and determine a set of feature information of the image to be processed;

    The first determining unit is configured to determine the pose information of the wearable device according to the set of characteristic information.
16. A wearable device, the wearable device includes a first memory and a first processor; wherein,

    The first memory is configured to store a computer program that can run on the first processor;

    The first processor is configured to execute the method according to any one of claims 1 to 7 when running the computer program.
17. A mobile device including a second acquiring unit, a second processing unit, and a second determining unit; wherein,

    The second acquiring unit is configured to acquire sensor data of the mobile device;

    The second processing unit is configured to perform SLAM processing on the sensor data to construct a scene map;

    The second determining unit is configured to determine the pose information of the mobile device based on the scene map.
18. A mobile device, the mobile device includes a second memory and a second processor; wherein,

    The second memory is configured to store a computer program that can run on the second processor;

    The second processor is configured to execute the method according to any one of claims 8 to 14 when running the computer program.
19. A computer storage medium, wherein the computer storage medium stores a computer program that, when executed by a first processor, implements the method according to any one of claims 1 to 7, or is executed by a second processor When executed, the method according to any one of claims 8 to 14 is realized.

Images