WO2025199844A1

WO2025199844A1 - Motion tracking with neural networks and depth images

Info

Publication number: WO2025199844A1
Application number: PCT/CN2024/084280
Authority: WO
Inventors: Xin Zhang; Shandong WANG; Xuhui YAN; Senhui Zhang; Anbang YAO
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2024-03-28
Filing date: 2024-03-28
Publication date: 2025-10-02
Anticipated expiration: 2026-09-28

Abstract

The motion of an object in a local area may be tracked using depth images captured by cameras arranged at predetermined locations in the local area (e.g., on the ceiling). Point clouds may be extracted from the depth images and fused into one point cloud. The point cloud may be encoded into a low-dimensional feature map, which may be combined with a template feature map that encodes reference locations of joints of the object. The combined feature map may be processed by a neural network (e.g., a graph convolutional network) to regress 3D locations of the joints. The 3D locations of the joints may indicate an estimated 3D pose of the object. Estimated 3D poses at different times may constitute the estimated motion of the object, which may be used to facilitate immersive projection of content items (e.g., image, video, audio, etc.) in the local area.

Description

MOTION TRACKING WITH NEURAL NETWORKS AND DEPTH IMAGES

Technical Field

This disclosure relates generally to neural networks (also referred to as “deep neural networks” or “DNNs” ) , and more specifically, to motion tracking with DNNs and depth images.

Background

The last decade has witnessed a rapid rise in artificial intelligence (AI) based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. Many DNNs, such as convolutional networks, can extract features from images for predicting classifications or states of objects captured in the images.

Brief Description of the Drawings

Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

FIG. 1 is a block diagram of a motion tracking system, in accordance with various embodiments.

FIG. 2 is a block diagram of an imaging module, in accordance with various embodiments.

FIG. 3 illustrates an example local area where content is displayed to a person based on estimated motions of the person, in accordance with various embodiments.

FIG. 4 illustrates an example motion tracking process, in accordance with various embodiments.

FIG. 5 illustrates an example template feature map, in accordance with various embodiments.

FIG. 6 illustrates 3D joint locations representing an estimated three-dimensional (3D) pose of a person, in accordance with various embodiments.

FIG. 7 illustrates an example DNN, in accordance with various embodiments.

FIG. 8 illustrates an AI-based motion tracking environment, in accordance with various embodiments.

FIG. 9 is a flowchart showing a method of motion tracking, in accordance with various embodiments.

FIG. 10 is a block diagram of an example computing device, in accordance with various embodiments.

Detailed Description

Overview

A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as “neural network operations” ) , such as matrix multiplication, convolution, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (1D) tensor) , matrix (which is two-dimensional (2D) tensor) , three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. The input to a convolution layer may include an input tensor (also referred to as “activation tensor” or “input feature map (IFM) ” ) that includes one or more activations (also referred to as “input elements” ) and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor) , a filter (a 3D weight tensor) , or a group of filters (a 4D weight tensor) . The output of a convolutional layer may be an output tensor, which is also referred to as output feature map (OFM) . The output tensor may be used as the input tensor of the next layer in the DNN.

Immersive projection systems are widely used in many applications. In an example, a room may be equipped with one or more projectors that can project computer-generated content onto one or more screens in the room so that people in the room can have an immersive and stereo experience. A projector or screen may be mounted on a ceiling, wall, or floor of the room. In some scenarios, a ceiling, wall, or floor of the room can function as a screen. However, many currently available immersive projection systems have no or very weak interaction capability due to the challenging human motion capture in this environment. Projection systems with strong interaction capability can allow people to interactively control the virtual content by their body posture or moving position in real-time, which will largely enhance the user experience and improve product values. For body motion tracking, many currently available motion capture methods are rarely considered because they would limit the users' movement and the cost is usually high. Also, some currently available projection methods suffer from low tracking accuracy as the camera images can be heavily affected by the projection effects, based on which accurate person detection and pose tracking can be hard to achieve.

Some currently available techniques can estimate 3D human pose from point clouds. For instance, three unsupervised losses may be used for learning 3D human key points from in-the-wild point clouds without any human labels. A cascaded architecture may be used to enhance point feature extraction from challenging point clouds. A 3D pose regression network may be trained in an end-to-end manner to extract body features and regress 3D joint locations. Such techniques may produce appropriate 3D key points results when the input human point clouds are complete and have minimal or no noise. However, in many environments, people are captured by multiple cameras overhead, and the extracted human point clouds are usually incomplete due to severe self-occlusion. Therefore, these currently available techniques often produce incorrect 3D results.

Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by using DNNs to track motions of objects in local areas based on depth images captured in the local areas. A motion tracking process may include a process of estimating 3D poses of the object, such as a process of regressing 3D locations of the deformed skeleton of the object.

In various embodiments of the present disclosure, a motion tracking system is used to track motions of objects in local areas. Examples of local areas include rooms, buildings, open areas, or other types of relatively small graphical areas. Examples of objects include people, robots, vehicles, animals, and so on. The motion tracking system may include both hardware (e.g., depth cameras) and software (e.g., modules for using DNNs to estimate 3D poses based on depth images captured by the depth cameras) . In an example, depth cameras may be arranged in a local area (e.g., a room) to capture depth images of an object in the local area. The depth cameras may be mounted on the ceiling of the room and point downwards. One or more depth cameras may be tilted.

Point clouds may be extracted from the depth images. Point clouds extracted from different depth images may be fused into one point cloud. A DNN (e.g., a point-based DNN) may be used to encode the point cloud into a low-dimensional feature map. The feature map may be a feature vector. The feature map may further be combined with a template feature map that encodes a template structure of the object. The template structure may represent a reference skeleton of the object that includes joints of the object and connections between the joints. As the object moves and changes pose, the skeleton of the object may be changed. Skeletons different from the reference skeleton may be referred to as deformed skeletons. The combined feature map may be processed by another DNN (e.g., a graph convolutional network) to regress the 3D locations of the joints. The 3D locations of the joints may be coordinates of the joints in a 3D space. The usage of the template feature map can improve the accuracy and robustness of learning the 3D locations of the joints from the point cloud. The motion tracking system can predict the full and reasonable 3D skeleton joints from incomplete and noisy point clouds from the depth cameras. The 3D locations of the joints of the object indicate the estimated 3D pose of the object at a time. Estimated 3D poses at different times may constitute the estimated motion of the object.

The estimated motion of the object may be used for dynamically presenting content in the local area. In examples where the object is a person, the estimated motion may be used to facilitate interaction between the person and the presented content. One or more projectors may also be present in the local area to project computer-generated content. The depth cameras and projector (s) may be fixed in a box-like device, which may be installed at the center of the ceiling. The depth cameras may capture depth images of an object.

The present disclosure provides a more efficient approach for tracking motion and estimating 3D poses. The approach can simplify the hardware setup and enables accurate 3D human body motion tracking to enhance real-time interaction capability. It can improve robustness and practicability for various 3D motion tracking applications, including immersive projection, augmented reality, virtual reality, sports analysis, telepresence, film and game production, action recognition, and so on.

For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or/and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

For the purposes of the present disclosure, the phrase “A or B” or the phrase "A and/or B" means (A) , (B) , or (A and B) . For the purposes of the present disclosure, the phrase “A, B, or C” or the phrase "A, B, and/or C" means (A) , (B) , (C) , (A and B) , (A and C) , (B and C) , or (A, B, and C) . The term "between, " when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

The description uses the phrases "in an embodiment" or "in embodiments, " which may each refer to one or more of the same or different embodiments. The terms "comprising, " "including, " "having, " and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above, " "below, " "top, " "bottom, " and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives “first, ” “second, ” and “third, ” etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

The terms “substantially, ” “close, ” “approximately, ” “near, ” and “about, ” generally refer to being within +/-20%of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., “coplanar, ” “perpendicular, ” “orthogonal, ” “parallel, ” or any other angle between the elements, generally refer to being within +/-5-20%of a target value as described herein or as known in the art.

In addition, the terms “comprise, ” “comprising, ” “include, ” “including, ” “have, ” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term “or” refers to an inclusive “or” and not to an exclusive “or. ”

The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

Example Motion Tracking System

FIG. 1 is a block diagram of a motion tracking system 100, in accordance with various embodiments. The motion tracking system 100 uses a deep learning approach to estimate 3D poses of objects from depth images of the objects. The motion tracking system 100 includes an interface module 110, depth cameras 120 (individually referred to as “depth camera 120” ) , an imaging module 130, a point cloud generator 140, a neural network 150, a concatenator 160, another neural network 170, an output module 180, and a datastore 190. In other embodiments, alternative configurations, different or additional components may be included in the motion tracking system 100. Further, functionality attributed to a component of the motion tracking system 100 may be accomplished by a different component included in the motion tracking system 100 or by a different module.

The interface module 110 facilitates communications of the motion tracking system 100 with other systems, devices, or modules. For example, the interface module 110 may receive images from an online system (e.g., a social media system, an online image gallery, an online search tool, etc. ) , a device, and so on. As another example, the interface module 110 may receive one or more data for training or testing the neural network 150 or 170. Yet another example, the interface module 110 may transmit data generated by the motion tracking system 100 to other systems, devices, or modules. For instance, the interface module 110 may transmit estimated 3D poses of objects, animations of objects, motion analysis results, or other types of motion tracking information to other systems, devices, or modules.

The depth cameras 120 captures depth images. The depth cameras 120 may be arranged in a local area and infer the depth of points in the local area. The depth cameras 120 may output depth images. A depth image may include depth pixels, each of which may encode a distance of a point from the depth camera 120. In some embodiments, the depth cameras 120 are placed at a particular place in the local area. The depth cameras 120 may be set to capture depth images at different angles. For instance, one or more depth cameras 120 may be tilted.

The imaging module 130 may control and manage the depth cameras 120. In some embodiments, the imaging module 130 may determine how many depth cameras 120 to be placed in a local area. The imaging module 130 may also determine where to place the depth cameras 120. The imaging module 130 may also synchronize the depth cameras 120 so that the depth cameras 120 may capture depth images at the same time. The imaging module 130 may deploy the depth cameras, e.g., by commanding the depth cameras to capture depth images. The imaging module 130 may receive depth images from the depth cameras. Different depth cameras may capture depth images simultaneously. These depth images may capture different portions of the local area. In some embodiments, there may be overlaps between depth images captured by different depth cameras. For instance, a first depth image and a second depth image from two adjacently located depth cameras may both capture a first portion of the local area, while the first depth image may also capture a second portion of the local area (which is not captured by the second depth image) and the other depth image may capture a third portion of the local area (which is not captured by the first depth image) . The imaging module 130 may provide the depth images for further processing. Certain aspects of the imaging module 130 are described below in conjunction with FIG. 2.

The point cloud generator 140 generates point clouds from depth images received from the imaging module 130. In some embodiments, the point cloud generator 140 may segment each depth image into depth pixels representing the target object ( “object segmentation pixels” ) and background depth pixels. The background depth pixels may represent other objects or features of the local area. The point cloud generator 140 may remove background depth pixels from the depth images. In an embodiment, the point cloud generator 140 may define a depth threshold and select depth pixels with values not meeting the depth threshold (e.g., above the depth threshold or below the depth threshold) . In another embodiment, the point cloud generator 140 may compare the depth images with a 3D reference model of the local area. The 3D reference model of the local area may represent the local area when the object is not present in the local area. The point cloud generator 140 may label depth pixels in the depth images that matches one or more features of the 3D reference model as background depth pixels. Additionally or alternatively, the point cloud generator 140 may use the Euclidean cluster extraction to further improve the segmentation process.

After the point cloud generator 140 extracts the object segmentation pixels, the point cloud generator 140 may convert the extracted pixels to 3D point clouds in camera spaces. The point cloud generator 140 may define the camera spaces based on camera intrinsic parameters, which may be determined by the imaging module 130. In some embodiments, the point cloud generator 140 may generate multiple point clouds, each point cloud generated from depth pixels extracted from a different depth image. Each point cloud may be in the camera space determined based on the intrinsic parameters of the depth camera that captured the corresponding depth image. The point cloud generator 140 may also transfer these point clouds to a unique world space based on extrinsic parameters of the depth cameras. The point cloud generator 140 may further fuse the point clouds into a single point cloud. The fused point cloud may be a complete point cloud of the object.

In some embodiments, the point cloud generator 140 may downsize the fused point cloud. For instance, the point cloud generator 140 may determine whether the total number of points in the fused point cloud is beyond a predetermined number. In response to determining that the total number of points in the fused point cloud is beyond the predetermined number, the point cloud generator 140 may down-sample the fused point cloud to generate a point cloud that has the predetermined number of points. In an example, the predetermined number may be 5000. Each point in the point cloud may have a 3D coordinate that indicates the position of the point in the world space. The point cloud generator 140 may also perform a normalization operation on the 3D coordinates of the points so that the normalized 3D coordinates of the points may indicate the locations of the points in a unit cube space. The point cloud in the unit cube space may be represented as P= {p₁, p₂, …, p_N} . The point cloud generator 140 may provide the point cloud to the neural network 150 for further processing.

The neural network 150 generates a feature map from the point cloud generated by the point cloud generator 140. In some embodiments, the neural network 150 may be a point-based DNN. The neural network 150 may receive the point cloud as an input and extracts features from the point cloud to generate the feature map. In some embodiments, the feature map has less dimensions than the point cloud. For instance, the feature map may have 1024 dimensions. The feature map may include a low-dimensional global features extracted by the neural network 150 from the point cloud. The neural network 150 may output the feature map and provides the feature map to the concatenator 160.

The concatenator 160 receives the feature map from the neural network 150 and a template feature map of the object. The template feature map may encode a reference structure of the object, such as a reference skeleton structure. The reference structure may include key points of the object and connections between the key points. A key point may be a joint of the object. In an example where the object is a person, the key points may include bone joints of the person. The connections between the key points may be determined based on the connections of the corresponding bone joints. The reference structure may represent a reference pose of the object. In some embodiments, the template feature map may have a spatial size of J×3, where J is the number of key point and each key point has a 3D coordinate indication the location of the key point. The concatenator 160 may perform a concatenation operation on the feature map from the neural network 150 and the template feature map of the object to generate a combined feature map.

In some embodiments, the combined feature map may be a tensor having a spatial size of J× (3+M) , where M is the number of dimensions in the feature map from the neural network 150. In an example where M=1024, the combined feature map may have a spatial size of J×1027. Through the concatenation operation, the feature map from the neural network 150 may be embedded in the template feature map. The combined feature map may include a graph structure that includes nodes and edges. A node may correspond to a key point in the template feature map. An edge may represent the connection between two or more key points in the template feature map. The concatenator 160 may provide the combined feature map to the neural network 170.

The neural network 170 receives the combined feature map as an input and estimates a 3D pose of the object using the combined feature map. In some embodiments, the neural network 170 may be a convolutional network, such as a graph convolutional network. In an example, the neural network 170 may be a semantic graph convolutional network that includes one or more semantic graph convolutional layers. The neural network 170 may extract features of the object from the combined feature map and regress 3D key point locations. The neural network 170 may output a tensor having a spatial size of J×3 that encodes the 3D locations of J key points of the object. The 3D locations of the J key points can define the estimated 3D pose of the object. In some embodiments, the 3D location of the key points may be represented as M= {j₁, j, …, j_J} .

Even though the neural networks 150 and 170 are shown as separate neural networks in FIG. 1, the neural networks 150 and 170 may be two portions of a single DNN. In some embodiments, the DNN may be trained with an end-to-end manner. For instance, training samples may be input into the neural network 150 and processed by the neural networks 150 and 170, and the internal parameters of the neural networks 150 and 170 may be adjusted based on a loss determined based on the outputs of the neural network 170 and ground-truth labels of the training samples. A training sample may be a point cloud representing an object. The ground-truth label of the training sample may be a known or verified 3D pose of the object. The ground-truth labels may be 3D key point labels. In some embodiments, the training samples are generated from depth images captured by depth cameras mounted on the ceiling of the local area, while the ground-truth labels may be generated using depth cameras captured by depth cameras mounted on both the ceiling and side walls of the local area. The ground-truth labels may be generated by fitting a parametric 3D model onto point clouds extracted from those depth images and using iterative optimization algorithms to estimate 3D shape and pose of objects, from which 3D key point coordinates can be computed and then used as the ground-truth labels.

The output module 180 processes 3D key point coordinates regressed by the neural network 170. In some embodiments, the output module 180 may transform the 3D key point coordinates from a canonical space to the world space. Additionally or alternatively, the output module 180 may perform filters (e.g., online filters) to smooth the 3D coordinates. The output module may also compute 3D key point angles or 3D rotations based on the locations of the key points. The output module 180 may output the estimated 3D pose of the object.

The output module 180 may also control projectors in the local area based on tracked motions of the object. In some embodiments, the output module 180 may control content items presented in the local area based on the estimated 3D pose of the object. For instance, the output module 180 may generate content items or modify content items based on the estimated 3D pose of the object. The output module 180 may command projectors in the local area to present the generated or modified content items. Content items may include image, video, audio, text, other types of content, or some combination thereof. The output module 180 can facilitate interactions of users with content items presented by the projectors. That way, the motion tracking results may be used to facilitate real-time interactive control of the projection system.

In some embodiments, users may use their hand gestures to switch content displays, use the body gestures to control the digital avatar to generate the animation, and so on. In an example, the output module 180 may zoom in an image or video presented to a person after it detects that the person made a movement indicating zooming in. As another example, the output module 180 may present a different image or video to a person after it detects that the person made a gesture indicating movement to the next content item. As yet another example, a game player can use their body movements to control actions of one or more virtual characters in the game. The output module 180 may determine various parameters of the estimated motion of the game player (e.g., speed, direction, timing, height of jump, trajectory, etc. ) and replicate the motion of the game player in the game.

The datastore 190 stores data associated with the motion tracking system 100, such as data received, generated, or used by components of the motion tracking system 100. For instance, the datastore 190 may store camera parameters (e.g., intrinsic parameters, extrinsic parameters, etc. ) of the annotation network. The datastore 190 may also store training data or validation data used to train or validate the neural networks 150 and 170. The datastore 190 may further store images received by the interface module 110, outputs of the neural networks 150 and 170, graphs representing estimated 3D poses generated by the output module 180, content items generated by the output module 180, and so on. In some embodiments, the motion tracking system 100 may include or be associated with more than one datastore. The datastore 190 may be implemented as a random-access memory (RAM) , such as a static RAM (SRAM) , disk storage, nearline storage, online storage, offline storage, and so on.

FIG. 2 is a block diagram of an imaging module 200, in accordance with various embodiments. The imaging module 200 may control and manage depth cameras in a local area. The imaging module 200 is an example of the imaging module 130 in FIG. 1. As shown in FIG. 2, the imaging module 200 includes a configuration module 210, a placement module 220, a calibration module 230, and a synchronization module 240. In other embodiments, alternative configurations, different or additional components may be included in the imaging module 200. Further, functionality attributed to a component of the imaging module 200 may be accomplished by a different component included in the imaging module 200 or by a different module.

The configuration module 210 may configure an immersive projection device for the local area. For instance, the configuration module 210 may define an immersive space in the local area for placing the immersive projection device. The immersive space may occupy the entire local area or a portion of the local area. The immersive space may have a shape, such as cuboid, cube, cylinder, hexagonal prism, or other shapes. The immersive projection device may include depth cameras (e.g., depth cameras 120 in FIG. 1) and one or more projectors that will be placed in various locations inside the immersive space. In some embodiments, the configuration module 210 may determine the dimensions of the immersive space. In an example where the immersive space may be a cuboid or cube space, the configuration module 210 may determine the length, width, or height of the space. For a cylindrical or hexagonal space, the configuration module 210 may determine the radius or height of the space. In some embodiments, the configuration module 210 may determine that the length and width of the immersive space should be the same or similar. In some embodiments, the configuration module 210 may determine that the length of the immersive space is between approximately three meters and approximately eight meters.

The configuration module 210 may determine constraints of the space for the immersive projection device. The configuration module 210 may determine that an area ratio is at least approximately 5%. The area ratio may be a ratio of the area of the immersive space to an area of the local area, such as the area of the ceiling of the local area. The configuration module 210 may also determine that a height ratio is between approximately 5%and approximately 20%. The height ratio may be a ratio of the height of the immersive space to the height of the local area. The configuration module 210 may also determine a minimum distance from the immersive space to one or more edges of the local area. In some embodiments, the minimum distance is denoted as where L denotes the length of the immersive space and W denotes the width of the immersive space. The configuration module 210 may further determine that a ratio of a person’s height to the heigh of the local area is at least approximately 2. The person may be the object whose motion is tracked using depth images captured by the depth cameras or may be presented computer-generated content by the projectors.

In some embodiments, the configuration module 210 may create a 3D immersive space by employing synchronized multi-channel video or projection technology and stereoscopic optoelectronic techniques. Users may be enveloped in the immersive space. The immersive space can provide the users with an immersive projection environment, which may be defined by a stereoscopic interface. The immersive projection device may be integrated with a speaker system and posture capture system, resulting in a lifelike, high-resolution 3D visual experience and a 360-degree interactive encounter. This setup can allow the user to be immersed in a natural environment, enhancing the overall sensory experience.

The placement module 220 may control placement of depth cameras in the space defined by the configuration module 210. In some embodiments, the placement module 220 may determine the number of depth cameras to be placed in the local area based on the shape or dimensions of the immersive space. For instance, the placement module 220 may determine that a cube room would need at least 5 depth cameras, a cylindrical room may need at least two depth cameras, or a hexagonal room may need at least six depth cameras. The placement module 220 may also determine the location and orientation (e.g., direction or position) of each of the depth cameras. In some embodiments, the placement module 220 may determine that N-1 depth cameras should have a downward direction, e.g., a direction from the ceiling to each wall, and a depth camera looking down perspective. Nmay be the total number of depth cameras in the local area. In an example where the immersive space is a cuboid or cube, the placement module 220 may distribute depth cameras on five faces of the immersive space to ensure full coverage of the space.

The placement module 220 may also set angles of the depth cameras to ensure that the depth cameras can capture the entire immersive space. In an example where the immersive space has a cube shape, the placement module 220 may determine that the camera angle on the side of the cube is titled downwards between approximately 30 degrees and approximately 55 degrees. The placement module 220 may adjust angles of depth cameras as the immersive space is changed. The angle between depth cameras (e.g., depth cameras mounted on the walls) may be 360°/ (N-1) . In some embodiments, the placement module 220 may ensure that there is sufficient overlap between the fields of view of adjacent depth cameras. The overlap between adjacent depth cameras may facilitation fusion of point clouds extracted from depth images generated by these depth cameras.

In some embodiments, the placement module 220 may control placement of projectors in the space defined by the configuration module 210. For instance, the placement module 220 may determine the number of projectors or characteristics of projects (e.g., projection range) based on the shape or dimensions of the immersive space. In an example, the placement module 220 may configure projectors to make sure the projection of content can cover the entire space of the local area or a specific portion of the local area.

The calibration module 230 calibrates the depth cameras in the local area. In some embodiments, the calibration module 230 may perform intrinsic or extrinsic calibration. The calibration module 230 may compute the intrinsic and extrinsic parameters of each depth camera. The calibration module 230 may use calibration tools such as a calibration board to determine accurate camera parameters. Camera parameters determined by the calibration module 230 may be used, e.g., by the point cloud generator 140, to extract point clouds from depth images captured by the depth cameras. The calibration module 230 may also calibrate projectors in the local area. In some embodiments, the calibration module 230 may calibrate projectors to correct the projection ranges and geometric distortions of the projectors.

The synchronization module 240 synchronizes depth cameras and projectors in the immersive projection device. In some embodiments, the synchronization module 240 may synchronize the depth cameras so that they can capture depth images simultaneously. That can make sure the depth cameras capture the same pose of the same object in the local area, which can facilitate tracking the motion of the object. For multi-camera synchronization, the synchronization module 240 may use depth cameras that have the external synchronization capabilities. In some embodiments, the synchronization module 240 may use hardware clocks, network time protocol, or other tools to ensure time synchronization among depth cameras and projectors.

Example Immersive Projection Based on Motion Tracking

FIG. 3 illustrates an example local area 300 where content is displayed to a person 310 based on estimated motions of the person 310, in accordance with various embodiments. For the purpose of illustration and simplify, the local area 300 is a room that has a cuboid shape. In other embodiments, the local area 300 may be a space within a larger area. For instance, the local area 300 may be an immersive space identified by the imaging module from a larger area.

As shown in FIG. 3, the local area 300 has a ceiling 301 and a floor 302. The local area 300 also has side walls between the ceiling 301 and floor 302. Depth cameras 320 (individually referred to as “depth camera 320” ) and one or more projectors (not shown in FIG. 3) are placed in the local area 300. The depth cameras 320 may be examples of the depth cameras 120 in FIG. 1. In FIG. 3, the depth cameras 320 are mounted on the ceiling 301 for capturing depth images of the person. The depth cameras 320 are placed at different spots on the ceiling 301 so that they can detect the entire local area 300. The depth cameras 320 may capture depth images with a downward direction. In some embodiments, the directions of different depth cameras 320 may be different. For instance, a depth camera 320 may be tiled for capturing depth images with a tilted angle, as opposed to the direction perpendicular to the ceiling 301 or the floor 302. Even though FIG. 3 shows three depth cameras 320, a different number of depth camera 320 may be used. The fields of view of the depth camera 320 may be overlapped, which can facilitate fusion of point clouds extracted from depth images captured by the depth cameras 320. The number of depth camera 320 installed in the local area 300, locations of the depth cameras 320, and configurations (e.g., fields of view, angle, etc. ) of the depth cameras 320 may be determined by the imaging module 130 in FIG. 1. The depth cameras 320 may be synchronized so that they can capture depth images simultaneously.

The motion of the person 310 may be detected based on depth images captured by the depth cameras 320. The detected motion of the person may be used to control the projectors. In FIG. 3, the projectors project an image 330 onto the side walls of the local area 300 so that the person 310 can view the image 330. The image 330 shows a scene in a bar where a person is sitting by a counter and a bar tender is making a drink. The image 330 may be a frame in a video, such as a movie, animation, game, and so on. The person can interact with the image by making gestures or other types of movements. For instance, the person 310 waves to the image 330. After the wave is detected, the image 330 may be changed to a new image showing the bar tender waving back to the person 310. In other embodiments, the person 310 may make other movements for controlling the projection differently.

Example Motion Tracking

FIG. 4 illustrates an example motion tracking process, in accordance with various embodiments. The process includes estimation of a 3D pose of an object in a local area. In some embodiments, the process may include estimation of a plurality of 3D poses of the object in a time window. The process may be performed by the motion tracking system 100 in FIG. 1. In the embodiments of FIG. 4, the process starts with depth images 401. The depth images 401 may be captured by multiple depth cameras placed in an immersion space within a local area. In some embodiments, the immersion space may include at least part of the ceiling of the local area.

The depth images 401 are converted to a point cloud 402 by a point extractor 410. The point cloud 402 may capture at least part of the object. The point cloud 402 may include a plurality of points in a 3D space having a shape of a cube. An example of the point extractor 410 may be the point cloud generator 140 in FIG. 1. The point cloud 402 is converted to a global feature map 403 by a point encoder 420. The global feature map 403 may be a lower-dimensional feature map compared with the point cloud 402. For instance, the total number of dimensions in the global feature map 403 may be less than the total number of points in the point cloud 402. An example of the pointer encoder 420 may be the neural network 150 in FIG. 1.

The global feature map 403 is combined with a template feature map 404 by a concatenator 430, which generates a concatenated feature map 405. An example of the concatenator 430 may be the concatenator 160 in FIG. 1. The template feature map 404 may encodes 3D locations of a number J key points of the object. The 3D locations of the joints may represent a reference structure of the object. The concatenator 430 may embed the global feature map 403 into the template feature map 404. In some embodiments, the concatenator 430 may combine the global feature map 403 and the template feature map 404 in a dimension. That dimension of the concatenated feature map 405 may equal the sum of the corresponding dimension of the global feature map 403 and the corresponding dimension of the template feature map 404. Another dimension of the concatenated feature map 405 may be J.

3D joint locations 406 are regressed from the concatenated feature map 405 by a DNN 440. An example of the DNN 440 may be the neural network 170 in FIG. 1. In some embodiments, the 3D joint locations 406 may represent a deformed structure of the object. The deformed structure may be a structure that is different from a reference structure of the object represented by the template feature map 404. As the object moves or the pose of the object changes, the 3D location of at least one key point may change. The 3D joint locations 406 may constitute the estimation of the pose of the object that is captured in the depth images 401. Even though not shown in FIG. 4, the 3D joint locations 406 may be further used in content presentation applications, such as immersive projection, virtual reality, augmented reality, mixed reality, and so on.

FIG. 5 illustrates an example template feature map 500, in accordance with various embodiments. The template feature map 500 represents a human skeleton structure. The template feature map 500 may be used as a reference structure or reference pose for tracking motions of people. In some embodiments, the template feature map 500 may be created for tracking motions of a particular person or a particular group of people. In other embodiments, the template feature map 500 may be created for tracking motions of any people. The template feature map 500 includes a plurality of key points 510, individually referred to as “key point 510. ” In some embodiments, each key point may correspond to a human bone joint. The key points 510 are connected using lines. The connections between the key points 510 may be determined based on the connections of the corresponding bone joints.

The template feature map 500 may be used as a representation of a 3D pose and a reference skeleton structure of a person, which may be defined by 3D locations of the key points 510. As the person moves, the location of at least one key point 510 may change, resulting in a different skeleton structure of the person. A skeleton structure different from the template feature map 500 may be referred to as a deformed structure. For the purpose of illustration, the template feature map 500 includes 21 key points 510 in FIG. 5. In other embodiments, the template feature map 500 may include different, fewer, or more key points 510. Also, the connections between the key points may be different from the connections shown in FIG. 5.

FIG. 6 illustrates 3D joint locations representing an estimated 3D pose of a person, in accordance with various embodiments. The 3D joint locations are represented by a graphical representation 610 in FIG. 6, which shows an estimation of a pose 620 of a person. The graphical representation 610 includes joints 630 (individually referred to as “joint 630” ) . The joints 630 are in a 3D space defined by the X, Y, and Z axes in FIG. 6. The location of a joint 630 may be the (X, Y, Z) coordinate of the joint 630 in the 3D space. A joint 630 may be a bone joint of the person. The pose 620 may be captured in one or more depth images, from which the 3D location of the joints 630 are determined using one or more DNNs. In some embodiments, the graphical representation 610 may be output from a DNN, e.g., the neural network 170 in FIG. 1 or the DNN 440 in FIG. 4.

Example DNN

FIG. 7 illustrates an example DNN 700, in accordance with various embodiments. At least part of the DNN 700 (or part of the DNN 700) may be an example of the neural network 150 or 170 in FIG. 1, the point encoder 420 in FIG. 4, or the DNN 440 in FIG. 4. In the embodiments of FIG. 7, the DNN 700 includes a sequence of layers comprising a plurality of convolutional layers 710 (individually referred to as “convolutional layer 710” ) , a plurality of pooling layers 720 (individually referred to as “pooling layer 720” ) , and a plurality of fully-connected layers 730 (individually referred to as “fully-connected layer 730” ) . In other embodiments, the DNN 700 may include fewer, more, or different layers. In an inference of the DNN 700, the layers of the DNN 700 execute tensor computation that includes many tensor operations, such as convolution (e.g., multiply-accumulate (MAC) operations, etc. ) , pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc. ) , other types of tensor operations, or some combination thereof.

The convolutional layers 710 summarize the presence of features in the input to the DNN 700. The convolutional layers 710 function as feature extractors. The first layer of the DNN 700 is a convolutional layer 710. In an example, a convolutional layer 710 performs a convolution on an input tensor 740 (also referred to as IFM 740) and a filter 750. As shown in FIG. 7, the IFM 740 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 740 includes 3 input channels, each of which is represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also referred to as input points) in each row and seven input elements in each column. The filter 750 is represented by a 3×3×3 3D matrix. The filter 750 includes 3 kernels, each of which may correspond to a different input channel of the IFM 740. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 7, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel includes 3 weights in each row and three weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 750 in extracting features from the IFM 740.

The convolution includes MAC operations with the input elements in the IFM 740 and the weights in the filter 750. The convolution may be a standard convolution 763 or a depthwise convolution 783. In the standard convolution 763, the whole filter 750 slides across the IFM 740. All the input channels are combined to produce an output tensor 760 (also referred to as OFM 760) . The OFM 760 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also referred to as output points) in each row and five output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 7. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 760.

The multiplication applied between a kernel-sized patch of the IFM 740 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernel-sized patch of the IFM 740 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the “scalar product. ” Using a kernel smaller than the IFM 740 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 740 multiple times at different points on the IFM 740. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 740, left to right, top to bottom. The result from multiplying the kernel with the IFM 740 one time is a single value. As the kernel is applied multiple times to the IFM 740, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 760) from the standard convolution 763 is referred to as an OFM.

In the depthwise convolution 783, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 7, the depthwise convolution 783 produces a depthwise output tensor 780. The depthwise output tensor 780 is represented by a 5×5×3 3D matrix. The depthwise output tensor 780 includes 3 output channels, each of which is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements in each row and five output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 740 and a kernel of the filter 750. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots) , the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips) , and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes) . In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 793 is then performed on the depthwise output tensor 780 and a 7×1×3 tensor 790 to produce the OFM 760.

The OFM 760 is then passed to the next layer in the sequence. In some embodiments, the OFM 760 is passed through an activation function. An example activation function is rectified linear unit (ReLU) . ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 710 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 760 is passed to the subsequent convolutional layer 710 (i.e., the convolutional layer 710 following the convolutional layer 710 generating the OFM 760 in the sequence) . The subsequent convolutional layers 710 perform a convolution on the OFM 760 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 710, and so on.

In some embodiments, a convolutional layer 710 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions F×F×D pixels) , the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time) , and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 710) . The convolutional layers 710 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 700 includes 76 convolutional layers 710. In other embodiments, the DNN 700 may include a different number of convolutional layers.

The pooling layers 720 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 720 is placed between two convolution layers 710: a preceding convolutional layer 710 (the convolution layer 710 preceding the pooling layer 720 in the sequence of layers) and a subsequent convolutional layer 710 (the convolution layer 710 subsequent to the pooling layer 720 in the sequence of layers) . In some embodiments, a pooling layer 720 is added after a convolutional layer 710, e.g., after an activation function (e.g., ReLU, etc. ) has been applied to the OFM 760.

A pooling layer 720 receives feature maps generated by the preceding convolution layer 710 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the CNN and avoids over-learning. The pooling layers 720 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map) , max pooling (calculating the maximum value for each patch of the feature map) , or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2×2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 720 applied to a feature map of 6×6 results in an output pooled feature map of 3×3. The output of the pooling layer 720 is inputted into the subsequent convolution layer 710 for further feature extraction. In some embodiments, the pooling layer 720 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

The fully-connected layers 730 are the last layers of the CNN. The fully-connected layers 730 may be convolutional or not. The fully-connected layers 730 may also be referred to as linear layers. In some embodiments, a fully-connected layer 730 (e.g., the first fully-connected layer in the DNN 700) may receive an input operand. The input operand may define the output of the convolutional layers 710 and pooling layers 720 and includes the values of the last feature map generated by the last pooling layer 720 in the sequence. The fully-connected layer 730 may apply a linear transformation to the input operand through a weight matrix. The weight matrix may be a kernel of the fully-connected layer 730. The linear transformation may include a tensor multiplication between the input operand and the weight matrix. The result of the linear transformation may be an output operand. In some embodiments, the fully-connected layer may further apply a nonlinear transformation (e.g., by using a nonlinear activation function) on the result of the linear transformation to generate an output operand. The output operand may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 7, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 730 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function.

Example AI-based Motion Tracking Environment

FIG. 8 illustrates an AI-based motion tracking environment 800, in accordance with various embodiments. The AI-based motion tracking environment 800 includes a motion tracking system 810, client devices 820 (individually referred to as client device 820) , and a third-party system 830. In other embodiments, the AI-based motion tracking environment 800 may include fewer, more, or different components. For instance, the AI-based motion tracking environment 800 may include a different number of client devices 820 or more than one third-party system 830.

The motion tracking system 810 tracks motions of objects in local areas. For instance, the motion tracking system 810 may track motions of an object by estimating 3D poses of the object from depth images captured by depth cameras placed in the local area where the object is located. The motion tracking system 810 may receive the depth images from the depth cameras, one or more client devices 820, or the third-party system 830. Also, the motion tracking system 810 may transmit information of estimated 3D poses of the object to one or more client devices 820 or the third-party system 830. Additionally or alternatively, the motion tracking system 810 may transmit content items generated using the estimated 3D poses of the object to one or more client devices 820 or the third-party system 830. An example of the motion tracking system 810 is the motion tracking system 100 in FIG. 1.

The client devices 820 are in communication with the motion tracking system 810. For example, the client device 820 may receive 3D pose graphical representations from the motion tracking system 810 and display the 3D pose graphical representations to one or more users associated with the client device 820. As another example, a client device 820 may facilitate an interface with one or more depth cameras in a local area and may send commands to the depth cameras to capture depth images to be used by the motion tracking system 810. Additionally or alternatively, the client device 820 may facilitate an interface with one or more projectors in a local area and may provide content items to the projectors for the projectors to present the content items in the local area. The client device 820 may generate the content items using motion tracking results from the motion tracking system 810. A client device may have one or more users, whose motions may be tracked by the motion tracking system 810.

In some embodiments, a client device 820 may execute one or more applications allowing one or more users of the client device 820 to interact with the motion tracking system 810. For example, a client device 820 executes a browser application to enable interaction between the client device 820 and the motion tracking system 810. In another embodiment, a client device 820 interacts with the motion tracking system 810 through an application programming interface (API) running on a native operating system of the client device 820, such as or ANDROID^TM.

A client device 820 may be one or more computing devices capable of receiving user input as well as transmitting and/or receiving data via the network 840. In one embodiment, a client device 820 is a conventional computer system, such as a desktop or a laptop computer. Alternatively, a client device 820 may be a device having computer functionality, such as a personal digital assistant (PDA) , a mobile telephone, a smartphone, an autonomous vehicle, or another suitable device. A client device 820 is configured to communicate via the network 840. In an embodiment, a client device 820 is an integrated computing device that operates as a standalone network-enabled device. For example, the client device 820 includes display, speakers, microphone, camera, and input device. In another embodiment, a client device 820 is a computing device for coupling to an external media device such as a television or other external display and/or audio output system. In this embodiment, the client device 820 may couple to the external media device via a wireless interface or wired interface and may utilize various functions of the external media device such as its display, speakers, microphone, camera, and input devices. Here, the client device 820 may be configured to be compatible with a generic external media device that does not have specialized software, firmware, or hardware specifically for interacting with the client device 820.

The third-party system 830 is an online system that may communicate with the motion tracking system 810 or at least one of the client devices 820. In some embodiments, the third-party system 830 may provide data to the motion tracking system 810 for 3D pose estimation. The data may include depth images, data for training DNNs, data for validating DNNs, and so on. The third-party system 830 may be a social media system, an online image gallery, an online searching system, and so on. Additionally or alternatively, the third-party system 830 may use results of 3D pose estimation in various applications. For instance, the third-party system 830 may use motion tracking results from the motion tracking system 810 for action recognition, sport analysis, virtual reality, augmented reality, film and game production, telepresence, and so on.

The motion tracking system 810, client devices 820, and third-party system 830 are connected through a network 840. The network 840 may comprise any combination of local area and/or wide area networks, using both wired and/or wireless communication systems. In one embodiment, the network 840 may use standard communications technologies and/or protocols. For example, the network 840 may include communication links using technologies such as Ethernet, 8010.11, worldwide interoperability for microwave access (WiMAX) , 3G, 4G, code division multiple access (CDMA) , digital subscriber line (DSL) , etc. Examples of networking protocols used for communicating via the network 840 may include multiprotocol label switching (MPLS) , transmission control protocol/Internet protocol (TCP/IP) , hypertext transport protocol (HTTP) , simple mail transfer protocol (SMTP) , and file transfer protocol (FTP) . Data exchanged over the network 840 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML) . In some embodiments, all or some of the communication links of the network 840 may be encrypted using any suitable technique or techniques.

Example Method of Motion Tracking

FIG. 9 is a flowchart showing a method 900 of motion tracking, in accordance with various embodiments. The method 900 may be used for 3D motion tracking. The method 900 may be performed by the motion tracking system 100 in FIG. 1. Although the method 900 is described with reference to the flowchart illustrated in FIG. 9, many other methods for motion tracking may alternatively be used. For example, the order of execution of the steps in FIG. 9 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

The motion tracking system 100 generates 910 a point cloud of an object using one or more depth images that capture the object in a local area. In some embodiments, the one or more depth images are a plurality of depth images. The motion tracking system 100 converts the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images. The motion tracking system 100 generates a fused point cloud using the plurality of point clouds. The motion tracking system 100 generates the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.

In some embodiments, the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area. Locations of the plurality of depth cameras in the local area are determined based on the shape of the local area. In some embodiments, the plurality of depth cameras is synchronized and captures the plurality of depth images at the same time.

The motion tracking system 100 extracts 920, by a first neural network, a feature map from the point cloud. In some embodiments, the first neural network is a point-based neural network. In some embodiments, the first neural network is the neural network 150 in FIG. 1.

The motion tracking system 100 generates 930 a combined feature map using the extracted feature map and a template feature map. The template feature map represents a template structure that includes joints and one or more connections between the joints. In some embodiments, the motion tracking system 100 concatenates the extracted feature and the template feature map. A dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.

In some embodiments, the combined feature map is a graph that includes nodes and one or more edges connecting the nodes. A node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.

The motion tracking system 100 estimates 940, by a second neural network, a pose of the object in the local area using the combined feature map. In some embodiments, the first neural network is a graph convolutional network. In some embodiments, the second neural network is the neural network 170 in FIG. 1. In some embodiments, motion tracking system 100 estimates the pose of the object by determining, by the second neural network, locations of the joints in the local area. In some embodiments, the motion tracking system 100 generates a content item based on the estimated pose of the object and provides the content item for display in the local area.

Example Computing Device

FIG. 10 is a block diagram of an example computing device 1000, in accordance with various embodiments. In some embodiments, the computing device 1000 can be used as at least part of the motion tracking system 100 in FIG. 1. The computing device 1000 may be an example of a client device 820 in FIG. 8 or an example of the third-party system 830 in FIG. 8. A number of components are illustrated in FIG. 10 as included in the computing device 1000, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1000 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1000 may not include one or more of the components illustrated in FIG. 10, but the computing device 1000 may include interface circuitry for coupling to the one or more components. For example, the computing device 1000 may not include a display device 1006, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1006 may be coupled. In another set of examples, the computing device 1000 may not include an audio input device 1018 or an audio output device 1008, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1018 or audio output device 1008 may be coupled.

The computing device 1000 may include a processing device 1002 (e.g., one or more processing devices) . The processing device 1002 processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory. The computing device 1000 may include a memory 1004, which may itself include one or more memory devices such as volatile memory (e.g., DRAM) , nonvolatile memory (e.g., read-only memory (ROM) ) , high bandwidth memory (HBM) , flash memory, solid state memory, and/or a hard drive. In some embodiments, the memory 1004 may include memory that shares a die with the processing device 1002. In some embodiments, the memory 1004 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for motion tracking, e.g., the method 900 described above in conjunction with FIG. 9 or some operations performed by the motion tracking system 100. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1002.

In some embodiments, the computing device 1000 may include a communication chip 1012 (e.g., one or more communication chips) . For example, the communication chip 1012 may be configured for managing wireless communications for the transfer of data to and from the computing device 1000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

The communication chip 1012 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family) , IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment) , Long-Term Evolution (LTE) project along with any amendments, updates, and/or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2" ) , etc. ) . IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1012 may operate in accordance with a Global System for Mobile Communication (GSM) , General Packet Radio Service (GPRS) , Universal Mobile Telecommunications System (UMTS) , High Speed Packet Access (HSPA) , Evolved HSPA (E-HSPA) , or LTE network. The communication chip 1012 may operate in accordance with Enhanced Data for GSM Evolution (EDGE) , GSM EDGE Radio Access Network (GERAN) , Universal Terrestrial Radio Access Network (UTRAN) , or Evolved UTRAN (E-UTRAN) . The communication chip 1012 may operate in accordance with CDMA, Time Division Multiple Access (TDMA) , Digital Enhanced Cordless Telecommunications (DECT) , Evolution-Data Optimized (EV-DO) , and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1012 may operate in accordance with other wireless protocols in other embodiments. The computing device 1000 may include an antenna 1022 to facilitate wireless communications and/or to receive other wireless communications (such as AM or FM radio transmissions) .

In some embodiments, the communication chip 1012 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet) . As noted above, the communication chip 1012 may include multiple communication chips. For instance, a first communication chip 1012 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1012 may be dedicated to longer-range wireless communications such as global positioning system (GPS) , EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1012 may be dedicated to wireless communications, and a second communication chip 1012 may be dedicated to wired communications.

The computing device 1000 may include battery/power circuitry 1014. The battery/power circuitry 1014 may include one or more energy storage devices (e.g., batteries or capacitors) and/or circuitry for coupling components of the computing device 1000 to an energy source separate from the computing device 1000 (e.g., AC line power) .

The computing device 1000 may include a display device 1006 (or corresponding interface circuitry, as discussed above) . The display device 1006 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD) , a light-emitting diode display, or a flat panel display, for example.

The computing device 1000 may include an audio output device 1008 (or corresponding interface circuitry, as discussed above) . The audio output device 1008 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

The computing device 1000 may include an audio input device 1018 (or corresponding interface circuitry, as discussed above) . The audio input device 1018 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output) .

The computing device 1000 may include a GPS device 1016 (or corresponding interface circuitry, as discussed above) . The GPS device 1016 may be in communication with a satellite-based system and may receive a location of the computing device 1000, as known in the art.

The computing device 1000 may include another output device 1010 (or corresponding interface circuitry, as discussed above) . Examples of the other output device 1010 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

The computing device 1000 may include another input device 1020 (or corresponding interface circuitry, as discussed above) . Examples of the other input device 1020 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

The computing device 1000 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultramobile personal computer, etc. ) , a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1000 may be any other electronic device that processes data.

Select Examples

The following paragraphs provide various examples of the embodiments disclosed herein.

Example 1 provides a method, including generating a point cloud of an object using one or more depth images that capture the object in a local area; extracting, by a first neural network, a feature map from the point cloud; generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints; and estimating, by a second neural network, a pose of the object in the local area using the combined feature map.

Example 2 provides the method of example 1, in which the one or more depth images are a plurality of depth images, and generating the point cloud of the object includes converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images; generating a fused point cloud using the plurality of point clouds; and generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.

Example 3 provides the method of example 1 or 2, in which generating the combined feature map includes concatenating the extracted feature and the template feature map, in which a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.

Example 4 provides the method of any one of examples 1-3, in which the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.

Example 5 provides the method of any one of examples 1-4, in which estimating, by the second neural network, the pose of the object in the local area includes determining, by the second neural network, locations of the joints in the local area.

Example 6 provides the method of any one of examples 1-5, in which the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.

Example 7 provides the method of any one of examples 1-6, further including generating a content item based on the estimated pose of the object; and providing the content item for display in the local area.

Example 8 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including generating a point cloud of an object using one or more depth images that capture the object in a local area; extracting, by a first neural network, a feature map from the point cloud; generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints; and estimating, by a second neural network, a pose of the object in the local area using the combined feature map.

Example 9 provides the one or more non-transitory computer-readable media of example 8, in which the one or more depth images are a plurality of depth images, and generating the point cloud of the object includes converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images; generating a fused point cloud using the plurality of point clouds; and generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.

Example 10 provides the one or more non-transitory computer-readable media of example 8 or 9, in which generating the combined feature map includes concatenating the extracted feature and the template feature map, in which a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.

Example 11 provides the one or more non-transitory computer-readable media of any one of examples 8-10, in which the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.

Example 12 provides the one or more non-transitory computer-readable media of any one of examples 8-11, in which estimating, by the second neural network, the pose of the object in the local area includes determining, by the second neural network, locations of the joints in the local area.

Example 13 provides the one or more non-transitory computer-readable media of any one of examples 8-12, in which the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.

Example 14 provides the one or more non-transitory computer-readable media of any one of examples 8-13, in which the operations further comprise generating a content item based on the estimated pose of the object; and providing the content item for display in the local area.

Example 15 provides an apparatus, including a computer processor for executing computer program instructions; and one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including generating a point cloud of an object using one or more depth images that capture the object in a local area, extracting, by a first neural network, a feature map from the point cloud, generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints, and estimating, by a second neural network, a pose of the object in the local area using the combined feature map.

Example 16 provides the apparatus of example 15, in which the one or more depth images are a plurality of depth images, and generating the point cloud of the object includes converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images; generating a fused point cloud using the plurality of point clouds; and generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.

Example 17 provides the apparatus of example 15 or 16, in which generating the combined feature map includes concatenating the extracted feature and the template feature map, in which a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.

Example 18 provides the apparatus of any one of examples 15-17, in which the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.

Example 19 provides the apparatus of any one of examples 15-18, in which estimating, by the second neural network, the pose of the object in the local area includes determining, by the second neural network, locations of the joints in the local area.

Example 20 provides the apparatus of any one of examples 15-19, in which the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.

The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

A method, comprising:

generating a point cloud of an object using one or more depth images that capture the object in a local area;

extracting, by a first neural network, a feature map from the point cloud;

generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints; and

estimating, by a second neural network, a pose of the object in the local area using the combined feature map.
The method of claim 1, wherein the one or more depth images are a plurality of depth images, and generating the point cloud of the object comprises:

converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images;

generating a fused point cloud using the plurality of point clouds; and

generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.
The method of claim 1, wherein generating the combined feature map comprises:

concatenating the extracted feature and the template feature map,

wherein a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.
The method of claim 1, wherein the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.
The method of claim 1, wherein estimating, by the second neural network, the pose of the object in the local area comprises:

determining, by the second neural network, locations of the joints in the local area.
The method of claim 1, wherein the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.
The method of claim 1, further comprising:

generating a content item based on the estimated pose of the object; and

providing the content item for display in the local area.
One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:

generating a point cloud of an object using one or more depth images that capture the object in a local area;

extracting, by a first neural network, a feature map from the point cloud;

generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints; and

estimating, by a second neural network, a pose of the object in the local area using the combined feature map.
The one or more non-transitory computer-readable media of claim 8, wherein the one or more depth images are a plurality of depth images, and generating the point cloud of the object comprises:

converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images;

generating a fused point cloud using the plurality of point clouds; and

generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.
The one or more non-transitory computer-readable media of claim 8, wherein generating the combined feature map comprises:

concatenating the extracted feature and the template feature map,

wherein a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.
The one or more non-transitory computer-readable media of claim 8, wherein the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.
The one or more non-transitory computer-readable media of claim 8, wherein estimating, by the second neural network, the pose of the object in the local area comprises:

determining, by the second neural network, locations of the joints in the local area.
The one or more non-transitory computer-readable media of claim 8, wherein the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.
The one or more non-transitory computer-readable media of claim 8, wherein the operations further comprise:

generating a content item based on the estimated pose of the object; and

providing the content item for display in the local area.
An apparatus, comprising:

a computer processor for executing computer program instructions; and

one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:

generating a point cloud of an object using one or more depth images that capture the object in a local area,

extracting, by a first neural network, a feature map from the point cloud,

generating a combined feature map using the extracted feature map and a template feature map, the template feature map representing a template structure that includes joints and one or more connections between the joints, and

estimating, by a second neural network, a pose of the object in the local area using the combined feature map.
The apparatus of claim 15, wherein the one or more depth images are a plurality of depth images, and generating the point cloud of the object comprises:

converting the plurality of depth images to a plurality of point clouds using depth pixels extracted from the plurality of depth images;

generating a fused point cloud using the plurality of point clouds; and

generating the point cloud by reducing a total number of points in the fused point cloud to a predetermined number.
The apparatus of claim 15, wherein generating the combined feature map comprises:

concatenating the extracted feature and the template feature map,

wherein a dimension of the combined feature map is a sum of a dimension of the extracted feature map and a dimension of the template feature map.
The apparatus of claim 15, wherein the combined feature map is a graph that includes nodes and one or more edges connecting the nodes, a node corresponds to a joint in the template structure encoded by the template feature map, and the second neural network is a graph convolutional network.
The apparatus of claim 15, wherein estimating, by the second neural network, the pose of the object in the local area comprises:

determining, by the second neural network, locations of the joints in the local area.
The apparatus of claim 15, wherein the one or more depth images are a plurality of depth images captured by a plurality of depth cameras in the local area, and locations of the plurality of depth cameras in the local area are determined based on a shape of the local area.