KR20250171282A - Integration of dense pose and 3D body mesh reconstruction - Google Patents

Abstract

Methods and systems for generating a 3D body mesh are disclosed. The system receives an image comprising a depiction of a real-world object in a real-world environment. The system applies a first machine learning model to a portion of the image depicting the real-world object to predict a tensor of heatmaps representing vertex positions of a plurality of triangles of a 3D mesh corresponding to the real-world object. First and second heatmaps of the tensor represent first and second groups of possible coordinates for a first vertex of a first triangle among the plurality of triangles, respectively. The system generates a 3D mesh based on a selected subset of the tensor of heatmaps.

Description

Integration of dense pose and 3D body mesh reconstruction

This application claims the benefit of priority to Greek Patent Application No. 20230100198, filed March 8, 2023, and U.S. Patent Application No. 18/340,736, filed June 23, 2023, each of which is incorporated herein by reference in its entirety.

The present disclosure generally relates to providing augmented reality (AR) experiences using interactive applications.

Augmented reality (AR) is a modification of a virtual environment. For example, in virtual reality (VR), the user is fully immersed in a virtual world, whereas in AR, the user is immersed in a world where virtual objects are combined or superimposed on the real world. AR systems aim to create and present virtual objects that interact realistically with the real-world environment and each other. Examples of AR applications include single- or multi-player video games and instant messaging systems.

In drawings that are not necessarily drawn to scale, similar numbers may describe similar components in different drawings. To facilitate identification of a particular element or act in the discussion, the most significant digit or digits in the reference number refer to the drawing number in which that element is first introduced. Some non-limiting examples are illustrated in the accompanying drawings:
FIG. 1 is a schematic representation of a networked environment in which the present disclosure may be deployed, according to some examples.
Figure 2 is a schematic representation of a messaging system, according to some examples, having both client-side and server-side functionality.
Figure 3 is a schematic representation of a data structure as maintained in a database, according to some examples.
Figure 4 is a schematic representation of a message according to some examples.
FIG. 5 is a block diagram illustrating an exemplary sparse pose estimation system according to some examples.
Figures 6 to 9 are schematic representations of various loss calculations for training a machine learning model of a sparse pose estimation system, according to some examples.
FIG. 10 is a flowchart illustrating exemplary operations of a sparse pose estimation system according to some examples.
FIG. 11 is a schematic representation of a machine in the form of a computer system on which a set of instructions may be executed to cause the machine to perform one or more of the methodologies discussed herein, according to some examples.
Figure 12 is a block diagram illustrating a software architecture within which examples can be implemented.
FIG. 13 illustrates a system in which a head-wearable device may be implemented, according to some examples.

The following description includes systems, methods, techniques, instruction sequences, and computing machine program products that implement illustrative examples of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth to provide an understanding of the various examples. However, it will be apparent to one skilled in the art that the examples may be practiced without these specific details. These specific details generally do not necessarily show well-known command instances, protocols, structures, and techniques in detail.

Typically, VR and AR systems capture an image of a user and, using a depth sensor, acquire a depth map of the real-world human body depicted in the image to display images representing the given user. By processing the depth map and the image together, VR and AR systems can detect the user's position within the image and appropriately modify the user or the background within the image. While these systems work well, the need for a depth sensor limits their scope of application. This is because adding depth sensors to user devices for the purpose of image modification increases the overall cost and complexity of the devices, making them less attractive and more difficult to implement in mobile device settings.

Certain systems estimate three-dimensional (3D) meshes of people depicted in images to provide AR/VR experiences. Understanding human 3D poses is essential for human-computer interaction, motion capture, action recognition, and augmented and virtual reality (AR/VR). Human mesh reconstruction has recently gained attention based on the premise that information beyond the 3D skeleton can unlock new experiences requiring detailed estimation of 3D human geometry, such as avatar construction for VR or visual effects, and virtual try-on for AR, and telepresence.

Despite rapid progress in 3D mesh reconstruction technology, meshes from current systems are still far from pixel-accurate when projected back into the image domain. Mesh reconstruction evaluations are based on the 3D skeleton or 3D mesh and do not reflect 2D reprojection accuracy. However, when people are close to the camera, small 3D errors can lead to very visible 2D reprojection inaccuracies. For primarily selfie-based AR use cases, this can "break the illusion": while misalignments are readily perceived by the user, the depth component of 3D pose accuracy can be considered secondary because it is accounted for by the 2D reprojection.

Typical systems rely on dense-pose to generate 3D meshes. Specifically, dense-pose densely associates pixels with surface coordinates, either through UV-coordinate regression or by predicting 3D canonical coordinates. UVs represent the axes of a 2D texture, distinct from the X, Y, and Z axes that represent the 3D object model. By treating this as a continuous regression task, localization of individual mesh vertices is avoided, allowing them to be trained with annotations collected for randomly selected points and to work with a reasonable number of dense predictions. However, despite its efficiency, this approach has found little direct application for mesh recovery.

Some other conventional systems use pre-cropped images to predict 3D meshes through parametric human models. A convolutional encoder is used to minimize reprojection errors of 2D joints, while a discriminator determines the validity of the 3D parameters. Such systems are quite complex to implement on mobile device platforms and still produce pixel-accurate results.

The disclosed techniques attempt to improve the way 3D meshes are generated for one or more real-world objects (e.g., real-world people) depicted in images or videos. The disclosed techniques use one or more machine learning models to predict dense poses and localize mesh vertices. Specifically, the disclosed techniques predict sparse poses, which represent a tensor of heatmaps for the 2D positions of a sparse subset of body vertices. These vertices are configured to form a low-polygon approximation of a full-blown, high-resolution mesh, which has two advantages. First, the disclosed techniques can seamlessly interpolate between 2D vertices belonging to a triangle using barycentric coordinates. This allows the disclosed techniques to predict continuous UV fields within the triangle as a weighted combination of vertex UV values and to utilize dense-pose datasets for supervision. Second, the disclosed techniques can sequentially localize these vertices in 2D using argsoftmax, and then lift them into 3D using per-vertex monocular depth predictions. This provides a bottom-up estimation of 3D mesh geometry that can be projected back onto the image with high pixel-level accuracy, as accurately as the original sparse pose prediction.

The disclosed machine learning models are trained based on ground-truth dense-pose image information and multiple losses. For example, the disclosed techniques use a surface-based loss that enforces a softmax-based posterior over sparse-pose vertices to approximate the center of gravity coordinates of any pixel with UV annotations. This allows dense-pose to supervise sparse-pose competition across vertex channels for a given pixel. The disclosed techniques introduce a second surface-based loss that enforces sparse-pose to position vertices such that the image coordinates of the annotated UV values align with their annotated positions. This enables dense-pose to supervise sparse-pose competition for three vertex channels associated with any UV-annotated pixel. The disclosed techniques provide an architecture that allows per-vertex depth regression based on a 3D pose skeleton regressed by a separate task head. This enables leveraging the sparse 3D skeleton supervision and driving the remaining mesh vertices with more easily predictable offsets. An additional task is provided to predict parts of the mesh that are not visible within the frame (occluded/self-occluded/clipped vertices), allowing the disclosed techniques to combine sparse-pose-based vertices with an "inpainted mesh." All of the aforementioned layers feed from a common backbone and provide an end-to-end trainable architecture capable of jointly supervising multiple data sources. These losses enable training the disclosed sparse-pose system (machine learning models) with dense-pose accuracy comparable to strong baselines for dense-pose estimation.

According to the disclosed techniques, a system receives an image comprising a depiction of a real-world object in a real-world environment. The disclosed system applies a first machine learning model to a portion of the image depicting the real-world object to predict a tensor of heatmaps representing vertex positions of a plurality of triangles of a 3D mesh corresponding to the real-world object. First and second heatmaps of the tensor represent first and second groups of possible coordinates for a first vertex of a first triangle among the plurality of triangles, respectively. The disclosed system generates a 3D mesh based on a selected subset of the tensor of heatmaps.

This simplifies the process of adding AR graphics to images or videos, significantly reducing design constraints and costs in generating such AR graphics, and reducing processing complexity, power, and memory requirements. This also improves the illusion that the AR graphics are part of the real-world environment depicted in the image or video depicting real-world objects. This enables seamless and efficient addition of AR graphics to underlying images or videos in real time on small mobile devices. The disclosed techniques can be applied entirely or primarily on a mobile device without requiring the mobile device to transmit the images/videos to a server. In other examples, the disclosed techniques can be applied entirely or primarily on a remote server, or split between the mobile device and the server.

This improves the overall user experience when using electronic devices. Furthermore, by providing these AR experiences without using a depth sensor, the overall amount of system resources required to accomplish a task is reduced. As used herein, the terms "article of clothing," "fashion item," and "garment" are used interchangeably and are understood to have the same meaning. An article of clothing, garment, or fashion item may include a shirt, a skirt, a dress, shoes, a purse, a furniture item, a household item, eyewear, glasses, an AR logo, an AR emblem, pants, shorts, a jacket, a T-shirt, a blouse, glasses, jewelry, earrings, rabbit ears, a hat, earmuffs, or any other suitable item or object.

networked computing environment

FIG. 1 is a block diagram illustrating an exemplary interaction system (100) for facilitating interactions (e.g., exchanging text messages, conducting text audio and video calls, or playing games) over a network. The interaction system (100) includes a number of user systems (102), each of which hosts a number of applications, including an interaction client (104) and other applications (106). Each interaction client (104) is communicatively coupled to other instances of the interaction client (104) (e.g., hosted on each of the other user systems (102), an interaction server system (110), and third-party servers (112) via one or more communications networks, including a network (108) (e.g., the Internet). The interaction client (104) may also communicate with locally hosted applications (106) using Application Program Interfaces (APIs).

Each user system (102) may include a number of user devices, such as a mobile device (114), a head-wearable device (116), and a computer client device (118), communicatively connected to exchange data and messages.

An interaction client (104) interacts with other interaction clients (104) and with an interaction server system (110) via a network (108). Data exchanged between interaction clients (104) (e.g., interactions (120)) and data exchanged between interaction clients (104) and the interaction server system (110) include functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

The interaction server system (110) provides server-side functionality to interaction clients (104) via a network (108). While certain functions of the interaction system (100) are described herein as being performed by the interaction clients (104) or by the interaction server system (110), the location of certain functionality within the interaction clients (104) or the interaction server system (110) may be a design choice. For example, it may be technically desirable to initially place certain technologies and functionality within the interaction server system (110), but later migrate such technologies and functionality to the interaction clients (104) where the user system (102) has sufficient processing capacity.

The interaction server system (110) supports various services and operations provided to interaction clients (104). These operations include sending data to the interaction clients (104), receiving data from the interaction clients (104), and processing data generated by the interaction clients (104). This data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchanges within the interaction system (100) are initiated and controlled through functions available through the user interfaces (UIs) of the interaction clients (104).

Now, looking specifically at the interaction server system (110), an Application Program Interface (API) server (122) is coupled to the interaction servers (124) to provide programmatic interfaces to make the functions of the interaction servers (124) accessible to interaction clients (104), other applications (106), and third-party servers (112). The interaction servers (124) are communicatively coupled to a database server (126) to facilitate access to a database (128) that stores data associated with interactions processed by the interaction servers (124). Similarly, a web server (130) is coupled to the interaction servers (124) and provides web-based interfaces to the interaction servers (124). To this end, the web server (130) processes incoming network requests via Hypertext Transfer Protocol (HTTP) and some other related protocols.

The API server (122) receives and transmits interaction data (e.g., commands and message payloads) between the interaction servers (124) and client systems (102) (and, for example, the interaction clients (104) and other applications (106)) and third-party servers (112). Specifically, the API server (122) provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the interaction clients (104) and other applications (106) to invoke the functionality of the interaction servers (124). The API server (122) provides functions such as account registration; login functionality; the transmission of interaction data from one interaction client (104) to another interaction client (104) via the interaction servers (124); the communication of media files (e.g., images or video) from an interaction client (104) to the interaction servers (124); the configuration of collections of media data (e.g., stories); Exposes various functions supported by the interaction servers (124), including retrieving a list of a user's friends of the user system (102); retrieving messages and content; adding and deleting entities (e.g., friends) to the entity relationship graph; locating friends within the entity relationship graph; and opening application events (e.g., related to the interaction client (104)).

The interaction servers (124) host a number of systems and subsystems described below with reference to FIG. 2.

Linked Applications

Returning to the interaction client (104), features and functionalities of external resources (e.g., linked applications (106) or applet) are made available to the user through the interface of the interaction client (104). In this context, “external” refers to the fact that the application (106) or applet is external to the interaction client (104). External resources are often provided by third parties, but may also be provided by the creator or provider of the interaction client (104). The interaction client (104) receives a user selection of an option to launch or access features of such external resources. The external resource may be an application (106) installed on the user system (102) (e.g., a “native app”), or a smaller version of an application (e.g., an “applet”) hosted on the user system (102) or remotely from the user system (102) (e.g., on third-party servers (112)). A smaller version of an application includes a subset of the application's features and functionality (e.g., a full-scale native version of the application) and is implemented using markup-language documents. In some examples, a smaller version of an application (e.g., an "applet") is a web-based, markup-language version of the application and is embedded in the interactive client (104). In addition to using markup-language documents (e.g., .*ml files), an applet may include a scripting language (e.g., .*js files or .json files) and style sheets (e.g., .*ss files).

In response to receiving a user selection of an option to launch an external resource or access its features, the interaction client (104) determines whether the selected external resource is a web-based external resource or a locally installed application (106). In some cases, applications (106) that are locally installed on the user system (102) may be launched independently and separately from the interaction client (104), such as by selecting an icon corresponding to the application (106) on the home screen of the user system (102). Miniature versions of these applications may be launched or accessed through the interaction client (104), and in some instances, no portion of the mini-application may be accessible outside of the interaction client (104), or only limited portions may be accessible. A mini-application may be launched by the interaction client (104) receiving a markup language document associated with the mini-application, for example, from a third-party server (112) and processing such document.

In response to determining that the external resource is a locally installed application (106), the interaction client (104) instructs the user system (102) to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client (104) communicates with (for example) third-party servers (112) to obtain a markup language document corresponding to the selected external resource. The interaction client (104) then processes the obtained markup language document and presents the web-based external resource within the user interface of the interaction client (104).

An interaction client (104) may notify users of the user system (102), or other users associated with such users (e.g., "friends"), of activity occurring on one or more external resources. For example, the interaction client (104) may provide participants in a conversation (e.g., a chat session) on the interaction client (104) with notifications regarding current or recent use of an external resource by one or more members of a group of users. One or more users may be invited to participate in an active external resource or to launch an external resource that has been recently used (in a group of friends) but is currently inactive. An external resource may provide participants in the conversation, each using its own interaction client (104), with the ability to share an item, status, state, or location from the external resource in the chat session with one or more members of the group of users. A shared item may be an interactive chat card that members of the chat can interact with, for example, to launch a corresponding external resource, view specific information within the external resource, or take a member of the chat to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on the interaction client (104). The external resource can optionally include different media items in the responses based on the current context of the external resource.

The interaction client (104) may present to the user a list of external resources (e.g., applications (106) or applets) available for launching or accessing a given external resource. This list may be presented in a context-sensitive menu. For example, icons representing different applications (106) (or applets) may vary based on how the menu is launched by the user (e.g., from a conversational interface or a non-conversational interface).

In some examples, the interaction client (104) may utilize the sparse pose estimation system (500) to generate an AR experience in which one or more AR graphic elements are overlaid in a pixel-precise manner on a person depicted in an image or video. The interaction client (104) may render an image or video depicting a real-world person wearing a fashion item based on a 3D mesh predicted/generated by the sparse pose estimation system (500). In this manner, a user may see how the user would look wearing one or more virtual fashion items. The clothing, garment, or fashion item may include a shirt, a skirt, a dress, shoes, a purse, a furniture item, a household item, eyewear, glasses, an AR logo, an AR emblem, pants, shorts, a jacket, a t-shirt, a blouse, glasses, jewelry, earrings, rabbit ears, a hat, earmuffs, or any other suitable item or object. Additional details of such an AR experience are discussed below with respect to the sparse pose estimation system (500) of FIG. 5 .

System Architecture

FIG. 2 is a block diagram illustrating additional details regarding an interaction system (100), according to some examples. Specifically, the interaction system (100) is depicted as including an interaction client (104) and interaction servers (124). The interaction system (100) embodies a number of subsystems supported by the interaction client (104) on the client side and by the interaction servers (124) on the server side. Exemplary subsystems are discussed below and may include a sparse pose estimation system (500) that enables a user to launch an AR experience. An exemplary implementation of the sparse pose estimation system (500) is illustrated and described below with reference to FIG. 5 .

Specifically, the sparse pose estimation system (500) is a component that can be accessed by an AR/VR application implemented on a user system (102). The AR/VR application uses an RGB camera to capture a monocular image of a real-world object. The AR/VR application applies various trained machine learning techniques or machine learning models to the captured image of the real-world object to generate body landmark features representing the real-world object depicted in the images or videos and to apply one or more AR visual effects to the captured image or video based on the body landmark features. In some implementations, the AR/VR application continuously captures images of the user and updates the body landmark features in real time or periodically to continuously or periodically update the applied one or more visual effects. This allows the user to move around in the real world and also see the one or more visual effect updates in real time.

The image processing system (202) provides various features that enable a user to capture and augment (e.g., annotate or otherwise modify or edit) media content associated with a message.

The camera system (204) includes control software (e.g., within a camera application) that interacts with and controls the hardware camera hardware of the user system (102) (e.g., directly or through operating system controls) to modify and enhance real-time images captured and displayed through the interaction client (104).

The augmentation system (206) provides functions related to the generation and presentation of augmentations (e.g., media overlays) for images captured in real time by cameras of the user system (102) or retrieved from memory of the user system (102). For example, the augmentation system (206) operatively selects, presents, and displays media overlays (e.g., image filters or image lenses) to the interaction client (104) to augment real-time images received via the camera system (204) or stored images retrieved from memory (1302) of the user system (102) (as shown in FIG. 13). These augmentations are selected by the augmentation system (206) and presented to the user of the interaction client (104) based on, for example, a number of data and inputs such as:

● Geolocation of the user system (102); and

● User entity relationship information of the user system (102).

Augmentation may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) in the user system (102) for communication in a message, or to video content, such as a video content stream or feed transmitted from an interaction client (104). In this way, the image processing system (202) may interact with and support various subsystems of the communication system (208), such as the messaging system (210) and the video communication system (212).

A media overlay may include text or image data that may be overlaid on top of a photograph taken by the user system (102) or a video stream generated by the user system (102). In some examples, the media overlay may be a location overlay (e.g., Venice Beach), the name of a live event, or a merchant name overlay (e.g., Beach Coffee House). In further examples, the image processing system (202) uses the geolocation of the user system (102) to identify a media overlay that includes the merchant name in the geolocation of the user system (102). The media overlay may include other indicia associated with the merchant. The media overlays may be stored in a database (128) and accessed via a database server (126).

The image processing system (202) provides a user-based publishing platform that allows users to select geolocations on a map and upload content associated with the selected geolocations. Users can also specify situations in which specific media overlays should be presented to other users. The image processing system (202) generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

The augmented creation system (214) supports augmented reality developer platforms and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) of the interactive client (104). The augmented creation system (214) provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technology, and templates.

In some examples, the augmented creation system (214) provides a merchant-based publishing platform that allows merchants to select specific augmentations associated with geolocations through a bidding process. For example, the augmented creation system (214) associates the media overlay of the highest-bidding merchant with the corresponding geolocation for a predefined amount of time.

The communication system (208) is responsible for facilitating and processing multiple forms of communication and interaction within the interaction system (100), and includes a messaging system (210), an audio communication system (216), and a video communication system (212). The messaging system (210) is responsible for enforcing temporary or time-limited access to content by the interaction clients (104). The messaging system (210) incorporates multiple timers (e.g., within a short-term timer system (218)) that selectively enable access to (e.g., for presentation and display) messages and associated content via the interaction clients (104) based on duration and display parameters associated with the message, or collection of messages (e.g., a story). Additional details regarding the operation of the short-term timer system (218) are provided below. The audio communication system (216) enables and supports audio communications (e.g., real-time audio chat) between the multiple interaction clients (104). Similarly, the video communication system (212) enables and supports video communication (e.g., real-time video chat) between multiple interactive clients (104).

The user management system (220) is operationally responsible for managing user data and profiles, and includes an entity relationship (ER) system (222) that maintains information about relationships between users of the interaction system (100).

The collection management system (224) is operationally responsible for managing sets or collections of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages containing images, video, text, and audio) may be organized into "event galleries" or "event stories." These collections may be made available for a specific time period, such as the duration of an event to which the content relates. For example, content related to a music concert may be made available as a "story" for the duration of the music concert. The collection management system (224) may also be responsible for displaying icons on the user interface of the interaction client (104) that provide notification of a specific collection. The collection management system (224) includes a curation function that allows a collection manager to manage and curate a specific collection of content. For example, a curation interface may enable an event organizer to curate a collection of content related to a specific event (e.g., to delete inappropriate content or duplicate messages). Additionally, the collection management system (224) automatically curates content collections by employing machine vision (or image recognition technology) and content rules. In certain instances, users may be compensated for including user-generated content in the collection. In such cases, the collection management system (224) automatically compensates these users for using their content.

The map system (226) provides various geographic location features and supports the presentation of map-based media content and messages by the interaction client (104). For example, the map system (226) enables the display of user icons or avatars (e.g., stored in the profile data (302) of FIG. 3) on the map to indicate the current or past locations of the user's "friends" as well as media content (e.g., collections of messages including photos and videos) generated by such friends within the context of the map. For example, a message posted by a user to the interaction system (100) from a particular geographic location may be displayed to the particular user's "friends" on the map interface of the interaction client (104) within the context of the map at that particular location. Users can further share their location and status information with other users of the interaction system (100) via the interaction client (104) (e.g., using an appropriate status avatar), which location and status information is similarly displayed to selected users within the context of the map interface of the interaction client (104).

The game system (228) provides various gaming features within the context of the interactive client (104). The interactive client (104) provides a game interface that provides a list of available games that can be launched by a user within the context of the interactive client (104) and played with other users of the interactive system (100). The interactive system (100) also allows a particular user to invite other users to participate in the play of a particular game by issuing invitations from the interactive client (104) to other users. The interactive client (104) also supports audio, video, and text messaging (e.g., chats) within the context of gameplay, provides leaderboards for games, and supports the provision of in-game rewards (e.g., coins and items).

The external resource system (230) provides an interface for the interactive client (104) to communicate with remote servers (e.g., third-party servers (112)) to launch or access external resources, i.e., applications or applets. Each third-party server (112) hosts, for example, a markup language (e.g., HTML5)-based application or a smaller version of an application (e.g., a game, utility, payment, or ride-sharing application). The interactive client (104) can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the third-party servers (112) associated with the web-based resource. The applications hosted by the third-party servers (112) are programmed in JavaScript using a Software Development Kit (SDK) provided by the interactive servers (124). The SDK includes Application Programming Interfaces (APIs) having functions that can be called or invoked by the web-based application. Interaction servers (124) host JavaScript libraries that provide access to a given external resource for specific user data of the interaction client (104). HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies may be used.

To integrate the SDK's features into a web-based resource, the SDK is downloaded from the interaction servers (124) to a third-party server (112) or otherwise received by the third-party server (112). Once downloaded or received, the SDK is incorporated as part of the application code of the web-based external resource. The web-based resource's code can then call or invoke specific functions of the SDK to integrate features of the interaction client (104) into the web-based resource.

The SDK stored on the interaction server system (110) effectively provides a bridge between external resources (e.g., applications (106) or applets) and the interaction client (104). This provides the user with a seamless experience communicating with other users on the interaction client (104) while also preserving the look and feel of the interaction client (104). To bridge the communication between the external resource and the interaction client (104), the SDK facilitates communication between third-party servers (112) and the interaction client (104). The WebViewJavaScriptBridge running on the user system (102) establishes two unidirectional communication channels between the external resource and the interaction client (104). Messages are transmitted asynchronously between the external resource and the interaction client (104) over these communication channels. Each SDK function invocation is transmitted as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and transmitting a message with that callback identifier.

By using the SDK, not all information from the interaction client (104) is shared with the third-party servers (112). The SDK limits which information is shared based on the needs of the external resource. Each third-party server (112) provides an HTML5 file corresponding to a web-based external resource to the interaction servers (124). The interaction servers (124) can add a visual representation (e.g., box art or other graphics) of the web-based external resource to the interaction client (104). Once the user selects a visual representation or instructs the interaction client (104) through the GUI of the interaction client (104) to access features of the web-based external resource, the interaction client (104) obtains the HTML5 file and instantiates resources to access features of the web-based external resource.

The interactive client (104) presents a graphical user interface (e.g., a landing page or title screen) for the external resource. During, before, or after presenting the landing page or title screen, the interactive client (104) determines whether the launched external resource is previously authorized to access the user data of the interactive client (104). In response to determining that the launched external resource is previously authorized to access the user data of the interactive client (104), the interactive client (104) presents another graphical user interface of the external resource that includes the functions and features of the external resource. In response to determining that the launched external resource is not previously authorized to access the user data of the interactive client (104), after a threshold time period (e.g., 3 seconds) of displaying the landing page or title screen of the external resource, the interactive client (104) slides up a menu for authorizing the external resource to access the user data (e.g., animates the menu by lifting it from the bottom of the screen to the middle or other portion of the screen). This menu identifies the type of user data that the external resource is authorized to use. In response to receiving a user selection of the Accept option, the interaction client (104) adds the external resource to the list of authorized external resources and allows the external resource to access user data from the interaction client (104). The external resource is authorized by the interaction client (104) to access user data under the OAuth 2 framework.

The interaction client (104) controls the type of user data shared with external resources based on the type of external resource being authorized. For example, external resources including full-scale applications (e.g., application (106)) are provided access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics). As another example, external resources including smaller versions of the applications (e.g., web-based versions of the applications) are provided access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with different avatar characteristics). Avatar characteristics include different ways to customize the look and feel of the avatar, such as different poses, facial features, clothing, and so on.

The advertising system (232) enables third parties to purchase advertisements for presentation to end-users via operationally interactive clients (104), and also handles the delivery and presentation of such advertisements.

Data Architecture

FIG. 3 is a schematic diagram illustrating data structures (300) that may be stored in a database (304) of an interactive server system (110), according to certain examples. While the contents of the database (304) are shown as including a number of tables, it will be appreciated that the data may be stored in other types of data structures (e.g., as an object-oriented database).

The database (304) includes message data stored within a message table (306). This message data includes, for any particular message, at least message sender data, message recipient (or receiver) data, and a payload. Additional details regarding information that may be included in a message and within the message data stored within the message table (306) are described below with reference to FIG. 4.

The entity table (308) stores entity data and is linked (e.g., for reference) to the entity graph (310) and profile data (302). Entities for which records are maintained in the entity table (308) may include individuals, corporate entities, organizations, objects, locations, events, etc. Regardless of entity type, any entity for which the interaction server system (110) stores data may be a recognized entity. Each entity has a unique identifier as well as an entity type identifier (not shown).

The entity relationship graph (310) stores information about relationships and associations between entities. Such relationships may be, for example, social, professional (e.g., work within a corporation or organization), interest-based, or activity-based. Certain relationships between entities may be unidirectional, such as an individual user subscribing to digital content from a commercial or publishing user (e.g., a newspaper or other digital media outlet, or a brand). Other relationships may be bidirectional, such as "friend" relationships between individual users of the interaction system (100).

Specific permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friendship relationship between individual users) may include authorization for the posting of digital content items between the individual users, but may also impose specific restrictions or filters on the posting of such digital content items (e.g., based on content characteristics, location data, or visual data). Similarly, a subscription relationship between an individual user and a commercial user may impose varying degrees of restrictions on the posting of digital content from the commercial user to the individual user, and may significantly limit or block the posting of digital content from the individual user to the commercial user. A specific user, for example, may record specific restrictions (e.g., via privacy settings) in a record for that entity in the entity table (308). These privacy settings may apply to all types of relationships within the context of the interaction system (100), or may be selectively applied to specific types of relationships.

Profile data (302) stores multiple types of profile data for a particular entity. The profile data (302) may be selectively used and presented to other users of the interaction system (100) based on privacy settings specified by the particular entity. If the entity is an individual, the profile data (302) may include, for example, a username, phone number, address, settings (e.g., notification and privacy settings), as well as a user-selected avatar representation (or a collection of such avatar representations). A particular user may then optionally include one or more of these avatar representations within the content of messages communicated via the interaction system (100) and on map interfaces displayed to other users by the interaction clients (104). The collection of avatar representations may include "state avatars," which present a graphical representation of a state or activity that the user may choose to communicate at a particular time.

If the entity is a group, the profile data (302) for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the associated group.

The database (304) also stores augmentation data, such as overlays or filters, in an augmentation table (312). The augmentation data is associated with and applied to videos (for which data is stored in the video table (314)) and images (for which data is stored in the image table (316)).

In some examples, filters are overlays that are displayed over an image or video while being presented to the receiving user. The filters may be of various types, including user-selected filters from a set of filters presented to the sending user by the interaction client (104) when the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters) that may be presented to the sending user based on their geographic location. For example, geolocation filters specific to a neighborhood or specific location may be presented within the user interface by the interaction client (104) based on geolocation information determined by a Global Positioning System (GPS) unit of the user system (102).

Another type of filter is a data filter that can be optionally presented to the transmitting user by the interaction client (104) based on information collected by the user system (102) during the message generation process or other inputs. Examples of data filters include the current temperature at a particular location, the current speed at which the transmitting user is traveling, the battery life for the user system (102), or the current time.

Other augmented data that may be stored within the image table (316) includes augmented reality content items (e.g., corresponding to applying "lenses" or augmented reality experiences). Augmented reality content items may be real-time special effects and sounds that can be added to images or videos.

The story table (318) stores data regarding collections of messages and associated image, video, or audio data, which are compiled into collections (e.g., stories or galleries). The creation of a particular collection may be initiated by a particular user (e.g., each user, for which a record is maintained in the entity table (308)). A user may create a "personal story" in the form of a collection of content created and transmitted/broadcasted by that user. To this end, the user interface of the interaction client (104) may include a user-selectable icon to enable the transmitting user to add specific content to his or her personal story.

A collection can also comprise a "Live Story," a collection of content from multiple users, generated manually, automatically, or using a combination of manual and automated techniques. For example, a "Live Story" can comprise a curated stream of user-submitted content from various locations and events. Users whose client devices are location-enabled and who are present at a common location event at a specific time can be presented with the option to contribute content to a particular Live Story, for example, through the user interface of the interactive client (104). The Live Story can be identified to the user by the interactive client (104) based on their location. The end result is a "Live Story" told from a community perspective.

An additional type of content collection is known as a "location story," which allows a user whose user system (102) is located within a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some instances, contributions to a location story may employ a second degree of authentication to verify that the end-user belongs to a specific organization or other entity (e.g., is a student on a university campus).

As mentioned above, the video table (314), in some examples, stores video data associated with messages for which records are maintained in the message table (306). Similarly, the image table (316) stores image data associated with messages, for which message data is stored in the entity table (308). The entity table (308) may associate various augmentations from the augmentation table (312) with various images and videos stored in the image table (316) and the video table (314).

The databases (304) also include trained machine learning (ML) techniques (307) that store parameters of one or more machine learning models trained during training of the sparse pose estimation system (500). For example, the trained machine learning techniques (307) store trained parameters of one or more artificial neural network machine learning models or techniques.

Data Communication Architecture

FIG. 4 is a schematic diagram illustrating the structure of a message (400), according to some examples, generated by an interaction client (104) for communication to additional interaction clients (104) via interaction servers (124). The contents of a particular message (400) are used to populate a message table (306) stored within a database (304) accessible by the interaction servers (124). Similarly, the contents of the message (400) are stored in memory as "in-transit" or "in-flight" data of the user system (102) or the interaction servers (124). The message (400) is depicted as including the following exemplary components:

● Message identifier (402): A unique identifier that identifies the message (400).

● Message text payload (404): Text generated by the user through the user interface of the user system (102) and included in the message (400).

● Message Image Payload (406): Image data captured by the camera component of the user system (102) or retrieved from the memory component of the user system (102) and included in the message (400). Image data for a transmitted or received message (400) may be stored in the image table (316).

● Message Video Payload (408): Video data captured by the camera component or retrieved from the memory component of the user system (102) and included in the message (400). Video data for a transmitted or received message (400) may be stored in the image table (316).

● Message audio payload (410): Audio data captured by a microphone or retrieved from a memory component of the user system (102) and included in the message (400).

● Message Augmentation Data (412): Augmentation data representing augmentations to be applied to the message image payload (406), message video payload (408), or message audio payload (410) of the message (400) (e.g., filters, stickers, or other annotations or enhancements). Augmentation data for a transmitted or received message (400) may be stored in an augmentation table (312).

● Message Duration Parameter (414): A parameter value that indicates the amount of time in seconds that the content of the message (e.g., message image payload (406), message video payload (408), message audio payload (410)) is presented or made accessible to the user via the interaction client (104).

● Message Geolocation Parameters (416): Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter (416) values may be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image within a message image payload (406), or a particular video within a message video payload (408).

● Message Story Identifier (418): Identifier values that identify one or more content collections (e.g., “stories” identified in the story table (318)) with which a particular content item within the message image payload (406) of the message (400) is associated. For example, each of multiple images within the message image payload (406) may be associated with multiple content collections using identifier values.

● Message Tag (420): Each message (400) may be tagged with multiple tags, each of which indicates a subject matter of the content included in the message payload. For example, if a particular image included in the message image payload (406) depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tag (420). The tag value may be manually generated based on user input or automatically generated using, for example, image recognition.

● Message Sender Identifier (422): An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of the user system (102) through which the message (400) was generated and from which the message (400) was transmitted.

● Message recipient identifier (424): An identifier (e.g., messaging system identifier, email address, or device identifier) that indicates the user of the user system (102) to which the message (400) is addressed.

The contents (e.g., values) of the various components of the message (400) may be pointers to locations within a table where content data values are stored therein. For example, an image value within the message image payload (406) may be a pointer to (or address of) a location within the image table (316). Similarly, values within the message video payload (408) may point to data stored within the image table (316), values within the message augmentation data (412) may point to data stored within the augmentation table (312), values within the message story identifier (418) may point to data stored within the story table (318), and values within the message sender identifier (422) and message recipient identifier (424) may point to user records stored within the entity table (308).

Rare pose estimation system

FIG. 5 is a block diagram illustrating an exemplary sparse pose estimation system (500), according to some examples. The sparse pose estimation system (500) includes a set of components (510) that operate on a set of input data (e.g., one or more monocular images or videos depicting real-world objects, such as a person wearing a fashion item (e.g., a real-world fashion item or an AR fashion item) or training data). The set of input data may be obtained from one or more database(s) ( FIG. 3 ) during training phases and/or may be obtained from an RGB camera of a user system (102), for example, when an AR/VR application is being used by an interactive client (104). The sparse pose estimation system (500) includes one or more machine learning models. The sparse pose estimation system (500) includes an object detection module (512), a tensor prediction module of heatmaps (513), an initial 3D mesh module (514), a 3D key point estimation module (515), an intermediate 3D mesh module (516), a reconstructed 3D mesh module (517), an AR effect module (519), an image rendering module (518), and a 3D body tracking module (513).

In some examples, the sparse pose estimation system (500) receives an image comprising a depiction of a real-world object in a real-world environment. The sparse pose estimation system (500) applies a first machine learning model to a portion of the image depicting the real-world object to predict a tensor of heatmaps representing vertex positions of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object. A first heatmap of the tensor of heatmaps represents a first group of possible coordinates for a first vertex of a first triangle of the plurality of triangles. A second heatmap of the tensor of heatmaps represents a second group of possible coordinates for a second vertex of the first triangle. The sparse pose estimation system (500) processes the tensor of heatmaps to select a subset of the tensor of heatmaps, the subset of the tensor of heatmaps comprising a first coordinate of the first group of possible coordinates and a second coordinate of the second group of possible coordinates. The sparse pose estimation system (500) generates a 3D mesh based on a selected subset of the tensor of heatmaps.

In some examples, the sparse pose estimation system (500) applies a second machine learning model to the image to predict multiple bounding boxes for each of multiple real-world objects depicted in the image. The sparse pose estimation system (500) selects a first bounding box from the multiple bounding boxes to identify a portion of the image. In some examples, a tensor of heatmaps represents a sparse pose corresponding to a range of possible vertices of the dense pose.

In some examples, the 3D mesh includes an initial 3D mesh. The sparse pose estimation system (500) applies a second machine learning model to a portion of the image to predict a set of 3D key points corresponding to the skeleton of a real-world object. In parallel with the initial 3D mesh, the sparse pose estimation system (500) generates an intermediate 3D mesh using a set of 3D key points representing depth per vertex of the real-world object. The sparse pose estimation system (500) fuses the initial 3D mesh with the intermediate 3D mesh to form a reconstructed 3D mesh.

In some examples, the sparse pose estimation system (500) trains a first machine learning model using training data including a plurality of training images depicting training real-world objects and corresponding ground truth dense poses and training 3D meshes corresponding to the training real-world objects, each ground truth dense pose representing coordinates of respective 3D meshes of the training real-world objects. In some examples, the sparse pose estimation system (500) trains the first machine learning model based on a plurality of losses.

In some examples, the first loss among the plurality of losses includes a cross-entropy loss. The sparse pose estimation system (500) obtains a first training image among the plurality of training images and a first ground truth dense pose corresponding to a first training real-world object depicted in the first training image. The sparse pose estimation system (500) uses the first ground truth dense pose to compute the center-of-gravity coordinates of individual triangles of a training 3D mesh corresponding to specific pixels of the first training real-world object. The sparse pose estimation system (500) applies a first machine learning model to the first training image to estimate a tensor of individual heatmaps corresponding to the first training real-world object. The sparse pose estimation system (500) generates predicted posterior coordinates using a selection of a group of coordinates of the tensors of the individual heatmaps corresponding to specific pixels of the first training real-world object. The sparse pose estimation system (500) calculates a deviation including cross entropy loss between predicted posterior coordinates and center of gravity coordinates, and updates the parameters of the first machine learning model based on the deviation.

In some examples, each heatmap of the tensor of individual heatmaps represents a respective group of possible coordinates for an individual vertex of an individual triangle of an individual mesh corresponding to a first training real-world object. The sparse pose estimation system (500) updates scores associated with the possible coordinates for the individual vertices based on a cross-entropy loss. In some examples, the second loss of the plurality of losses comprises a vertex localization loss. In such cases, the sparse pose estimation system (500) integrates the scores of the group of possible coordinates for each vertex of the individual mesh corresponding to the first training real-world object to generate estimated coordinates for each vertex of the individual mesh. The sparse pose estimation system (500) computes a deviation between a particular vertex of the estimated coordinates of the individual mesh and a corresponding vertex of the training 3D mesh of the first training real-world object, and computes a first portion of the second loss based on the calculated deviation between the particular vertex and the corresponding vertex. In some cases, the sparse pose estimation system (500) computes a first edge based on the difference between a first pair of estimated coordinates of a particular triangle of an individual mesh, and computes a second edge based on the difference between a second pair of coordinates of a corresponding triangle of the training 3D mesh. The sparse pose estimation system (500) computes a second portion of the second loss based on the deviation between the first edge and the second edge.

In some examples, the second loss among the plurality of losses comprises a segmentation loss. In such cases, the sparse pose estimation system (500) identifies a plurality of portions of the first training real-world object, each of the plurality of portions comprising a respective collection of triangles. The sparse pose estimation system (500) selects a particular portion among the plurality of portions. The sparse pose estimation system (500) retrieves a group of coordinates of a tensor of individual heatmaps corresponding to the particular portion. The sparse pose estimation system (500) computes a sum of scores associated with the retrieved group of coordinates. The sparse pose estimation system (500) compares the sum of scores to a threshold value, and based on the result of comparing the sum of scores to the threshold value, updates the parameters of the first machine learning model to maximize the sum of scores for the retrieved group of coordinates.

In some examples, the second loss among the multiple losses includes a spatial consistency loss. The sparse pose estimation system (500) obtains ground truth pixel values of a training image for a given set of center-of-gravity coordinates corresponding to individual triangles of a training 3D mesh. The sparse pose estimation system (500) calculates estimated pixel values based on the predicted posterior coordinates, and calculates spatial consistency loss based on the deviation between the ground truth pixel values and the estimated pixel values.

In some examples, the sparse pose estimation system (500) estimates offsets between vertices of a 3D mesh by applying a second machine learning model to a portion of the image. In some examples, the sparse pose estimation system (500) trains the second machine learning model based on the offset loss by calculating a first offset between a first vertex and a second vertex of a training 3D mesh and calculating a second offset between the first vertex and the second vertex of an individual mesh generated using a tensor of individual heatmaps. The sparse pose estimation system (500) calculates the offset loss based on the deviation between the first offset and the second offset.

In some examples, the second machine learning model estimates the depth of the 3D mesh. In some examples, a subset of the tensors of the heatmaps represents a set of vertices of multiple triangles, each associated with a score exceeding a threshold score. In some examples, the sparse pose estimation system (500) displays one or more AR elements on the image based on the 3D mesh.

In some examples, the object detection module (512) receives an image or video (501) from a camera of the user system (102) or another local and/or remote device. For example, the object detection module (512) may access a real-time video feed captured by the user system (102). The object detection module (512) may detect an input or request from a user to perform an AR action, such as launching an AR experience. For example, the object detection module (512) may detect a hand gesture of a user of the user system (102) in the real-time video feed. The hand gestures may include a pinch gesture, a snapping gesture, a pointing gesture, or any other suitable gesture. In some cases, the hand gesture taps or corresponds to a displayed option corresponding to AR experience actions, such as overlaying one or more AR graphics on a person depicted in the image or video (501) (e.g., the hands may be positioned so as to overlap). Additionally, or alternatively, the object detection module (512) can detect a voice input received from a user that includes a spoken command to perform an AR action, such as applying an AR-like item to a person depicted in an image or video (501) (e.g., a shirt, pants, or an entire outfit including pants and a shirt).

The object detection module (512) applies one or more trained machine learning models to the image or video (501) in response to a request to perform an AR action. The output of the machine learning models generates bounding boxes centered on each instance of the image depicting a particular object, such as a bounding box for each person depicted in the image or video (501). The object detection module (512) then crops a portion of the image or video (501) corresponding to each bounding box. The object detection module (512) provides the cropped portions of the image or video (501) or portions thereof to a heatmap prediction module (513) to predict a tensor of heatmaps for the object(s) depicted in the cropped portions. The object detection module (512) also provides the cropped portions of the image or video (501) or portions thereof to a 3D key point estimation module (515) to predict or estimate 3D key points of the object(s) depicted in the cropped portions.

In some examples, the heatmap prediction module (513) is trained to predict a tensor of heatmaps for an object depicted in a portion of a received image. This tensor corresponds to a sparse pose of the object depicted in the portion of the received image and can be used to generate an initial 3D mesh model by being processed by the 3D mesh module (514). Specifically, the tensor of sparse poses or heatmaps provides a score or probability of a mapping between a pixel of the object in the portion of the image and an individual vertex of the 3D mesh. That is, the 3D mesh is composed of a plurality of triangles, each having three vertices. Each heatmap in the tensor of heatmaps is associated with a corresponding single vertex in one of the plurality of triangles, and provides a probability distribution representing the likelihood that the coordinates of the single vertex map to a pixel of the object depicted in the image map. The coordinates (e.g., X, Y) of the single vertex with the highest probability or score are the most likely vertices located on the 3D mesh.

In some examples, the initial 3D mesh module (514) identifies vertices in the tensor of heatmaps with the largest values, scores, or probabilities. The initial 3D mesh module (514) then generates a 3D mesh by forming triangles using the identified vertices. In parallel or concurrently with the initial 3D mesh module (514) processing the output of the heatmap prediction module (513) to generate the initial 3D mesh, the intermediate 3D mesh module (516) processes 3D key points representing the skeleton of an object depicted in the image or video (501). The skeleton may be output by a 3D key point estimation module (515), which implements a machine learning model trained to generate or predict skeletons. The intermediate 3D mesh module (516) generates the intermediate 3D mesh using the predicted skeleton.

In some examples, both the initial 3D mesh output by the initial 3D mesh module (514) and the intermediate 3D mesh output by the intermediate 3D mesh module (516) are provided together to the reconstructed 3D mesh module (517). The reconstructed 3D mesh module (517) fuses or combines the initial 3D mesh (generated using sparse poses) with the intermediate 3D mesh (generated using 3D key points) to form a final or reconstructed 3D mesh. This reconstructed 3D mesh is provided to the AR effects module (519) for use in applying one or more AR graphics corresponding to the selected AR experience.

In some examples, the AR effects module (519) retrieves one or more AR graphics and applies or overlays the one or more AR graphics on an image or video (501) using the reconstructed 3D mesh. The output of the AR effects module (519) is provided to the image rendering module (518) to display the image to a user of the user system (102), and continuously updates the positioning of the one or more AR graphics using tracking information of objects depicted in the image calculated by the heat map prediction module (513) in real time.

In some examples, the heatmap prediction module (513) implements one or more machine learning models that are trained and supervised using ground truth density-poses of objects depicted in training images and based on ground truth or pseudo-ground truth 3D meshes. The heatmap prediction module (513) receives training data including a plurality of sets of training images depicting real-world objects and a corresponding ground truth density-pose of each object depicted in the training images and a ground truth or pseudo-ground truth 3D mesh of each object. The machine learning models are trained based on a plurality of different types of losses, including cross-entropy loss (or center of gravity loss), partial segmentation loss, vertex localization loss, spatial consistency loss, and/or offset loss.

During training, the heatmap prediction module (513) generates predictions of a tensor of heatmaps of objects depicted in one or more training images. The heatmap prediction module (513) uses one or more of a plurality of losses to calculate various deviations from supervised data (e.g., generated from ground truth dense poses and/or ground truth 3D meshes) and updates the parameters of the machine learning models based on the various deviations. The heatmap prediction module (513) then repeats this training process for other sets of training data until a stopping criterion is reached. At that point, the heatmap prediction module (513) completes training, and the parameters of the machine learning models are stored and used to predict sparse poses for new images or videos received in real time. The calculation of deviations based on each of these losses is discussed in turn below.

FIG. 6 is a schematic representation of a cross-entropy loss (600) (or center-of-gravity loss) calculation for training a machine learning model of a sparse pose estimation system (500), according to some examples. Specifically, as illustrated in FIG. 6 , a machine learning model (610) (e.g., implemented by a heatmap prediction module (513)) generates a tensor of heatmaps (620) for first training real-world objects depicted in received images (e.g., first training images). To calculate the cross-entropy loss, the heatmap prediction module (513) obtains a first training image and a first ground truth dense pose corresponding to the first training real-world object. The heatmap prediction module (513) uses the first ground truth density to compute a set of barycentric coordinates (e.g., v _k , v _i , v _j ) of individual triangles (650) of the training 3D mesh (640) corresponding to specific pixels of the first training real-world object. The heatmap prediction module (513) identifies UV coordinates (652) corresponding to the centroid of the set of barycentric coordinates (e.g., v _k , v _i , v _j ). The set of barycentric coordinates (e.g., v _k , v _i , v _j ) represents a range of possible coordinates that may correspond to each vertex of the triangle (650) based on the UV coordinates (652).

The heatmap prediction module (513) generates predicted posterior coordinates (630) using a selection of a group of coordinates of a tensor of individual heatmaps corresponding to a particular pixel of a first training real-world object. For example, the heatmap prediction module (513) selects, for a particular pixel, a group of coordinates (X, Y) of some or all of the triangles of the 3D mesh (640). In some examples, the heatmap prediction module (513) selects a group of coordinates for a subset of triangles that has a maximum or greater score or probability than other coordinates (e.g., a triplet of coordinates) for that subset of triangles. The heatmap prediction module (513) computes a deviation corresponding to the cross-entropy loss between the predicted posterior coordinates (630) and the corresponding centroid coordinates of the individual triangles (650) (e.g., a set of centroid coordinates (e.g., v _k , v _i , v _j )) or corresponding UV coordinates (652). The heatmap prediction module (513) updates the parameters of the machine learning model (610) based on the deviation.

Specifically, the cross-entropy loss transforms a dense-pose-based annotation of a pixel x with UV coordinates u into constraints on the sparse pose scores s = S[x1, x2, :] at that pixel. This is a competition task between sparse-pose vertices for the occupancy of a given pixel, implemented via a softmax-based posterior. If a vertex v lands exactly on a given pixel, the system imposes a standard cross-entropy loss on that point using its one-hot encoding. Since the dense-pose ground truth is randomly collected, the UV value u of any pixel will typically be represented as a combination of three barycenter coordinates (βi, βj, βk) for a single triangle of the underlying mesh formed by vertices (i, j, k). These can be interpreted as a discrete distribution supported on the triangle vertices and can be used to penalize the softmax-based posterior using the general definition of the cross-entropy loss:

Here

That is, the sparse pose-based posterior q is forced to align with the center-of-gravity distribution q. If there is a direct pixel-to-vertex correspondence, this corresponds to a "one-hot" cross-entropy loss.

Here, v will be the ground truth vertex.

FIG. 7 is a schematic representation of the calculation of a partial segmentation loss (700) for training a machine learning model of a sparse pose estimation system (500), according to some examples. Specifically, as illustrated in FIG. 7 , a mesh (710) may be segmented and labeled into multiple parts, such as a head part, an arm part (720), and a body part (730). Each triangle of the mesh (710) may be assigned to a specific part. The partial segmentation loss (700) calculates a deviation based on whether a vertex and/or triangle associated with a set of specific vertices of a sparse pose calculated by the machine learning model (610) falls within the correct part. To calculate this loss, the heatmap prediction module (513) selects a specific part from among the multiple parts and retrieves a group of coordinates of the tensor of individual heatmaps corresponding to the specific part. The heatmap prediction module (513) calculates the sum of probabilities or scores associated with a group of retrieved coordinates and compares the sum of scores with a threshold (e.g., a maximum value such as '1'). The heatmap prediction module (513) updates the parameters of the machine learning model (610) based on the result of comparing the sum of scores with the threshold to maximize the sum of scores for the group of retrieved coordinates.

Specifically, for every visible body pixel x, the part label l is known, and the set of vertices spanning all body parts containing S _l is also known. Based on this information, the part-level posterior distribution can be computed as follows:

By collapsing the sparse pose-level softmax posterior q _v , this posterior distribution is supervised based on the standard cross-entropy loss obtained from one-hot encoding of the partial labels.

FIG. 8 is a schematic representation of the calculation of a vertex localization loss (800) for training a machine learning model of a sparse pose estimation system (500), according to some examples. To calculate this loss, the heatmap prediction module (513) integrates scores of a group of possible coordinates for each vertex of an individual mesh corresponding to a first training real-world object to generate estimated coordinates for each vertex of the individual mesh. For example, each vertex (X _k , Y _k ) may be calculated based on the following mathematical equation:

Here, H _k corresponds to a tensor of heatmaps provided by the machine learning model (610).

For example, as illustrated in FIG. 8, a given triangle (810) is formed by a set of three adjacent vertices corresponding to possible vertex coordinates resulting from the integration of scores. The heatmap prediction module (513) calculates the deviation between a specific vertex coordinate (812) of the estimated coordinates of an individual mesh (e.g., a set of three adjacent vertices) and the corresponding vertex coordinate (822) of the ground truth (training) 3D mesh of the first training real-world object. The heatmap prediction module (513) calculates a first part of the loss (800) based on the calculated deviation between the specific vertex and the corresponding vertex. That is, the first part of the loss (800) can be calculated as follows:

In some examples, the heatmap prediction module (513) calculates a first edge (816) based on the difference between a first pair of estimated coordinates (812 and 814) of a particular triangle of an individual mesh. The heatmap prediction module (513) calculates a second edge (826) based on the difference between a second pair of coordinates (822 and 824) of a corresponding triangle of the ground truth 3D mesh. The heatmap prediction module (513) calculates a second portion of the loss (800) based on the deviation between the first edge and the second edge. That is, the second portion of the loss (800) may be calculated as follows:

.

FIG. 9 is a schematic representation of the computation of a spatial consistency loss (900) for training a machine learning model of a sparse pose estimation system (500), according to some examples. To compute this loss (900), a heatmap prediction module (513) obtains ground truth pixel values of a training image for a given set of center-of-gravity coordinates corresponding to individual triangles (920) of a ground truth 3D mesh (910). The heatmap prediction module (513) computes estimated pixel values based on the predicted posterior coordinates (940), and computes a spatial consistency loss (900) based on the deviation between the ground truth pixel values and the estimated pixel values. Specifically, this loss transforms a dense pose annotation of a pixel (x, u) into constraints on sparse pose heatmaps Sv = S[:, :, v], v ∈ (i, j, k) of three vertices (i, j, k) that are used to compute the barycenter coordinates (βi, βj, βk) of u. These three barycenter coordinates allow the heatmap prediction module (513) to localize the corresponding point of the pixel on the surface as a weighted combination of these three vertices. When projected onto an image, this relationship can still be approximately maintained. The loss equation, when combined with the barycenter weights, specifies the estimated 2D positions of the three vertices, and the position of x should be recovered according to:

Here

Here, the heatmap prediction module (513) obtains the 2D position xv of each vertex v as a differentiable SoftArgmax operation for each heatmap. In effect, this allows the heatmap to properly localize these vertices even in the absence of direct annotations to them.

In some examples, the heatmap prediction module (513) calculates an offset loss. To do so, the heatmap prediction module (513) applies another machine learning model to a portion of the image to estimate offsets between adjacent and non-adjacent vertices of the 3D mesh. The heatmap prediction module (513) calculates a first offset between a first vertex and a second vertex of the ground truth 3D mesh and a second offset between the first vertex and the second vertex of the generated individual mesh using the tensor of the individual heatmaps. The heatmap prediction module (513) calculates an offset loss based on the deviation between the first offset and the second offset. In some cases, this machine learning model can be used to estimate the depth of the 3D mesh.

That is, a perspective camera model is adopted. Each vertex can be located on a ray intersecting the image plane at a sparse pose-based 2D position. The heatmap prediction module (513) limits 3D lifting to the task of estimating the depth of a vertex along that ray. In some cases, the heatmap prediction module (513) directly predicts the depth of each vertex. In some cases, the heatmap prediction module (513) uses an extension of depth prediction used in 3D pose estimation. The heatmap prediction module (513) predicts the depth of a vertex through its depth relationship to its parent skeletal joint. Vertex-joint assignments are obtained from a precomputed partial segmentation of a low-poly mesh that yields a single parent for every vertex. This allows the heatmap prediction module (513) to treat the problems of 3D pose and 3D mesh estimation as two modular tasks that can be performed in a cascade and trained on complementary datasets. Specifically, the heatmap prediction module (513) predicts the inverse depth, which is better behaved and directly correlated with the object scale. Given the estimated inverse depth of a parent joint sj and the position prediction for the 2D vertex projection = (x,y), the heatmap prediction module (513) predicts the inverse depth (or scale) sv of a vertex v as follows: s= _sj +S[x,y,v], where S is an H tensor that predicts the relative depth of every vertex with respect to its parent node.

Assuming that the heatmap prediction module (513) has a focal length f, the heatmap prediction module (513) can unproject a vertex to its 3D position as follows:

.

To supervise the depths of individual vertices, the heatmap prediction module (513) relies on weak supervision, since direct mesh supervision may not exist.

FIG. 10 is a flowchart of a process (1000) performed by a sparse pose estimation system (500), according to some examples. Although the flowchart may depict operations as a sequential process, many operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. The process terminates when its operations are completed. The process may correspond to a method, procedure, or the like. The steps of these methods may be performed in whole or in part, may be performed in conjunction with some or all of the steps of other methods, and may be performed by any number of different systems, or any portion thereof, such as a processor included in any of these systems.

In operation (1001), a sparse pose estimation system (500) (e.g., a user system (102) or a server) receives an image containing a depiction of a real-world object in a real-world environment, as discussed above.

In operation (1002), the sparse pose estimation system (500) applies a first machine learning model to a portion of an image depicting a real-world object, as discussed above, to predict a tensor of heatmaps representing vertex positions of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object. A first heatmap of the tensor of heatmaps represents a first group of possible coordinates for a first vertex of a first triangle among the plurality of triangles, and a second heatmap of the tensor of heatmaps represents a second group of possible coordinates for a second vertex of the first triangle.

In operation (1003), the sparse pose estimation system (500) processes the tensor of heatmaps to select a subset of the tensor of heatmaps, the subset of the tensor of heatmaps comprising a first coordinate of a first group of possible coordinates and a second coordinate of a second group of possible coordinates as discussed above.

In operation (1004), the sparse pose estimation system (500) generates a 3D mesh based on a selected subset of the tensor of heatmaps, as discussed above.

machine architecture

FIG. 11 is a schematic representation of a machine (1100) on which instructions (1102) (e.g., software, programs, applications, applets, apps, or other executable code) may be executed to cause the machine (1100) to perform one or more of the methodologies discussed herein. For example, the instructions (1102) may cause the machine (1100) to perform one or more of the methods described herein. The instructions (1102) transform a typical, non-programmed machine (1100) into a specific machine (1100) programmed to perform the described and exemplified functions in the described manner. The machine (1100) may operate as a standalone device or may be coupled (e.g., networked) to another machine. In a networked deployment, the machine (1100) can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine (1100) can include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), another smart device, a web appliance, a network router, a network switch, a network bridge, or any machine that can sequentially or otherwise execute instructions (1102) that specify actions to be taken by the machine (1100). Additionally, while a single machine (1100) is illustrated, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions (1102) to perform any one or more of the methodologies discussed herein. The machine (1100) may include, for example, any one of a number of server devices forming part of a user system (102) or an interactive server system (110). In some examples, the machine (1100) may also include both client and server systems, with certain operations of a particular method or algorithm being performed on the server side and certain operations of a particular method or algorithm being performed on the client side.

The machine (1100) may include processors (1104), memory (1106), and input/output (I/O) components (1108), which may be configured to communicate with each other via a bus (1110). In an example, the processors (1104) (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor (1112) and a processor (1114) that execute instructions (1102). The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes also referred to as "cores") capable of executing instructions concurrently. While FIG. 11 illustrates multiple processors (1104), the machine (1100) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

The memory (1106) includes a main memory (1116), a static memory (1118), and a storage unit (1120), all of which are accessible to the processors (1104) via the bus (1110). The main memory (1106), the static memory (1118), and the storage unit (1120) store instructions (1102) that implement any one or more of the methodologies or functions described herein. The instructions (1102) may also, during their execution by the machine (1100), reside, completely or partially, within the main memory (1116), within the static memory (1118), within a machine-readable medium (1122) within the storage unit (1120), within at least one of the processors (1104) (e.g., within a cache memory of a processor), or any suitable combination thereof.

The I/O components (1108) may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, and the like. The specific I/O components (1108) included in a particular machine will depend on the type of machine. For example, a portable machine, such as a mobile phone, may include a touch input device or other such input mechanism, whereas a headless server machine may not include such a touch input device. It will be appreciated that the I/O components (1108) may include many other components not shown in FIG. 11. In various examples, the I/O components (1108) may include user output components (1124) and user input components (1126). User output components (1124) may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. User input components (1126) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), tactile input components (e.g., a physical button, a touch screen that provides position and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), etc. Any biometric data collected by the biometric components is captured and stored with user consent and deleted upon user request. Additionally, this biometric data may be used for very limited purposes, such as identity verification. To ensure the limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is restricted, at best, to authorized personnel. Any use of biometric data may be strictly limited to identity verification purposes, and this data will not be shared or sold to any third party without the user's explicit consent. Furthermore, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

In further examples, the I/O components (1108) may include, among various other components, biometric components (1128), motion components (1130), environmental components (1132), or position components (1134). For example, biometric components (1128) include components that detect facial expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identify a person (e.g., voice identification, retina identification, face identification, fingerprint identification, or electroencephalogram-based identification), and the like. Motion components (1130) include acceleration sensor components (e.g., an accelerometer), gravity sensor components, rotation sensor components (e.g., a gyroscope), and the like.

Environmental components (1132) include, for example, one or more cameras (having still image/photo and video capabilities), a light sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the air), or other components that can provide instructions, measurements, or signals corresponding to the surrounding physical environment.

With respect to cameras, the user system (102) may have a camera system that includes, for example, front cameras on the front of the user system (102) and rear cameras on the back of the user system (102). The front cameras may be used, for example, to capture still images and video (e.g., “selfies”) of a user of the user system (102), which may then be augmented with the augmented data described above (e.g., filters). The rear cameras may be used, for example, to capture still images and video in a more traditional camera mode, which images may similarly be augmented with the augmented data. In addition to the front and rear cameras, the user system (102) may also include a 360° camera for capturing 360° photos and videos.

Additionally, the camera system of the user system (102) may include dual rear cameras (e.g., a main camera as well as a depth-sensing camera) on the front and rear sides of the user system (102), or even triple, quadruple, or quintuple rear camera configurations. Such multiple camera systems may include, for example, a wide camera, an ultra-wide camera, a telephoto camera, a macro camera, and a depth sensor.

Position components (1134) include location sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., an altimeter or a barometer that detects air pressure from which altitude can be derived), and orientation sensor components (e.g., a magnetometer).

Communications may be implemented using a wide variety of technologies. The I/O components (1108) further include communication components (1136) operable to couple the machine (1100) to a network (1138) or devices (1140) via their respective couplings or connections. For example, the communication components (1136) may include a network interface component or other suitable device for interfacing with the network (1138). In further examples, the communication components (1136) may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., ^Bluetooth® Low Energy), Wi- ^Fi® components, and other communication components that provide communication via other modalities. The devices (1140) may be another machine or any of a wide variety of peripheral devices (e.g., peripheral devices coupled via USB).

Additionally, the communication components (1136) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (1136) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors that detect one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph ^™ , MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones that identify tagged audio signals). Additionally, various information may be derived through the communication components (1136), such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting NFC beacon signals that can indicate a specific location, etc.

Various memories (e.g., main memory (1116), static memory (1118), and memory of the processors (1104)) and storage units (1120) may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions (1102)), when executed by the processors (1104), cause various operations to be implemented as disclosed examples.

The commands (1102) may be transmitted or received over a network (1138) using a transmission medium, via a network interface device (e.g., a network interface component included in the communication components (1136)), and using any one of several well-known transport protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, the commands (1102) may be transmitted or received using a transmission medium via coupling to devices (1140) (e.g., peer-to-peer coupling).

Software architecture

FIG. 12 is a block diagram (1200) illustrating a software architecture (1202) that may be installed on any one or more devices described herein. The software architecture (1202) is supported by hardware, such as a machine (1204) that includes processors (1206), memory (1208), and I/O components (1210). In this example, the software architecture (1202) may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture (1202) includes layers such as an operating system (1212), libraries (1214), frameworks (1216), and applications (1218). Operationally, the applications (1218) initiate API calls (1220) through the software stack and receive messages (1222) in response to the API calls (1220).

The operating system (1212) manages hardware resources and provides common services. The operating system (1212) includes, for example, a kernel (1224), services (1226), and drivers (1228). The kernel (1224) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (1224) provides, among other functionalities, memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services (1226) may provide other common services for other software layers. Drivers (1228) are responsible for controlling or interfacing with the underlying hardware. For example, drivers (1228) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, and the like.

Libraries (1214) provide common low-level infrastructure used by applications (1218). Libraries (1214) may include system libraries (1230) (e.g., the C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. Additionally, the libraries (1214) may include API libraries (1232), such as media libraries (e.g., libraries that support presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), H.264 or Advanced Video Coding (AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), Portable Network Graphics (PNG), etc.), graphics libraries (e.g., OpenGL framework used to render graphical content on a display in two dimensions (2D) and three dimensions (3D), database libraries (e.g., SQLite, which provides various relational database functionality), web libraries (e.g., WebKit, which provides web browsing functionality), and the like. The libraries (1214) may also include a wide variety of other libraries (1234) that provide many different APIs to the application (1218).

Frameworks (1216) provide common high-level infrastructure used by applications (1218). For example, frameworks (1216) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (1216) may provide a wide range of other APIs that may be used by applications (1218), some of which may be specific to a particular operating system or platform.

In an example, the applications (1218) may include a wide variety of other applications, such as a home application (1236), a contacts application (1238), a browser application (1240), a book reader application (1242), a location application (1244), a media application (1246), a messaging application (1248), a game application (1250), and a third-party application (1252). The applications (1218) are programs that execute functions defined in the programs. Various programming languages structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language), may be employed to create one or more of the applications (1218). In a specific example, a third-party application (1252) (e.g., an application developed using an ANDROID ^™ or IOS ^™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system, such as IOS ^™ , ANDROID ^™ , WINDOWS® Phone, or another mobile operating system. In this example, the third-party application (1252) may invoke API calls (1220) provided by the operating system 1212 to facilitate the functionalities described herein.

System with head-wearable device

FIG. 13 illustrates a system (1300) including a head-worn device (116) having a selector input device, according to some examples. FIG. 13 is a high-level functional block diagram of an exemplary head-worn device (116) communicatively coupled to a mobile device (114) and various server systems (1304) (e.g., an interaction server system (110)) via various networks (108).

The head-wearable device (116) includes one or more cameras, each of which may be, for example, a visible light camera (1306), an infrared emitter (1308), and an infrared camera (1310).

The mobile device (114) connects to the head-worn device (116) using both a low-power wireless connection (1312) and a high-speed wireless connection (1314). The mobile device (114) is also connected to a server system (1304) and a network (1316).

The head-wearable device (116) further includes two image displays of the image display of the optical assembly (1318). The two image displays of the optical assembly (1318) include one associated with a left lateral side of the head-wearable device (116) and one associated with a right lateral side. The head-wearable device (116) also includes an image display driver (1320), an image processor (1322), low power circuitry (1324), and high speed circuitry (1326). The image display of the optical assembly (1318) is for presenting images and videos, including images that may include a graphical user interface for a user of the head-wearable device (116).

The image display driver (1320) commands and controls the image display of the optical assembly (1318). The image display driver (1320) may either pass image data directly to the image display of the optical assembly (1318) for presentation, or may convert the image data into a signal or data format suitable for passing to an image display device. For example, the image data may be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, etc., and the still image data may be formatted according to compression formats such as PNG (Portable Network Group), JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or EXIF (exchangeable image file format).

A head-wearable device (116) includes a frame and stems (or temples) extending from lateral sides of the frame. The head-wearable device (116) further includes a user input device (1328) (e.g., a touch sensor or a push button) comprising an input surface on the head-wearable device (116). The user input device (1328) (e.g., a touch sensor or a push button) receives input selections from a user for manipulating a graphical user interface of a presented image.

The components depicted in FIG. 13 for the head-wearable device (116) are positioned on one or more circuit boards, e.g., a PCB or a flexible PCB, at the rims or temples. Alternatively, or additionally, the depicted components may be positioned on chunks, frames, hinges, or bridges of the head-wearable device (116). The left and right visible-light cameras (1306) may include digital camera elements, such as complementary metal oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other respective visible or optical capturing elements that may be used to capture data, including images of scenes with unknown objects.

The head-wearable device (116) includes a memory (1302) storing instructions for performing a subset or all of the functions described herein. The memory (1302) may also include a storage device.

As illustrated in FIG. 13, the high-speed circuit (1326) includes a high-speed processor (1330), memory (1302), and high-speed wireless circuitry (1332). In some examples, an image display driver (1320) is coupled to the high-speed circuitry (1326) and is operated by the high-speed processor (1330) to drive left and right image displays of the image display of the optical assembly (1318). The high-speed processor (1330) may be any processor capable of managing the operation and high-speed communications of any general computing system required for the head-wearable device (136). The high-speed processor (1330) includes the processing resources necessary to manage high-speed data transfer over a high-speed wireless connection (1314) to a wireless local area network (WLAN) using the high-speed wireless circuitry (1332). In certain instances, the high-speed processor (1330) executes an operating system, such as the LINUX operating system or another such operating system of the head-worn device (116), which is stored in the memory (1302) for execution. In addition to any other responsibilities, the high-speed processor (1330) executing the software architecture for the head-worn device (116) is used to manage data transfer with the high-speed wireless circuit (1332). In certain instances, the high-speed wireless circuit (1332) is configured to implement the Institute of Electrical and Electronic Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some instances, other high-speed communication standards may be implemented by the high-speed wireless circuit (1332).

The low-power wireless circuitry (1334) and high-speed wireless circuitry (1332) of the head-wearable device (136) may include short-range transceivers (Bluetooth ^™ ) and wireless wide-area, local, or wide-area network transceivers (e.g., cellular or WiFi). A mobile device (114), including a transceiver that communicates via the low-power wireless connection (1312) and high-speed wireless connection (1314), may be implemented using details of the architecture of the head-wearable device (116), which may be other elements of the network (1316).

Memory (1302) includes any storage device capable of storing various data and applications, including, in particular, camera data generated by the left and right visible light cameras (1306), the infrared camera (1310), and the image processor (1322), as well as images generated for display on the image display of the image display of the optical assembly (1318) by the image display driver (1320). Memory (1302) is illustrated as being integrated with high-speed circuitry (1326), but in some examples, memory (1302) may be a separate, standalone element of the head-wearable device (116). In these specific examples, electrical routing lines may provide access to memory (1302) from image processor (1322) or low-power processor (1336) through a chip including high-speed processor (1330). In some examples, the high-speed processor (1330) may manage the addressing of the memory (1302) such that the low-power processor (1336) boots the high-speed processor (1330) whenever a read or write operation involving the memory (1302) is required.

As illustrated in FIG. 13, a low-power processor (1336) or high-speed processor (1330) of a head-wearable device (136) may be coupled to a camera (visible light camera (1306); infrared emitter (1308) or infrared camera (1310)), an image display driver (1320), a user input device (1328) (e.g., a touch sensor or push button), and a memory (1302).

The head-wearable device (116) is connected to a host computer. For example, the head-wearable device (116) is paired with a mobile device (114) via a high-speed wireless connection (1314) or connected to a server system (1304) via a network (1316). The server system (1304) may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with the mobile device (114) and the head-wearable device (116) via the network (1316).

The mobile device (114) includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication over a network (1316), a low-power wireless connection (1312), or a high-speed wireless connection (1314). The mobile device (114) may additionally store at least a portion of instructions for generating binaural audio content in a memory of the mobile device (114) to implement the functionality described herein.

The output components of the head-wearable device (116) include visual components such as displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light emitting diode (LED) displays, projectors, or waveguides. The image displays of the optical assembly are driven by an image display driver (1320). The output components of the head-wearable device (116) additionally include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, and the like. Input components of the head-worn device (116), the mobile device (114), and the server system (1304), such as the user input device (1328), may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing devices), tactile input components (e.g., a physical button, a touch screen that provides position and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

The head-wearable device (136) may also include additional peripheral device elements. These peripheral device elements may include biometric sensors, additional sensors, or display elements that are integrated with the head-wearable device (116). For example, the peripheral device elements may include any I/O components, including output components, motion components, position components, or any other such elements described herein.

For example, biometric components may include components that detect facial expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biometric signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalography-based identification). Any biometrics collected by biometric components are captured and stored only with user permission and are deleted upon user request. Additionally, such biometric data may be used for very limited purposes, such as identity verification. To ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is restricted, at best, to authorized personnel. Any use of biometric data may be strictly limited to identity verification purposes, and such data will not be shared or sold to any third party without the user's explicit consent. Additionally, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of such sensitive information.

Motion components may include acceleration sensor components (e.g., an accelerometer), gravity sensor components, rotation sensor components (e.g., a gyroscope), etc. Position components may include position sensor components that generate location coordinates (e.g., a Global Positioning System (GPS) receiver component), Wi-Fi or Bluetooth ^™ transceivers that generate positioning system coordinates, altitude sensor components (e.g., an altimeter or a barometer that detects air pressure from which altitude can be derived), heading sensor components (e.g., a magnetometer), etc. These positioning system coordinates may also be received from the mobile device (114) via the low power wireless connection (1312) and the high speed wireless connection (1314) via the low power wireless circuitry (1334) or the high speed wireless circuitry (1332).

Glossary

A "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible media for facilitating communication of such instructions. The instructions may be transmitted or received over a network using a transmission medium via a network interface device.

A "client device" refers to any machine that interfaces to a communications network to obtain resources, for example, from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, a portable digital assistant (PDAs), a smartphone, a tablet, an ultrabook, a netbook, a laptop, a multi-processor system, a microprocessor-based or programmable consumer electronics device, a game console, a set-top box, or any other communications device that a user can use to access a network.

A "communications network" refers to one or more portions of a network, which may be, for example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or portion of a network may comprise a wireless or cellular network, and the combination may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless combination. In this example, the combination may implement any of a variety of data transport technologies, such as Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP) including 3G, Fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, others defined by various standards setting organizations, other long-distance protocols, or other data transport technologies.

A "component" refers to a device, physical entity, or logic having boundaries defined by, for example, function or subroutine calls, branch points, APIs, or other technologies that provide for the partitioning or modularization of specific processing or control functions. Components can be combined with other components through their interfaces to execute a machine process. A component may be a packaged functional hardware unit designed to be used with other components and part of a program that performs a specific function among related functions. Components may comprise either software components (e.g., code embodied on a machine-readable medium) or hardware components.

A "hardware component" is a tangible unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a group of processors) of a computer system may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform specific operations as described herein.

Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processors. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the functions for which they are configured and are no longer general-purpose processors. It will be appreciated that the decision to implement a hardware component mechanically, in dedicated, permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a tangible entity, i.e., an entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or to perform particular operations described herein.

Considering examples where hardware components are temporarily configured (e.g., programmed), it is not necessary for each hardware component to be configured or instantiated at any one time instance. For example, if a hardware component comprises a general-purpose processor configured by software to be a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors (e.g., including different hardware components) at different times. Thus, the software configures a particular processor or processors to, for example, configure a particular hardware component at one time instance and a different hardware component at a different time instance. Hardware components can provide information to and receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled. When multiple hardware components exist simultaneously, communication can be achieved via signal transmission between or among two or more of the hardware components (e.g., via appropriate circuits and buses). In instances where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information in memory structures accessible to the multiple hardware components. For example, one hardware component may perform an operation and store the output of the operation in a memory device communicatively coupled to it. Additional hardware components may then access the memory device at a later time to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and manipulate resources (e.g., collections of information). The various operations of the exemplary methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented components that operate to perform one or more operations or functions described herein.

As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be at least partially processor-implemented, with a particular processor or processors being examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components. Furthermore, one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), which are accessible over a network (e.g., the Internet) and via one or more suitable interfaces (e.g., APIs). Performance of certain of the operations may not only reside within a single machine, but may also be distributed among processors deployed across multiple machines. In some examples, the processors or processor-implemented components may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components may be distributed across multiple geographic locations.

The term "computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Thus, the terms encompass both storage devices/media and carrier waves/modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably throughout the present disclosure. A "short-term message" refers to a message that is accessible for a limited time duration, for example. A short-term message can be text, an image, a video, etc. The access time for a short-term message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the receiver. Regardless of the setting technique, the message is temporary.

"Machine storage medium" refers to, for example, a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and/or data. Accordingly, the term may be understood to include, but is not limited to, solid-state memories, including memory internal or external to processors, and optical and magnetic media. Specific examples of machine storage media, computer storage media, and device storage media include, by way of example, non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably throughout the present disclosure.

The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term "signal medium." A "non-transitory computer-readable storage medium" refers to a tangible medium that can store, encode, or carry instructions for execution by a machine, for example. A "signal medium" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, for example, and includes digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in the present disclosure.

A "user device" refers to, for example, a device that is accessed, controlled, or owned by a user and with which the user interacts to perform actions or interactions with other users or computer systems. A "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible media for facilitating communication of such instructions. The instructions may be transmitted or received over a network using a transmission medium via a network interface device. A "client device" refers to any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, a portable digital assistant (PDA), a smartphone, a tablet, an ultrabook, a netbook, a laptop, a multi-processor system, a microprocessor-based or programmable consumer electronics device, a game console, a set-top box, or any other communications device that a user can use to access a network.

A "communications network" refers to one or more portions of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or portion of a network may comprise a wireless or cellular network, and the combination may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless combination. In this example, the combination may implement any of a variety of data transport technologies, such as Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP) including 3G, Fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, others defined by various standards setting organizations, other long-distance protocols, or other data transport technologies.

The components may comprise either software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and may be configured or arranged in a particular physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a group of processors) of a computer system may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein.

Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform specific operations. The hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component may also include programmable logic or circuitry temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the functions for which they are configured and are no longer general-purpose processors. It will be appreciated that the decision to implement a hardware component mechanically, in dedicated, permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a tangible entity, i.e., an entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or to perform particular operations described herein.

The various operations of the exemplary methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented components that operate to perform one or more of the operations or functions described herein.

Changes and modifications to the disclosed examples may be made without departing from the scope of this disclosure. These and other changes or modifications are intended to be included within the scope of this disclosure as expressed in the following claims.

Claims (20)

As a method,
A step of receiving an image containing a description of a real-world object;
A step of predicting a tensor of heatmaps representing vertex positions of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object by applying a first machine learning model to a portion of the image, wherein a first heatmap of the tensor of heatmaps represents a first group of possible coordinates for a first vertex of a first triangle of the plurality of triangles, and a second heatmap of the tensor of heatmaps represents a second group of possible coordinates for a second vertex of the first triangle;
A step of selecting a subset of the tensor of the heatmaps from the tensor of the heatmaps, wherein the subset of the tensor of the heatmaps includes a first coordinate of the first group of the possible coordinates and a second coordinate of the second group of the possible coordinates; and
A method comprising generating the 3D mesh based on a selected subset of tensors of the heatmaps. In the first paragraph,
A step of predicting a plurality of bounding boxes for each of a plurality of real-world objects depicted in the image by applying a second machine learning model to the image; and
A method further comprising the step of selecting a first bounding box from the plurality of bounding boxes to identify a portion of the image. In claim 1 or 2,
A method for representing a sparse pose corresponding to the range of possible vertices of a dense pose by tensor of the above heatmaps. In any one of the first to third paragraphs,
The above 3D mesh includes an initial 3D mesh,
A step of predicting a set of 3D key points corresponding to the skeleton of the real-world object by applying a second machine learning model to a portion of the image;
In parallel with the initial 3D mesh, generating an intermediate 3D mesh using a set of 3D key points representing depth per vertex of the real-world object; and
A method further comprising the step of forming a reconstructed 3D mesh by fusing the initial 3D mesh with the intermediate 3D mesh. In any one of the first to fourth paragraphs,
A method further comprising the step of training the first machine learning model using training data comprising a plurality of training images depicting real-world objects and corresponding ground truth dense-poses and training 3D meshes corresponding to the training real-world objects, wherein each ground truth dense-pose represents coordinates of respective 3D meshes of the training real-world objects. In paragraph 5,
A method further comprising the step of training the first machine learning model by calculating multiple losses. In any one of claims 1 to 6,
The first loss among the above multiple losses includes cross entropy loss, and the method includes:
A step of obtaining a first training image among the plurality of training images and a first ground truth density pose corresponding to a first training real-world object depicted in the first training image;
A step of calculating the center of gravity coordinates of individual triangles of the training 3D mesh corresponding to specific pixels of the first training real-world object using the first measured density pose;
A step of estimating tensors of individual heatmaps corresponding to the first training real-world objects by applying the first machine learning model to the first training image;
generating predicted posterior coordinates using a selection of a group of coordinates of a tensor of said individual heatmaps corresponding to said particular pixel of said first training real-world object;
A step of calculating a deviation including the cross entropy loss between the predicted posterior coordinates and the center of gravity coordinates; and
A method further comprising the step of updating parameters of the first machine learning model based on the deviation. In paragraph 7,
A method further comprising: each heatmap of the tensor of said individual heatmaps representing a respective group of possible coordinates for an individual vertex of an individual triangle of an individual mesh corresponding to said first training real-world object; and updating scores associated with each group of said possible coordinates for said individual vertex based on said cross entropy loss. In any one of claims 1 to 8,
The second loss among the above multiple losses includes vertex localization loss, and the method includes:
generating estimated coordinates for each vertex of the individual mesh by integrating scores of the group of possible coordinates for each vertex of the individual mesh corresponding to the first training real-world object;
A step of calculating a deviation between a specific vertex of the estimated coordinates of the individual mesh and a corresponding vertex of the training 3D mesh of the first training real-world object; and
A method further comprising the step of calculating a first portion of the second loss based on the calculated deviation between the specific vertex and the corresponding vertex. In paragraph 9,
Computing a first edge based on the difference between a first pair of estimated coordinates of a specific triangle of the individual mesh;
Computing a second edge based on the difference between a second pair of coordinates of corresponding triangles of the training 3D mesh; and
A method further comprising the step of calculating a second portion of the second loss based on a deviation between the first edge and the second edge. In any one of claims 1 to 10,
The second loss among the above multiple losses includes a partial segmentation loss, and the method comprises:
A step of identifying a plurality of portions of the first training real-world object, each of the plurality of portions comprising a respective collection of triangles;
A step of selecting a specific portion from among the above multiple portions;
A step of retrieving a group of coordinates of the tensor of the individual heatmaps corresponding to the specific portion;
A step of calculating the sum of scores associated with the group of coordinates searched above;
a step of comparing the sum of the above scores with a threshold value; and
A method further comprising a step of maximizing the sum of scores for the group of retrieved coordinates by updating parameters of the first machine learning model based on a result of comparing the sum of the scores with the threshold value. In any one of claims 1 to 11,
The second loss among the above multiple losses includes spatial consistency loss, and the method comprises:
Obtaining ground truth pixel values of the first training image for a given set of center-of-gravity coordinates corresponding to the individual triangles of the training 3D mesh;
A step of calculating an estimated pixel value based on the predicted posterior coordinates; and
A method further comprising the step of calculating the spatial consistency loss based on the deviation between the measured pixel value and the estimated pixel value. In any one of claims 1 to 12,
A method further comprising the step of applying a second machine learning model to a portion of the image to estimate offsets between vertices of the 3D mesh. In any one of claims 1 to 13,
Compute a first offset between a first vertex and a second vertex of the training 3D mesh;
Compute a second offset between the first vertex and the second vertex of the individual mesh generated using the tensor of the individual heatmaps;
A method further comprising the step of training the second machine learning model based on the offset loss by calculating an offset loss based on a deviation between the first offset and the second offset. In any one of claims 1 to 14,
The above second machine learning model is a method for estimating the depth of the 3D mesh. In any one of claims 1 to 15,
A method in which a subset of the tensors of the above heatmaps each represents a set of vertices of the plurality of triangles associated with a score exceeding a threshold score. In any one of claims 1 to 16,
A method further comprising the step of displaying one or more augmented reality (AR) elements on the image based on the 3D mesh. As a system,
at least one processor; and
At least one memory component having stored therein instructions that, when executed by said at least one processor, cause said at least one processor to perform operations, said operations comprising:
The act of receiving an image containing a depiction of a real-world object;
An operation of predicting a tensor of heatmaps representing vertex positions of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object by applying a first machine learning model to a portion of the image, wherein a first heatmap of the tensor of heatmaps represents a first group of possible coordinates for a first vertex of a first triangle of the plurality of triangles, and a second heatmap of the tensor of heatmaps represents a second group of possible coordinates for a second vertex of the first triangle;
An operation of selecting a subset of the tensor of the heatmaps from the tensor of the heatmaps, wherein the subset of the tensor of the heatmaps includes a first coordinate of the first group of the possible coordinates and a second coordinate of the second group of the possible coordinates; and
A system comprising an operation for generating the 3D mesh based on a selected subset of tensors of the heatmaps. In Article 18,
The above actions further include an action of displaying one or more augmented reality (AR) elements on the image based on the 3D mesh. A non-transitory computer-readable storage medium having instructions stored thereon,
The above instructions, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising:
The act of receiving an image containing a depiction of a real-world object;
An operation of predicting a tensor of heatmaps representing vertex positions of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object by applying a first machine learning model to a portion of the image, wherein a first heatmap of the tensor of heatmaps represents a first group of possible coordinates for a first vertex of a first triangle of the plurality of triangles, and a second heatmap of the tensor of heatmaps represents a second group of possible coordinates for a second vertex of the first triangle;
An operation of selecting a subset of the tensor of the heatmaps from the tensor of the heatmaps, wherein the subset of the tensor of the heatmaps includes a first coordinate of the first group of the possible coordinates and a second coordinate of the second group of the possible coordinates; and
A non-transitory computer-readable storage medium comprising an operation for generating the 3D mesh based on a selected subset of tensors of the heatmaps.