WO2022081745A1

WO2022081745A1 - Real-time rendering of 3d wearable articles on human bodies for camera-supported computing devices

Info

Publication number: WO2022081745A1
Application number: PCT/US2021/054831
Authority: WO
Inventors: Brian POLLER; Daniel SPECIALE; Paxshal Mehta
Original assignee: Maze Ar Llc
Priority date: 2020-10-13
Filing date: 2021-10-13
Publication date: 2022-04-21

Abstract

A system and method for providing real-time rendering of three-dimensional wearable articles on human bodies using camera-supported computing devices. In some instances, the system may determine an object model describing an object, which object model may include dimensions and/or visual attributes of the object. The system may determine a user model including a skeletal model with articulation points and outer dimensions of the user. In some implementations, the system may overlay the object model over the user model, for example, including providing a graphical user interface in which the object model is overlaid over an image based on the user model and the articulation points. In some instances, the user model may not be visibly rendered over an image of the user in the graphical user interface, so that the object model is displayed directly on the image of the user.

Description

REAL-TIME RENDERING OF 3D WEARABLE ARTICLES ON HUMAN BODIES FOR CAMERA-SUPPORTED COMPUTING DEVICES

5 BACKGROUND

[0001] The present disclosure relates to technology for, among other things, providing an intelligent system for rendering wearable articles on human bodies. Some implementations relate to real-time 3D rendering of clothing on human bodies in augmented reality for camera- supported computing devices.

10 [0002] Conventionally, clothing manufacturers and/or retailers have sold clothes in physical stores where they maintain an inventory of different sizes at the store and staff the store with trained retail professionals, for example, to deal with inventory and help consumers determine sizes. To accommodate people’s various shapes and sizes, clothing retailers may sell clothes in a variety of sizes and many physical stores also include changing rooms, which allow 15 a consumer to try on an article of clothing before purchase, so the consumer can decide whether a particular article of clothing is suitable, for example, based on consideration of the size, fit, style, how the clothing interacts with the consumer’s body, and other factors. Accordingly, trying on clothing before purchasing increases consumer satisfaction.

[0003] Additionally, people wear clothing or other objects for fashion, comfort,

20 protection, identification, and many other purposes, so they may carefully plan clothing for each day even to the extend of trying on several articles of clothing before selecting one for a given occasion.

[0004] Purchasing articles of clothing remotely has become increasingly popular in recent years. With the advent of the Internet, many manufacturers and/or retailers facilitate easy 25 and convenient purchasing of clothing through a website or application. While online purchasing of clothing has increased the convenience of the consumer, the consumer cannot try on an article of clothing before purchase, which may lead to dissatisfaction regarding the size, style, fit, movement, or another element of the article of clothing. This lack of information about the product results in a significantly increased return rate. For example, consumers, knowing the 30 perils of online purchasing (e.g., poor size, undesirable fit or style), may purchase multiple copies of the same article of clothing in different sizes and/or colors with the express intent to return one or more of the copies, thereby causing numerous inventory, shipping, and other challenges for retailers. [0005] Some solutions use special hardware devices to scan a consumer’s body and generate a 3D avatar with photos of articles on the avatar. These methods are not interactive and offer a less realistic user experience. They may also require specialized and time and resourceconsuming programming, which reduce their adoption and increase computational resource utilization. The devices do not work well since they hinder consumer product interaction and lack user friendliness.

[0006] Accordingly, there is a need for an efficient and user-friendly mechanism for users to better determine the size, fit, color, or other features of an article of clothing.

SUMMARY

[0007] According to one innovative aspect of the subject matter described in this disclosure, a system may include: a computer-implemented method including: determining an object model describing an object, the object model including one or more dimensions and one or more visual attributes of the object; determining a user model including a skeletal model and outer dimensions describing a user, the skeletal model including two or more user articulation points of the user; overlaying the object model over the user model using the two or more user articulation points; and providing, for display on a client computing device, a graphical interface including the object model overlaid over the user model.

[0008] Implementations may include one or more of the following features. The computer-implemented method further including: that determining the object model includes receiving a two-dimensional image showing the object, dividing the two-dimensional image of the object into two or more image portions, and generating a three-dimensional model, wherein generating includes piecing together the two or more image portions; that determining the object model includes determining an object articulation point of the object; that overlaying the object model over the user model includes determining an articulation of the user associated with a user articulation point of the two or more user articulation points, associating the object articulation point with the user articulation point, and articulating the object model based on the articulation of the user, the user articulation point, and the object articulation point; and that determining the object model includes receiving one or more dimensions of the object, and scaling the object model using the one or more dimensions.

[0009] Implementations may include one or more of the following features. The computer-implemented method further including: that determining the user model includes receiving an image depicting the user, identifying positions of the two or more user articulation points in the image, and generating the skeletal model using the positions of the two or more user articulation points in the image; that determining the user model includes detecting an outline around the skeletal model of the user in the image, determining the outer dimensions of the user using the outline around the skeletal model of the user in the image, and generating the user model including a volumetric three-dimensional model based on the outer dimensions of the user; and that determining the user model includes receiving an image depicting the user, determining a reference distance in the image, the reference distance including a length of an object visible in the image, and scaling the user model using the reference distance.

[0010] Implementations may further include the following features. The computer- implemented method where further including: receiving a file including a renderable three- dimensional object model describing the object; transmitting the file including the renderable three-dimensional object model for execution on the client computing device; receiving a video data stream including a video of the user; detecting a change in position of the user in the video in real time; overlaying the object over the user model in the video based on the change in position of the user; that overlaying the object over the user model in the video based on the change in position of the user includes detecting an articulation of an elbow of the user, and articulating the object model at the elbow of the user; determining a second object model of the object, the object model and the second object model having different dimensions associated with different object sizes; determining a user dimension based on the user model; automatically determining one of the object model and the second object model based on the user dimension and the one or more dimensions of the object model and the second object model; and displaying, for display on the client computing device, a recommended size associated with the determined one of the object model and the second object model.

[0011] Further details for executing the system, methods, operations, and features described herein may include means, such as processors, non-transitory computer readable memory storing instructions that cause the system to execute the operations.

[0012] The features and advantages described herein are not all-inclusive and many additional features and advantages will be apparent to one or ordinary skill in the art in view of the figures and description. Moreover, it should be noted that the language used in the specification has been selected for readability and instructional purposes and not to limit the scope of the inventive subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The disclosure is illustrated by way of example, and not by way of limitation in the figures of the accompanying drawings in which like reference numerals are used to refer to similar elements.

[0014] Figure 1 is a flowchart of an example method for intelligent rendering of wearable articles on a human body.

[0015] Figure 2 is a flowchart of an example method for generating an object model.

[0016] Figure 3 is a flowchart of an example method for generating a user model.

[0017] Figure 4 is a flowchart of an example method of overlaying the object model on an image using the user model.

[0018] Figure 5 is a diagram of example objects and their dimensions.

[0019] Figures 6A and 6B are diagrams illustrating example articulation points on a human form.

[0020] Figure 6C is an example graphical user interface showing a skeletal model and outer dimensions of the user.

[0021] Figure 7 is a signal diagram illustrating example communication between software or hardware systems.

[0022] Figure 8 is a block diagram of an example system for real-time rendering of wearable articles on human bodies using camera-supported computing devices.

[0023] Figure 9 is a block diagram of an example computing system.

DETAILED DESCRIPTION

[0024] The technology disclosed in this application allows computing devices to efficiently determine object and/or user models and render articles or objects, such as clothing, on image or video using these models. For instance, the technology may render clothing in real time over a video of a user, for example, in augmented reality.

[0025] The technology described herein includes methodologies for remote visualization of how clothing, apparel, or other elements would look on a consumer. For instance, a consumer may select an article of clothing to “try on” using augmented reality. The technology may use body mapping logic to determine a skeletal model and add muscle, skin, and other body dimensions to generate a three-dimensional model of the consumer. Using measurements and fabric information received from a manufacturer or retailer of an article of clothing, the technology may generate a three-dimensional model of the article of clothing. The technology may then overlay the article of clothing on the user, so the user may view a realistic rendering of himself or herself wearing the article of clothing.

[0026] The model of the user may be created based on size information of the user. For example, the model may include skeletal or articulation information, which may be scaled to the user’s dimensions. Using references of known distances, the technology may determine various dimensions of the user. The model of the article of clothing may include size data for the article of clothing. The clothing model may be overlaid on the user model, so the user may determine how the size of the article of clothing fits on him or her.

[0027] The overlaid clothing model may move in real time with the user and articulating at different articulation locations. As the user rotates his or her body, articulates a joint, or otherwise moves, the technology may move a clothing model with the user, for example, based on changes to the user model. In some implementations, the clothing model may include information regarding material properties, such as the stretch of the fabric, the stiffness of the fabric, the drape of the fabric, and so forth, which may further help improve the simulated fit of the article of clothing.

[0028] In some implementations, the technology may use information (e.g., provided by a manufacturer and/or retailer) to generate a three-dimensional model of an article of clothing. The model may include articulation of the article of clothing at articulation points. In some implementations, the model may use size data and/or fabric data to model fabric stretch, stiffness, drape, and so forth.

[0029] The technology may receive and/or generate skeletal information about the user and, in some instances, generate one or more index or articulation points, which may identify articulation points of the user, such as shoulders, elbow, hips, wrists, and so forth. Using the image data and a known size reference (such as the user’s height, the user’s pupils, environmental references, and so forth), the technology may determine size data for the user, such as shoulder width, arm length, and so forth. The technology may also use image analysis to identify the overlying musculature and surface features of the user, thereby filling in the outer dimensions of the user.

[0030] The technology may overlay the article or clothing model on the user model. For example, the technology may dynamically overlay a three-dimensional clothing model, which may allow the user to move around and visualize how the article of clothing may move with him or her, thereby improving the user’s understanding of how the article of clothing looks or fits when worn. [0031] Some implementations of the technology may include a platform for determining and/or implementing model overlays that allows the processes to easily scaled. For instance, the platform (e.g., operable on a server) may provide portals and/or interfaces for receiving sizing or other information about articles of clothing from a manufacturer or retailer, which may, in some implementations, include three-dimensional models, renderable models, measurements, images, etc., as described below. The platform may also interact with users, for example, using a website, ifirame, plugin, mobile application, or other interface to receive physical information, such as user dimensions, body images or video, or other data to develop the user model. The platform may also use the article/object model to overlay an article of clothing or other object on the user’s client computing device, for example, to allow remote visualization of how clothing or other articles would look on a user while reducing computational resources used on the client device 750.

[0032] The technology may allow a real-time virtual-try on directly involving an image or video of the user and allow an object model to move in real time with a user, for example, by moving and/or articulating the object (e.g., clothing) with the user. For instance, as the user rotates his or her body, articulates a joint, or otherwise moves, the overlaid object/clothing model may move with the user thereby allowing the user to determine how the article of clothing will move with him or her. In some implementations, the clothing model may include information regarding material properties, such as the stretch of the fabric, the stiffness of the fabric, the drape of the fabric, and so forth, which may further help improve the simulated fit of the article of clothing or other object.

[0033] The features and advantages described herein are not all-inclusive and many additional features and advantages are within the scope of the present disclosure. Moreover, it should be noted that the language used in the present disclosure has been principally selected for readability and instructional purposes, and not to limit the scope of the subject matter disclosed herein.

[0034] With reference to the figures, reference numbers may be used to refer to components found in any of the figures, regardless whether those reference numbers are shown in the figure being described. Further, where a reference number includes a letter referring to one of multiple similar components (e.g., component 000a, 000b, and OOOn), the reference number may be used without the letter to refer to one or all of the similar components.

[0035] Figure 1 is a flowchart of an example method for intelligent rendering of wearable articles on a human body. For instance, the method of Figure 1 describes a process for generating realistic augmented reality clothing by, for example, determining a three-dimensional model of a user, determining a three-dimensional model of an object or article of clothing, and overlaying the object model over the user model. In some instances, the user model may be matched to an image or video of a user but not visible (e.g., not displayed or rendered by the model AR engine 808) in the image, so that when the object model is overlaid over the user model, it appears as though the object (e.g., the article of clothing) is being worn by the user in the image or video, for example, in augmented reality.

[0036] The operations of the method in Figure 1 may be exchanged, reordered, omitted, or changed without departing from the scope of this disclosure. For example, the other figures and methods described herein may further inform the operations of Figure 1. Furthermore, although Figure 1 is described in reference to the model augmented reality (AR) engine 808, it may be performed by separate devices, components of the model AR engine 808, or otherwise without departing from the scope of this disclosure.

[0037] At 102, the model AR engine 808 may determine one or more object model(s) describing a set of objects. Additional details for determining an object model are described elsewhere herein, for example, in reference to Figure 2. Each object model may include dimensions and/or visual attributes of an object. For instance, the object model and/or a header of the object model file may include information describing a width, height, scale, clothing size, thickness, bust size, chest circumference, sleeve length, inseam, waist circumference, neck circumference, shoulder width, length, or other dimensions of the object, as described below. It should be noted that, because the way an object fits may be affected by many different dimensions, the model and/or dimension data may include shape, outline, or other data that may be scaled using one or more dimension, as described in further detail below. In some implementations, the object model may additionally or alternatively include clothing sizing information, such as small, medium, large, 1, 2, 9, 10, 44, 32x34, slim fit, loose fit, etc.

[0038] Additionally, in some instances, the object model may include data describing one or more visual attributes of an object. For instance, a visual attribute may include an image (e.g., from one or multiple angles, of an entire object or its components), pattern, two or three- dimensional shape, texture, color, shade, sheen, or other visual aspect of the object.

Additionally, the object model or describe may include various attributes that affect how a model is rendered, how the model moves, or other aspects for generating and/or using the object model, such as fabric features, stretchability of the object or its parts and/or directions, foldability, folds, seams, flexibility, drape, texture, weight, transparentness, clinginess, stiffness, or other attributes of the object and/or its material. [0039] In some implementations, the object model may include articulation, anchor, or index points (also referred to herein as object articulation points), which indicate certain features of the object model. In some implementations, an articulation point may be a point at which an object bends, stretches, or hangs. In some implementations, the articulation point may indicate where to match or overlay object and user models, where the object or user models stretch, or other points. In some instances, an object articulation point may be used by the model AR engine 808 to match the object model with a user model (e.g., with user model articulation points), so that the model AR engine 808 can determine over which points corresponding points on an object model should be overlaid.

[0040] Depending on the implementation, the articulation or index points of the object model may be automatically determined by the model AR engine 808 or may be manually input by a user, such as an administrative user at a retailer or manufacturer. For example, the model AR engine 808 may automatically identify and/or assign, based on a photograph of pants, articulation/index points where the pants would typically bend, stretch, or move. Additionally or alternatively, the object articulation points may indicate points on the object model that match up with and/or are overlaid on user articulation points (e.g., a shirt sleeve may articulate at any point, such as where a user’s elbow is in the shirt sleeve). In some instances, the model AR engine 808 may receive several images or a video of an object and use the articulation of the object in the images/video to determine the articulation points. In some implementations, a user may use a graphical user interface provided by or in association with the model AR engine 808 to manually select articulation and/or index points.

[0041] Depending on the implementation, the model AR engine 808 may generate a three-dimensional and/or renderable model of an object based on the various information about an object, as described in further detail elsewhere herein. For instance, the model AR engine 808 may fuse dimensional and/or visual information, etc., to generate a three-dimensional model. In some instances, the model AR engine 808 may refine a model based on additional information, such as fabric attributes, user feedback, or other data.

[0042] The object model, dimensional data, visual attributes, or other data about an object may be received from an administrative user or third-party computing system, for example, of a manufacturer. For example, the model AR engine 808 may receive pictures (e.g., from a front, back, side, top, etc.), sizing information, fabric data, articulation points, size chart data, stretchable regions, CAD (computer aided design) files, reference dimensions, and other information from a computing device or server of a manufacturer or retailer. In some implementations, the model AR engine 808 may communicate with various external systems to send and/or receive information via embedded APIs (application programming interfaces), a user portal, plugins, etc., as described in further detail below.

[0043] In some implementations, the model AR engine 808 may receive feedback from client devices (e.g., 750) and/or users, which feedback it may use to adapt an object model. For example, the model AR engine 808 may crowdsource feedback by displaying interactable graphical elements on a graphical interface, which allow users to resize, move, stretch, or otherwise provide input relating to the item. For example, if a threshold percentage or quantity of users provide input that an object has a stretchy waistband, that sizing is incorrect, or another object-attribute modifying input, the model AR engine 808 may automatically adjust the relevant parameter(s) of the object model.

[0044] At 104, the model AR engine 808 may determine a user model, for example, using or including a skeletal model and/or outer dimensions describing a user. Additional details for determining a user model are described elsewhere herein, for example, in reference to Figure 3. It should be noted that while the implementations herein are generally described in reference to articles of clothing, they may be applied to other objects, such as any garment (e.g., apron, jacket, ties, etc.), cosmetic (e.g., makeup, eye laches, false nails, etc.), accessory (e.g., jewelry, bag, hat, etc.), tattoo, ornamentation, or other object that could be visualized in association with a user. Various object models representing various object colors, sizes, or types may be accumulated by the model AR engine 808 into a digital catalog of object models.

[0045] The skeletal model may include two or more articulation or index points (referred to herein as user model articulation points or user articulation points) of the user. User articulation points may be points at which the skeletal model articulates or bends, such as a human joint (e.g., elbow, knee, hip, etc.). In some implementations, the user articulation points may also or alternatively serve as index points that may be used to match up with object articulation point(s) and/or used to scale the model, as described in further detail below. In some implementations, the determining the model may include receiving an image or video including or depicting the user and identifying positions of articulation points in and/or using the image. [0046] In some implementations, the model AR engine 808 may use computer learning, user instructions, computer vision, and/or other processes for generating a wire-skeletal model, as described in further detail in reference to Figure 3. For instance, the model AR engine 808 may capture a photograph/video of the user in a certain area of a camera/viewfmder, in certain positions, in certain movements, etc., which, in some implementations, the model AR engine 808 may match up to expected positions of index points or articulation points, thereby determining relative lengths and/or positions or portions of a user’s body (e.g., locations between articulation or index points). In some implementations, the model AR engine 808 may implement open- source or other software, such as PoseNet ™, which may be programmed to generate wire skeletal models, detect certain user features (e.g., joints, shoulders, elbows, necks, eyes, jaws, waists, hips, knees, etc.), or perform other operations. Accordingly, based on measurements, angles, pixels separating points, or other features, the model AR engine 808 may develop size data in a skeletal model of the user.

[0047] In some implementations, the model AR engine 808 may receive an image or scan of a body of a user from an image, scanner, or sensor of a client device 750 (e.g., a camera, lidar sensor, time-of-flight sensor, dual camera sensor, or other sensor of a smartphone or other device). In some implementations, the model AR engine 808 may also receive other input from a user, such as their normal clothing sizing, a waist measurement, weight, shoulder measurement, or height, etc. The additional inputs may help generate the skeletal model or flesh out outer dimensions. For instance, a user may input a shoulder width, upper arm length, and forearm length, which the model AR engine 808 may use to scale, correct, augment, or generate a skeletal model where junctions between each of these measurements may serve as articulation points (e.g., matching human joints, although other implementations are possible and contemplated herein).

[0048] In some implementations, the model AR engine 808 may use a reference dimension of an object in an image (e.g., a visible object of known length) or user input (e.g., a height, shoulder width, forearm length, etc.) to scale the skeletal model to a correct size. For example, while a skeletal model determined solely based on an image or video of a user may determine relative sizes of distances between articulation or index points, the model AR engine 808 may scale the distances appropriately based on a relative size of one of these distances with a reference measurement, for example, if a user is holding a credit card in her hand, the model AR engine 808 may use a known size of a credit card to determine a length of the user’s forearm and other aspects of her body.

[0049] In some implementations, the model AR engine 808 may use image analysis to identify an outline or outer dimensions of a user, for example, due to overlying musculature and surface features of the user. For instance, the model AR engine 808 may flesh out the wireskeleton model to have a volume that matches the actual volumes of a user. In some implementations, the model AR engine 808 may use user input to determine outer dimensions (e.g., waist size). In some implementations, the model AR engine 808 may infer the user’s outer surface using the skeletal model and edge-finding models. For example, the model AR engine 808 may determine an edge surrounding the skeletal model using a Kalman filter or edge-finding algorithm to detect an edge around a skeletal model. The model AR engine 808 may determine an angle or quantity of pixels between edges (e.g., from edge to edge; from a wire, from articulation point to an edge) to compute the outer dimensions of the user.

[0050] In some implementations, the model AR engine 808 may determine the outer dimensions of the user from various angles or positions. For instance, the model AR engine 808 may receive multiple images of a user from different angles or poses, or the model AR engine 808 may instruct a user in a video to turn, move within a frame of the camera, hold in various poses, stand in front of a white background, or perform other action, in order to improve determination of outer edges and volume of various body parts or angles. In some implementations, the model AR engine 808 may provide a graphical user interface in which a user may drag edges of the user model to actual edges of the user in the image in order to improve accuracy of model generation and/or size matching.

[0051] In some implementations, a user model may be previously generated, for example, based on a mannequin or similar pre-established form. The model AR engine 808 may then modify the length of features, such as shoulder width, torso length, arm length, etc., based on the structural model detected from the user. The model AR engine 808 may also increase the volume of the pre-established form based on the outer dimensions detected for the user.

Accordingly, the user model may be adapted from a previous three-dimensional, renderable model.

[0052] The model AR engine 808 may generate the user model in real time, for example, frame by frame from a video of the user, to accommodate the user’s movements and allow dynamic interaction with the object model (e.g., an article of clothing). The user model may be generated from live images/video from the camera of a user’s client device 750. In some implementations, the user may not be displayed or rendered over the image but may remain at the appropriate position matching the user in the image, so that when the object model is overlaid on the user model, it is also overlaid over the user’s actual position in the live image/video. Accordingly, in some implementations, the model AR engine 808 may allow an augmented reality display rather than simply displaying the object over an avatar.

[0053] In some instances, in order to reduce computational resources used, the model AR engine 808 may generate a user model for visible parts of a user. For instance, if only a t-shirt is being overlaid, the model AR engine 808 may only determine or generate a user model that would be under or interact with the t-shirt (e.g., from a waist up, from a length at the end/bottom edge of the t-shirt upwards, etc.). Accordingly, unnecessary parts of a user may not be modeled or interact with an object model in order to reduce processing time, latency, or bandwidth utilization.

[0054] At 106, the model AR engine 808 may determine a selected object from among a set of objects. For instance, the model AR engine 808 may provide a graphical user interface to the user where multiple objects (e.g., graphical images) and/or representations of the object models are shown (e.g., in a grid, scrollable list, etc.). The user may select an object from among the list to render or overlay on the image of the user (e.g., over a user model). In some instances, the model AR engine 808 may detect visual or audio cues to switch between object models. For instance, a user (e.g., holding a smartphone on which an object model is rendered in front of a mirror) standing in a field of view of a camera may speak, clap, snap, or make a gesture (e.g., holding a hand up), which may be detected by the model AR engine 808 and cause it to switch object models thereby allowing the user to virtually try on several items.

[0055] In some implementations, the model AR engine 808 may determine a second object model with one or more different dimensions than the first model. For instance, the second object may be a different article of clothing or a different size of the same article of clothing. The model AR engine 808 may determine, based on the user model (and/or a received or determined dimension) a dimension of the user and then automatically determine which object model matches the user better (e.g., closer, larger than, within a threshold range, etc.). Using this matching, the model AR engine 808 may automatically display the recommended object model and/or recommend a certain size or object.

[0056] At 108, the model AR engine 808 may overlay one or more object models of the selected object(s) over a user model. For instance, the model AR engine 808 may overlay an object model over a user on an image using the user model to determine a position, sizing, orientation, movement, articulation, or other interaction of the object model with the user. Additional details for determining a user model are described elsewhere herein, for example, in reference to Figure 4. Accordingly, the object model may be dynamically overlaid over the user model, which may allow the user to move around and visualize how the article of clothing may move with him or her. In this manner, the user may decide how the article of clothing looks or fits when worn.

[0057] In some implementations, the model AR engine 808 may overlay a three- dimensional clothing/object model over a three-dimensional user model in an augmented reality view. The clothing model may be overlaid on the user in real time, the user’s position may be tracked, reflected in the user model, and the overlay may be updated to follow the user model as the user moves, rotates, articulates joints, and so forth. [0058] In some implementations, when overlaying the object model, the model AR engine 808 may determine object articulation points of the object and, using the user model, determine articulation of the user. As a user articulates his or her limbs, the model AR engine 808 may update the articulation of the user model and match the articulation of the object model based on the articulation of the user model. For example, the model AR engine 808 may match object articulation points with user articulation points, so that the object model moves, articulates, rotates, stretches, etc., some or all of the articulation or index points of the object model may remain matched up with those of the user model. For example, the object articulation points of a shirt may be labeled as shoulder points and may remain over user articulation points labeled as shoulder points of the user. Some articulation points, such as elbow, arm, waist, etc., of the shirt may be defined with some stretch or leeway to allow the articulation points to depart from those on the user model. Accordingly, an object model of a shirt may move naturally as the user moves. The flexibility of the articulation points, drape of the cloth, stiffness of the cloth, sliding/friction of the cloth across a user model, etc., may be modified with different values to tune the realism of the rendered object.

[0059] In some implementations, multiple object models may be overlaid on the user model simultaneously. For instance, the model AR engine 808 may overlay a shirt and pants simultaneously using the same or similar operations as those for a single object model. For instance, the model AR engine 808 may automatically overlay a shirt over pants based on a user setting (e.g., indicating whether or not to tuck in a shirt), user model dimensions, and or object model dimensions.

[0060] In some implementations, a camera of the user may remain in a single location, may move around a room, may face a mirror, or may experience other changes. Accordingly, the user model may be updated in real time to allow accurate tracking of the user’s position, pose, and orientation.

[0061] At 110, the model AR engine 808 may provide the overlaid selected object model over a user and/or user model for display on a graphical user interface on a client computing device.

[0062] In some implementations, the model AR engine 808 may transmit a three- dimensional object model and a three-dimensional user model to a client device 750, which may overlay the clothing model on the user. In some instances, a retailer or manufacturer may use an ifirame on their website or application, which provides access to the functionality of the model AR engine 808, thereby powering try-on solutions on the retailer’s or manufacturer’s own portal while allowing communication with a server on which the model AR engine 808 or a portion thereof is operable.

[0063] In some implementations, the model AR engine 808 may receive a file including a renderable three-dimensional object model from computer-accessible storage and/or another computing device or server, which may be managed by a third-party, such as a manufacturer or retailer (although, it should be noted, that the model AR engine 808 or some of its functionality may be operated on a computing system of a manufacturer or retailer). The model AR engine 808 may transmit a file including the renderable three-dimensional object for execution on the client computing device, for example, so that the client computing device overlays the object model over an image of the user.

[0064] Figure 2 is a flowchart of an example method for generating an object model. Aspects of the method described in reference to Figure 2 may be modified or exchanged with those described herein, such as those described in reference to Figure 1. For instance, the method of Figure 2 may provide additional details to the operations of Figure 1. Operations of Figure 2 may be performed by the object model module 922 of the model AR engine 808, although other components of the model AR engine 808 may be used or otherwise organized without departing from the scope of this disclosure. Further, as noted above, while the operations are described in reference to the example of articles of clothing, they may be applied to other garments, cosmetics, accessories, tattoos, ornamentations, or objects.

[0065] At 202, the object model module 922 may receive object data including size information, image(s), computer-aided design file(s), attribute data, and/or identification information. For example, the object model module 922 may receive a file including a renderable, three-dimensional object file, object dimensions, photographs, visual attributes, material attributes or other data for generating an object model, as described in reference to 102 above.

[0066] The object model module 922 may collect manufacturing measurements and/or size chart data, dimension information, photos in front, back, and side, or other information. In some instances, the object model may receive, define, or allow definition of motion models for various materials, which may be automatically applied in object models of objects identified (e.g., when data is received describing the object) as having the same material. For instance, a certain material may have a physical spring model that makes the material move or drape like silk instead of vinyl.

[0067] In some implementations, the object model module 922 may receive scan data from a sensor device of a computing device, such as a smartphone, three-dimensional scanner, etc. For instance, the object model module 922 may utilize commercially available scanning technologies, such as mobile phone LIDAR scanners, to collect sizing and/or fabric information. Clothing may be scanned by the manufacturer, by personnel, consumers, any other person, or combinations thereof.

[0068] In some implementations, the object model module 922 may receive blueprints, clothing patterns, or fabric patterns (e.g., sizing, stitching, and shape details) for parts of an article of clothing, or other details for an article of clothing, which the object model module 922 may combine together to generate an object model.

[0069] In some implementations, the object model module 922 may receive a glTF (GL transmission format) formatted file (although other formats are possible), which encapsulates information for an object and/or which file may be generated using the received information. The object model module 922 may add header information providing additional details associated with an object such as clothing sizing, name or identification information, fabric or material details, or other information.

[0070] For example, the object model module 922 may use an embedded API to communicate with backend servers, an interface, or a plugin via which people or computing systems may provide information describing objects. For example, to create a representative three-dimensional clothing model of an article of clothing, the object model module 922 may receive size information and pictures of the article of clothing from the manufacturer and/or the retailer of an article of clothing. The model AR engine 808 may provide a portal and uploading process where data can be uploaded and automatically converted into a glTF file with information that may be processed and then rendered by the model AR engine 808.

[0071] In some implementations, the object model module 922 may receive two- dimensional or three-dimensional files showing all or portions of objects. For instance, the object model module 922 may receive files from various third-party sources, such as Browzwear™, Amerra™, or other sources providing dimensions, visual attributes, and/or allow articles of clothing to be modeled in two or three dimensions.

[0072] In some implementations, the object model module 922 may receive detailed computer files of the article of clothing, such as detailed three-dimensional renderings, manufacturing drawings, computerized cutting machine files, computerized sewing machine files, any other computer file, and combinations thereof. It should be noted that different companies may provide different levels of detail and the object model module 922 may adapt the model and/or quality thereof based on the received level of detail. For example, a nonmanufacturing retailer may provide generalized sizing information, such as from a size chart, and a picture. In some examples, a manufacturer may have more information available, and may therefore provide more detailed information.

[0073] Figure 5 is a diagram 500 of two objects including example measurements for each object, for example, the diagram 500 shows pants 502 and a shirt 504. The object model module 922 may use images showing a shape, such as those shown in the diagram 500, to determine a two-dimensional shape of objects and infer the three-dimensional shape of objects, for example, based on whether the object is being worn or lying flat on a surface.

[0074] The object model module 922 may receive, from a user input, object-attribute database, or third-party computing system, one or more absolute or relative dimensions. For instance, the object model module 922 may detect, based on an image of the object, relative dimensions of the torso length, chest width, sleeve length, waist circumference, leg inseam, or other dimensions, for example, based on angles or number of pixels in the image. In some instances, the dimension data may include the dimension in inches, millimeters, etc.

[0075] Returning to Figure 2, at 204, the object model module 922 may stitch portion(s) of images or other visual aspects of an object together into a three-dimensional object model of the object. For example, where images of the object are received by the model AR engine 808, the object model module 922 may use portions of the image to create the shape and/or pattern of the object model.

[0076] In some implementations, the object model module 922 may receive one or more two-dimensional images showing the object from one or more angles. The object model module 922 may divide the two-dimensional image of the object into multiple image portions to determine individual two-dimensional areas of the object. For instance, the object model module 922 may use edge detection (e.g., as described above in reference to the user model) to detect edges of the object, which may be used to cut a front, back, side, sleeves, pant legs, etc., into individual areas.

[0077] The object model module 922 may then piece or stitch together the individual portions into a three-dimensional model of the object. For instance, the object model module 922 may apply a front of a shirt to the front of a three-dimensional model of a shirt (e.g., a generic shirt model, a model generated based on the provided or detected dimensions of the shirt, etc.). Accordingly, the patterns and textures of the image may be applied to a three-dimensional structure to create a more realistic model. For example, the three-dimensional structure of the object may be based on a known or determined structure of a similar object (e.g., a generic shirt or pants model), a received or detected shape (e.g., from edge detection of the object in the image), and/or dimensions of the object. In some implementations, the object model module 922 may call other software or logic processes to perform these operations. For instance, the operation may use VStitcher™ or another three-dimensional modeling software to split up objects and stitch them together into a three-dimensional model, which model may be input into the object model module 922 or the object model module 922 may directly interface with the software.

[0078] At 206, the object model module 922 may determine one or more object articulation or movement points and/or fabric properties of object.

[0079] In some implementations, the object model module 922, in order to generate a realistic fit, update the three-dimensional object model to incorporate one or more articulation points representing intersections and other points around which there will likely be movement and intersection of different body parts. For example, the object model module 922 may incorporate articulation points at major movement points, such as the elbows, shoulders, knees, and so forth.

[0080] In some implementations, object articulation points may incorporate fabric information about the fabric, which may allow realistic interactions of the fabric with the user, including stretch at or within the vicinity of the articulation points, stretch at connectors, such as zippers or buttons, drape, bunching, and so forth. Accordingly, when overlaid over the user model, the three-dimensional object model may move with the user, thereby providing a realistic representation of how the article of clothing would fit and look on the user.

[0081] In some implementations, the articulation points of the object model may include other parameters, such as labels, whether or not they are allowed to move away from user articulation points, their flexibility, or other details. For instance, the model AR engine 808 may shift or stretch an articulation point when the object model is overlaid on a user model based on the size of a user model. For example, if a user model has arms that are longer than the sleeves of an object, the object may articulate at the point of the user’s elbows (e.g., based on the length of the user’s arms or position of their articulation points).

[0082] In some implementations, the object model module 922 may incorporate material details into the object model. Material details may include any type of detail about the material, such as fabric type, weave type, elasticity, stiffness, other details, and/or combinations thereof. Such material details may allow object model module 922 to generate more realistic object models and/or more realistic interactions with user models. For example, an object model that includes material details may be able to simulate how the clothing may drape, stretch, and otherwise move with the user. In some instances, the material details may include a stretch fit, mass spring motion, or other details. [0083] At 208, the object model module 922 may receive or determine dimensions of object and dimensionalize the object model (e.g., by scaling the object model in one or more directions). The dimension data may be received as discussed at 202 or a reference dimension for the object may be determined, such as a width, height, circumference, etc., which may be used to scale the object model to have the correct virtual dimensions. For example, the object model module 922 may receive size information from a clothing manufacturer and/or retailer, such as clothing sizing and/or detailed dimensions of the article of clothing (e.g., dimensions of bust, shoulder, waist, sleave, inseam, and so forth). For example, Figure 5 illustrates sample measurements, as discussed above.

[0084] At 210, the object model module 922 may aggregate additional information describing object attributes from one or more third-party sources. The model AR engine 808 may receive sizing and/or material information from outside sources via a website, mobile application, or other input method. For example, the model AR engine 808 may display input fields when a user selects a model for overlay or as feedback inputs when the model AR engine 808 overlays the object model over the user. The user inputs may allow users to modify or add sizing information, such as adjusting height, width, or circumference of objects, adjust drape or flexibility of the materials, move articulation points, or provide other inputs.

[0085] At 212, the object model module 922 may refine the object model using the additional information to improve model accuracy, for example, if the model AR engine 808 determines that a threshold number or percentage of users provide a certain input, the object model module 922 may duplicate and/or refine the object model based on the feedback. For instance, using the inputs noted at 210, the object model module 922 may adjust dimensions, colors, material details, or other attributes of the object model.

[0086] At 214, the object model module 922 may associate identification information with the object model and, in some instances, publish object model to web service with one or more object attributes. For instance, the object model module 922 may generate files, such as glTF files and header information indicating sizing, naming, brand, or other metadata describing the object or object model. The model AR engine 808 may provide a catalog of objects on a website or mobile application from which a user may select an object. Upon selection of the object, the model AR engine 808 may retrieve an object model file and/or transmit it to a client computing device, website, or other end device for rendering and/or overlaying over the user/user model. For example, the object model module 922 may provide code for execution by a user device in association with a third-party request for a page depicting the article of clothing, for example, via an iframe or another interface, as described in further detail elsewhere herein. In some implementations, the model AR engine 808 may include or be embedded as code in a third-party website or application.

[0087] Figure 3 is a flowchart of an example method for generating a user model. Aspects of the method described in reference to Figure 3 may be modified or exchanged with those described herein, such as those described in reference to Figure 1. For instance, the method of Figure 3 may provide additional details to the operations of Figure 1. Operations of Figure 3 may be performed by the user model module 924 of the model AR engine 808, although other components of the model AR engine 808 may be used or otherwise organized without departing from the scope of this disclosure.

[0088] Depending on the implementation, the user model module 924 may generate a three-dimensional user model in real time, which accommodates movements by the user. In some implementations, the user model module 924 may generate the user model using live images from a camera on the client device 750 of the user rather than, for instance, importing the model from another location and/or using an avatar. The user model module 924 may generate a live three-dimensional user model that may represent points for joint articulations and also the outer dimensions of the user. Furthermore, the user model may be prepared in real time using a digital camera, for example, by utilizing augmented reality technology to identify the skeletal structure and outer dimensions of the user to generate the user model.

[0089] At 302, the model AR engine 808 may receive user input defining user attribute(s) describing a user’s physical feature(s). For example, the user may provide input describing their waist size, height, inseam, weight, or other dimensions, as described above. [0090] At 304, the model AR engine 808 may receive an image (whether still or video) or scan data of user. For instance, as described above, the user model module 924 may receive a digital image or video from a camera on a user’s client computing device, such as a smartphone, laptop, or other device. In some implementations, the user model module 924 may analyze a video frame by frame or at another rate.

[0091] At 306, the model AR engine 808 may generate a skeletal structure model identifying points of articulation, for example, as pixel values or ranges. The skeletal data may be received from a third-party model or software, generated from user-provided dimensions, and/or determined from the image. For example, the user model module 924 may identify positions of two or more user articulation points in the image and generate the skeletal model using the positions of the two or more user articulation points in the image.

[0092] In some implementations, the user model module 924 may utilize machine learning, such as a trained classifier to identify articulation points from the image. For instance, the user model module 924 may be trained (e.g., using previously labeled machine learning training data sets) to identify points on a torso (e.g., at comers or points where the torso bends), multiple points on a head, points along arms, points on legs, etc. The skeletal model may be generated as a flat or planar model or with depth between articulation points. The user model module 924 may automatically label points, such as a left shoulder point, right shoulder point, right hip, left hip, right elbow, etc., for example, based on the locations at which they are detected.

[0093] In some implementations, the user model module 924 may use a third party, open-source application to generate the wire-skeletal structure of the user. For example, the user model module 924 may use the open-source software PoseNet™ to generate the user’s skeletal structure, although it should be understood that the user model module 924 may utilize any skeleton structure generating logic. For example, the user model module 924, potentially using third-party software or logic, may receive visual data from a digital camera and analyze the digital data to identify points of articulation, such as elbows, wrists, shoulders, neck, hips, knees, ankles, and so forth. The user model module 924 may then output the articulation points as pixel location values or ranges.

[0094] Figures 6A and 6B are diagrams 600a and 600b illustrating example articulation points 604 (note that only a single articulation point 604 is labeled) on a human form. Articulation points may be key points, anchor points, index points, or other points of reference on the user model and/or skeletal model, as discussed above. As illustrated in Figure 6A, the skeletal structure model may include articulation points, which may correspond to joints of the human. In some instances, the human may move and the user model module 924 may detect the movement of the articulation points, as illustrated in Figure 6B.

[0095] At 308, the user model module 924 may refine articulation point identification using movement or other data. In some implementations, the model AR engine 808 may instruct the user to move, for example, by articulating joints or changing a pose, which may aid the model user model module 924 in determining the articulation points or other details of the model. For example, as illustrated in Figure 6B, the diagram shows the human form articulated to a different position than in 6A and the articulation points have moved representing the change in angle, pose, or position. Accordingly, the output skeletal model may follow the movements of a user, which movements may help the user model module 924 identify articulation points.

[0096] At 310, the model AR engine 808 may receive one or more reference distances/dimensions and dimensionalize the skeletal structure model using the reference dimension(s). For instance, a reference dimension may be any measurement that may be imputed to the skeletal model, such as a distance between articulation points. In some implementations, the reference dimension may be a size of a known object in the image that may be at approximately the same depth in the image as the skeletal model, so that one or more dimensions of the skeletal model may be imputed from the reference dimension.

[0097] In some implementations, in order to dimensionalize the model, the user model module 924 identifies a reference distance within the field of view of the camera. The reference distance may be any known distance. Different reference distances may be used, including user height, known user dimensions (e.g., length from shoulder to elbow, length from elbow to wrist), or standard human dimensions. For example, the user model module 924 may use a standard human dimension that has little variability among different people, such as iris diameter, and which may be detected in the image of the user.

[0098] In some implementations, the user model module 924 may request user input regarding body dimensions, including height, weight, shoulder width, torso length, hip width, and other bodily measurements. In some implementations, user model module 924 may directly measure one or more parameters using the built-in scanners of a camera collecting the visual data, such as LIDAR or time-of-flight sensor (e.g., indicating depth information).

[0099] Non-user dimensions, such as a ruler or other identifiable reference point in the background, foreground, held by the user, or any other reference distance may be used. For example, the user model module 924 may detect a ruler, credit card, QR code, or phone. For example, if a user is holding their phone toward a mirror to view themselves, the user model module 924 may detect the phone and, using its known size (e.g., based on a type of device on which the software is being executed or based on the detected phone type), infer a reference dimension.

[0100] Depending on the implementation, the user model module 924 may utilize a single reference distance or multiple reference distances. Multiple reference distances may allow the user model module 924 to continually refine and update the distance measurements in the user model. In some implementations, the user model module 924 may utilize a combination of user reference distances and environmental reference distances (e.g., reference distances in the background and/or foreground), which may allow the user model module 924 to estimate the depth of the field of view and, thereby, generate a more robust user model.

[0101] In some implementations, the distances between articulation points of the skeletal model may be determined using computer learning, as described above. The user model module 924 may adjust the estimation of lengths or relative dimensions of the skeletal model. For example, the user model module 924 may use estimated distances (e.g., using LIDAR, time-of- flight, focal length, angles, pixel differences) in a computer-learning model to estimate where to locate points of a skeletal model. The user model module 924 may use a forward model to predict where the points are, and the user model module 924 may use the reference dimension(s) or measurement(s) to train the computer learning model thereby reducing differences between measurements and predictions over time.

[0102] At 312, the model AR engine 808 may identify outer edges of a user, for example, in the received image. For example, the user model module 924 may detect an outline around the skeletal model of the user in the image and determine outer dimensions using the outline. The term outer dimensions may be used herein used to express the detectable outer surface of the user. For example, the user model module 924 may use augmented reality, image processing, body scans, or other means to identify outer dimensions of the user. The user model module 924 may modify the skeletal model to incorporate the outer dimensions of the user to generate a volumetric user model. For example, because people’s bodies are highly varied, people having the same height may not have the same outer dimensions and/or surface area. The outer dimensions may be inclusive of the user’s muscle, fat, skin, hair, outer clothing, or any other outer surface.

[0103] In some implementations, the user model module 924 may utilize the interaction of the skeletal model with the user to identify the outer edges of the user. For example, a user may touch another portion of his or her body and the user model module 924 may identify such a touch and determine where in the two or three-dimensional space the touch occurred.

[0104] In some implementations, the user model module 924 may use edge-finding models to identify the outer surface of the user’s body. For example, an edge-finding model may identify differences between pixel values on the user and a background of the collected image. The difference between the identified edge and the wire-skeleton structure may be the outer surface of the user. The distance may be added to the three-dimensional user model of the user. As the user moves, the model AR engine 808 may continually analyze the outer dimensions of the user and determine a three-dimensional model that is representative of the user. In some implementations, the model AR engine 808 may direct the user to face forward and then turn to the side (or to be in a certain pose) to assist in more accurate user model generation.

[0105] Figure 6C illustrates an example graphical user interface 600c showing a skeletal model and outer dimensions of the user. For instance, the graphical user interface 600c may represent a digital viewfinder of a smartphone showing a real-time image of a user. The model AR engine 808 may overlay a graphical representation 622 of a skeletal model and/or articulation points. In some implementations, the user may interact with the graphical representation 622 to adjust the points, for example, to increase their accuracy, and/or they may be automatically detected, as discussed above. The articulation points may move with the user to remain over corresponding spots on the user as the user moves.

[0106] In some implementations, the graphical user interface 600c may determine and/or display an outline, edge, or outer surface of the user, for example, as illustrated at 624. As discussed above, the model AR engine 808 may determine a distance (e.g., in pixels, angle, etc.) from edge to edge (e.g., as in the distance illustrated at 626) and/or from skeletal model to edge (e.g., as in the distance illustrated at 628), which allows the model AR engine 808 to accurately determine a size and shape of the user, thereby allowing the model AR engine 808 to accurately measure the user. In some instances, as the user moves, the model AR engine 808 may use the different perspectives (e.g., based on the skeletal model) to detect shapes and/or dimensions to an outer surface/edge, thereby improving modeling from different perspectives.

[0107] In some implementations, where a certain perspective is not available or viewed by the model AR engine 808, the model AR engine 808 may infer dimensions based on comparisons with similar standard models. For example, if the model AR engine 808 cannot see a side view of the user’s thighs, it may infer the side dimension from a reference model with similar front-on thigh width (which may be scaled or selected from a group of similar models). [0108] In some implementations, the model AR engine 808 may instruct a user to stand in a recommended area in the field of view of a camera, for example, by displaying indicators and/or augmented reality elements on the graphical user interface 600c. The area may include an outline and/or an interface element may change (e.g., colors) to indicate correct positioning. For example, when the user is in a recommended area, the area or a graphical area may turn green in the interface and/or an object model may be rendered on the user.

[0109] For example, a user may use a graphical user interface to navigate through a set of clothing items, select a certain desired clothing item, and select a call-to-action graphical element (e.g., “try this item on”). The user may allow access to the camera on the user’s client device 750 (e.g., a smartphone, smart TV, laptop computer, etc.) and stand in the recommended area in the field of view of the camera. The model AR engine 808 may leverage augmented reality in order to deliver a virtual try-on experience, such as by overlaying the object model over the user using the dimensions and/or interactions of the user model. For instance, the object model may move with, be hidden by (e.g., even if the user model is not rendered over the user), or otherwise interact with the user model. It should be noted that implementations other than augmented reality are possible and contemplated herein, for instance, the graphical interfaces, overlays, etc., described herein may be used on still images, virtual reality, augmented reality, or other graphical depictions of users.

[0110] Returning to Figure 3, at 314, the model AR engine 808 may generate a volumetric user model including using a skeletal structure model and based on the outer dimensions or edges of the user. For example, the user model module 924 may generate a volumetric three-dimensional model using the skeletal model and outer dimensions relative to the skeletal model.

[0111] In some implementations, to generate the three-dimensional user model, the user model module 924 may use a Kalman filtering process to update underlying three-dimensional dimensions with two-dimensional information. As the user moves, pivots, articulates limbs, etc., the user model module 924 may (e.g., using the Kalman filter or other processes) to determine the user’s personalized dimensions as well as output the articulated pose of the user’s wireskeleton structure/model. It should be noted that although the Kalman filtering process is mentioned, many other algorithms and processes for generating a three-dimensional model are possible and contemplated herein.

[0112] In some implementations, the method of Figure 3 may return to 304, 308, 310, or 312, for example, to update the model for a subsequent segment or frame of a video (e.g., a live video). For example, the model AR engine 808 may generate and/or update the user model in real time, which accommodates movements by the user. In some implementations, the user model may be continually or repeatedly updated in real time.

[0113] In some implementations, the three-dimensional space in which a user model is generated in may be constant and/or match that of the field of view of a camera. For example, the camera collecting the image data for the user may be static or remain in one place, which may increase the fidelity or accuracy of the user model and overlay. In some implementations, the camera may be dynamic or mobile. For example, the camera may be located on a mobile device, and the user may be holding the mobile device in front of a mirror (or a user’s friend may be holding the mobile device for him or her), which may move contextual reference points, position of the user, distance from the camera, etc. As discussed herein, the model AR engine 808 may accommodate for these variables by using reference, index, or articulation points on the user to dimensionalize the three-dimensional user model.

[0114] Figure 4 is a flowchart of an example method of overlaying the object model on an image using the user model. Aspects of the method described in reference to Figure 4 may be modified or exchanged with those described herein, such as those described in reference to Figure 1. For instance, the method of Figure 4 may provide additional details to the operations of Figure 1. Operations of Figure 4 may be performed by the overlay module 926 of the model AR engine 808, although other components of the model AR engine 808 may be used or otherwise organized without departing from the scope of this disclosure.

[0115] The overlay module 926 may overlay the three-dimensional object model on the user model and/or the image using the user model in an augmented reality view. The object model may be generated, updated, and/or overlaid on the user in real time. The user model may be determined or generated in real time and the user model and/or object model may be updated to follow the user as the user moves, rotates, articulates joints, and so forth, which may help the user to visualize the fit of the article of clothing. In some implementations, the overlay module 926 may transmit an object model and/or a user model to a client device 750, which may overlay the object model on an image/video of the user captured by the client device 750 (although other devices and implementations are possible and contemplated). In some instances, by generating and/or updating one or more of the object model, the user model, and the interaction therebetween on a server and transmitting the data to the client device 750, the computational resources used on the client device 750 may be reduced, thereby allowing more realistic rendering on a device with limited processing capabilities, such as a smartphone or smart television.

[0116] As the object model is overlaid, the overlay module 926 may show interactions between the object model and the user model to illustrate how the object model may move, whether the clothing is loose or tight (e.g., based on relative dimensions of the user model and object model), and other object-user interactions, which may thereby improve user experience and determination of correct object sizes (e.g., clothing sizing when ordered from an online retailer).

[0117] At 402, the model AR engine 808 may receive a selection of an object from an application or link. For instance, a website or application may display a list of objects. The user may select an object from a graphical interface displayed on the website or application, which may cause the model AR engine 808 to perform operations, such as determining and/or generating the object model corresponding to the selected object. For example, the model AR engine 808 may retrieve the object model from a computer-accessible data storage device. [0118] At 404, the model AR engine 808 may receive image and/or video data showing the user and determine a user model. For example, the model AR engine 808, operating on a server 822 and/or client device 750, may receive a video stream and use it to determine or generate a user model, for example, in real time, as described in further detail above. [0119] At 406, the model AR engine 808 may determine and provide a recommended object size based on the object’s dimensions and/or the user’s dimensions. For example, the model AR engine 808 may compare the skeletal and/or outer dimensions of the user (e.g., of the user model) with the dimensions of different objects (e.g., of the object models), which may correspond to various sizes of a particular article of clothing. Based on the compared sizes, the model AR engine 808 may recommend which size fits the user. The overlay model may additionally or alternatively present the recommended size overlaid over the user model. For example, the model AR engine 808 may determine a waist circumference and leg length of pants (e.g., based on provided or determined dimensions of an object model of the pants) and compare these measurements against corresponding measurements of the user (e.g., based on provided or determined dimensions of a user model of the user). The model AR engine 808 may recommend a closest size (e.g., small, medium, large, etc.) of the object based on the closest measurements, the closest larger measurements, and/or a requested user fit (e.g., a user may request a tight fit, a loose fit, extra length, etc.).

[0120] In some implementations, the user may decide whether a particular size of an article of clothing is appropriate. For example, the user may virtually try on a small, medium, and large object model of an object to compare how they look when worn. The user may then determine which size looks best and may select (e.g., by purchasing) that object/article of clothing, thereby reducing redundant purchases of the same article of clothing in different sizes. [0121] At 408, the model AR engine 808 may match point(s) of object model of the selected object with the user model. For example, in some implementations, a three-dimensional object model may be placed over a user model, so that portions of the object model match up with the image of the user and relative sizes of the object may be compared. Reference, index, or articulation points may be matched between the user model and the object model, for example, an object articulation point labeled left shoulder may be matched or overlaid over (e.g., within a threshold distance) a user articulation point of a user model labeled as left shoulder. As described in further detail above, various articulation points may have associated various levels of flexibility in movement and/or positioning.

[0122] In some implementations, the overlay module 926 may use and/or communicate with various third-party applications to perform matching operations and/or computer learning models may be trained with training examples where index or reference points are matched. [0123] At 410, the model AR engine 808 may overlay one or more object models over the image or video, for example, over the user model corresponding to a user in the image or video. In some implementations, the object model may be overlaid on the user model using the matched index or articulation points, for example, by overlaying some or all of the object articulation points with user articulation points.

[0124] In some implementations, the overlay module 926 may articulate the object model, for example, about the user and/or object articulation points, so that the model is articulated corresponding to the user in the image (e.g., based on the user or skeletal model). For example, if a virtual model of a shirt with sleeves that are too short for the user is overlaid on the user, the overlay module 926 may articulate the sleeves based on the length of the user’s arms (e.g., at shoulder or elbow articulation points).

[0125] In some implementations, the overlay module 926 may render the object model by displaying the object model as a three-dimensional mesh, such as triangles or polygons that display the object’s surface, although, it should be noted that other methods of rendering models may be used.

[0126] In some implementations, as described above, multiple three-dimensional object models may be overlaid on a user based on rules or the order in which they are selected. For example, a pair of pants may be overlaid on the user/user model and a shirt may be overlaid over both the user/user model and the pants/pants object model (e.g., where the shirt extends past an edge of the pants).

[0127] At 412, the model AR engine 808 may identify one or more update(s) to a status or position of the user in image or video. For example, based on image comparison frame to frame, a change to the position of a user model, or other techniques, the overlay module 926 may determine that the user has changed positions in real time, so that the user model can be updated in real time.

[0128] In some implementations, the three-dimensional space in which the three- dimensional user model is generated in may be constant or varying, as described above, for example, based on a changing user position or camera angle. As the size of the user in the graphical interface changes, the overlay module 926 may match the dimensions of the user, user model, and/or object model, so that the size, position, orientation, rotation, angle, and/or articulation of the object model remains consistent with the user in the graphical (e.g., virtual or augmented reality) interface.

[0129] In some implementations, the model AR engine 808 may estimate a state of the user, for example, the model AR engine 808 may detect changes in positions of the articulation points and, using outer dimensions of the user model associated with articulation points (e.g., a defined distance from the articulation points), provide interaction with the object model. In some implementations, in order to reduce computational resources used to update a user model, once the outer dimensions or surfaces of the user are determined, the model AR engine 808 may update the volumetric user model using detected changes to the skeletal model or articulation points (e.g., rather than re-determining or computing an outer dimension). For example, the model AR engine 808 may detect a rotation of the articulation points of the skeletal model and correspondingly rotate the object model, for example, based on the matched articulation model. [0130] At 414, the model AR engine 808 may update the object model overlay based on the updated position, articulation, or other status of the user. Accordingly, as the user changes position or size, the object model may be updated so that the true, relative size of the object is retained. The overlay and/or models may be updated periodically or every frame (e.g., 30 frames per second, 60 frames per second, etc.) to show movement of the object model. For example, the operations of the method of Figure 4 may return to 404 receive an image or video frame, to 408 to match points of the object model with the user model, or to 412 to identify updates to the user model.

[0131] In some implementations, an overlaid three-dimensional object model may move with the user, such as by articulating or otherwise by rendering it to conform with the articulation point(s). For example, as the user bends his or her arm, the overlaid clothing model may bend at the user’s elbow.

[0132] In some implementations, the three-dimensional object model may conform to the outer dimensions of the user’s body. For example, based on the material (e.g., fabric) properties of the object model, the article of clothing may stretch, fold, drape, or otherwise change shape in response to the user’s body or changes in its position. In this manner, rather than a static image that is overlaid over the user, the user may gain a better understanding of how the article of clothing may fit and/or move with the user, thereby allowing the user to determine how the article of clothing looks on him or her and determine whether it is worth purchasing.

[0133] Figure 7 is a signal diagram illustrating example communication between systems, which may be different software or hardware modules or systems.

[0134] The operations described herein, the models, and systems can be incorporated within an environment within which information is passed between various platforms in connection with rendering the virtual try-on service or other technologies, for example, within a third-party platform. For instance, one or more implementations of the above-described technology could be implemented on a third-party website in accordance with the flow of interactions between a first-party system, third-party system, consumer system, and/or a retailer system, etc. For example, operations of Figure 7 may integrate the intelligent modeling described herein into a website of a third party, such as a retailer. Depending on the implementation, the operations may use an iframe, so that the retailer would use a few lines of code to use application programming interface(s) (APIs), application IDs, tokens, assets, and/or processes, although other technologies, plugins, or operations may be used to allow the technology to be embedded or used within various systems.

[0135] As illustrated, a user or client device 750 may invoke an iframe at 702, for example, by requesting use of the modeling service or other service of the model AR engine 808, from a UX (user experience, graphical user interface, or other front-end service) service 752, which may relay the request to an authentication service 754 and/or data service 756. For example, at 704, the UX service 752 may authenticate a third-party API key and check if the iframe can be accessed by sending a request to the data service 756.

[0136] At 706, if the API key and other access criteria are satisfied, the data service 756 may provide access to the iframe to the client device and/or to the UX service 752 (e.g., which may relay the access to the client device 750 via a website or application).

[0137] In some implementations, the user, using the client device 750, may login or setup a profile at 708 via communication between the client device 750 and the authentication service 754. At 710, the authentication service 754 may send a session token to the client device 750. At 712, the client device 750 may fetch or update user information with the authentication service 754 and/or data service 756. At 714, the authentication service 754 may respond to the request at 712 by providing user profile information or other details to the client device 750.

[0138] In some implementations, at 716, the client device 750 may transmit a request to the data service 756 (e.g., based on the user profile and/or session token) requesting to fetch a three-dimensional article model (e.g., an object model) and associated meta-data (e.g., in a glTF file and header) from the data service 756. At 718, the data service may respond with the three- dimensional article/object model and its metadata (e.g., in a glTF file).

[0139] At 720, the client device 750 may provide a virtual article overlay or try-on. For example, using a user model module 924 and/or overlay module 926, it may determine a user model and/or overlay the object model over the user model, as described in further detail above. [0140] Figure 8 is a block diagram of an example system 800 for real-time rendering of wearable articles on human bodies, for example, on camera-supported computing devices.

[0141] The illustrated system 800 may include client devices 750a... ,750n, which may run model AR engines 808a... 808n, a third-party server 818, and an application server 822, which are electronically communicatively coupled via a network 802 for interaction with one another, although other system configurations are possible including other devices, systems, and networks. For example, the system 800 could include any number of client devices 750, third- party servers 818, application servers 822, and other systems and devices.

[0142] The network 802 may include any number of networks and/or network types. For example, the network 802 may include, but is not limited to, one or more local area networks (LANs), wide area networks (WANs) (e.g., the Internet), virtual private networks (VPNs), wireless wide area network (WWANs), WiMAX® networks, personal area networks (PANs) (e.g., Bluetooth® communication networks), various combinations thereof, etc. These private and/or public networks may have any number of configurations and/or topologies, and data may be transmitted via the networks using a variety of different communication protocols including, for example, various Internet layer, transport layer, or application layer protocols. For example, data may be transmitted via the networks using TCP/IP, UDP, TCP, HTTP, HTTPS, DASH, RTSP, RTP, RTCP, VOIP, FTP, WS, WAP, SMS, MMS, XMS, IMAP, SMTP, POP, WebDAV, or other known protocols.

[0143] The client device 750 includes one or more computing devices having data processing and communication capabilities. The client device 750 may couple to and communicate with other client devices 750 and the other entities of the system 800 via the network 802 using a wireless and/or wired connection, such as the application server 822. In some implementations, the client device 750 may also be configured to directly with devices, such as sensors 810 (e.g., via Bluetooth or Wi-Fi). Examples of client devices 750 may include, but are not limited to, mobile phones, tablets, laptops, desktops, netbooks, server appliances, servers, virtual machines, TVs, etc. The system 800 may include any number of client devices 750, including client devices 750 of the same or different type.

[0144] A plurality of client devices 750a... 75 On are depicted in Figure 8 to indicate that the application server 822 and its components may aggregate information about and provide data associated with the systems and processes described herein to a multiplicity of users 814 and/or client devices 750a... 750n. In some instances, a first client device 750a may be used by an end user to apply models, while a second client device 750n may be used by an administrator to generate or modify object models. In some implementations, a single user 814 may use more than one client device 750, which the application server 822 may use to track and aggregate interaction data associated with the user 814 through a variety of different channels including online, physical, and phone (e.g., text, voice, etc.) channels, as discussed elsewhere herein. For instance, the application server 822 may send data, such as graphical user interfaces, receipts, and confirmation messages, etc., to a model AR engine 808a... 808n on a client device 750a... 750n. [0145] As described in further detail below, the model AR engine 808 may include computer logic for performing operations described herein. The model AR engine 808 and its components or functionalities may be distributed among multiple systems. For example, an instance or component of the model AR engine 808 executed on client device 750 may collect image data using a connected sensor 810, which it may transmit to an instance or component of the model AR engine 808 executed on the application server 822. The instance/component on the application server may generate a three-dimensional user model and transmit it to the client device 750. Additionally or alternatively, the instance/component of the model AR engine 808 executed on the client device 750 may generate or update the user model module 924 as described above.

[0146] Similarly, the model AR engine 808 may have user facing components or logic and retailer/manufacturing facing components or logic. For instance, the model AR engine 808 may separately interface with a third-party server 818 (or client device 750) for receiving object data or models and/or embedding functionality on websites or applications. Additionally or alternatively, the model AR engine 808 may have a consumer/user facing component that interacts with end users on a client device 750 for generating user models, selecting objects, and/or overlaying object models, as described herein. Depending on the implementation, the functionality of the model AR engine 808 may be automatically determined based on a defined role or access level of a user profile or login.

[0147] In some implementations, the sensor(s) 810 may be image or other sensors, such as an optical sensor, a LIDAR sensor, a time-of-flight sensor, a sonar sensor, etc., for determining the appearance, shape, size, depth, distance from the sensor, or other aspects of objects or users. For example, the model AR engine 808 may use the sensor’s 810 data, as described above.

[0148] The application server 822 may include a web server 824, a model AR engine 808, and a database 828. In some configurations, the model AR engine 808 may be distributed over the network 802 on disparate devices in disparate locations or may reside on the same locations, in which case the client device 750a and/or the application server 822 may each include an instance of the model AR engine 808 and/or portions thereof. The client devices 750 may also store and/or operate other software such as a model AR engine 808, an operating system, other applications, etc., that are configured to interact with the application server 822 via the network 802.

[0149] The application server 822 and the third-party server 818 have data processing, storing, and communication capabilities, as discussed elsewhere herein. For example, the servers 822 and/or 818 may include one or more hardware servers, server arrays, storage devices and/or systems, etc. In some implementations, the servers 822 and/or 818 may include one or more virtual servers, which operate in a host server environment.

[0150] The database 828 may be stored on one or more information sources for storing and providing access to data, such as the data storage device 908. The database 828 may store data user models, object models, objects, user profiles, client devices 750, instances of the model AR engine 808, items, preferences, etc.

[0151] A third-party server 818 can host services such as a third-party application (not shown), which may be individual and/or incorporated into the services provided by the application server 822. For example, the third-party server 818 may represent a computing system of a manufacturer or retailer via which object models or object data may be received or retrieved. In some instances, access to the model AR engine 808 may be provided via the third- party server 818, for example, via an ifirame, plugin, or embedded code, as described above.

[0152] It should be understood that the system 800 illustrated in Figure 8 is representative of an example system and that a variety of different system environments and configurations are contemplated and are within the scope of the present disclosure. For instance, various acts and/or functionality may be moved from a server to a client, or vice versa, data may be consolidated into a single data store or further segmented into additional data stores, and some implementations may include additional or fewer computing devices, services, and/or networks, and may implement various functionality client or server-side. Further, various entities of the system may be integrated into a single computing device or system or divided into additional computing devices or systems, etc.

[0153] Figure 9 is a block diagram of an example computing system 900, which may represent the computer architecture of a client device 750, third-party server 818, application server 822, and/or another device described herein, depending on the implementation. In some implementations, as depicted in Figure 9, the computing system 900 may include a web server 824, a model AR engine 808, or another application, depending on the configuration. For instance, a client device 750 may include a model AR engine 808; and the application server 822 may include the web server 824, a model AR engine 808, and/or components thereof, although other configurations are also possible and contemplated.

[0154] The web server 824 may include computer logic executable by the processor 904 to process content requests (e.g., to or from a client device 750). The web server 824 may include an HTTP server, a REST (representational state transfer) service, or other suitable server type. The web server 824 may receive content requests (e.g., product search requests, HTTP requests) from client devices 750, cooperate with the model AR engine 808 to determine the content, retrieve and incorporate data from the data storage device 908, format the content, and provide the content to the client devices 750.

[0155] In some instances, the web server 824 may format the content using a web language and provide the content to a corresponding model AR engine 808 for processing and/or rendering to the user for display. The web server 824 may be coupled to the data storage device 908 to store retrieve, and/or manipulate data stored therein and may be coupled to the model AR engine 808 to facilitate its operations.

[0156] The model AR engine 808 includes computer logic executable by the processor 904 on a client device 750 to provide for user interaction, receive user input, present information to the user via a display, and send data to and receive data from the other entities of the system 800 via the network 802. In some implementations, the model AR engine 808 may generate and present user interfaces based at least in part on information received from the web server 824 via the network 802. For example, a customer/user 814 may use the model AR engine 808 to receive the pages provided by the application server 822, view various objects available online, purchase products, etc. In some implementations, the model AR engine 808 includes a web browser and/or code operable therein, a customized client-side application (e.g., a dedicated mobile app), a combination of both, etc. For example, in some implementations, the model AR engine 808 may generate and display graphical user interfaces.

[0157] Depending on the implementation, the model AR engine 808 may include various components, modules, engines, sub-routines, etc. For example, as illustrated, the model AR engine 808 may include an object model module 922 for determining object models, a user model module 924 for determining user models, an overlay module 926 for overlaying object models, and/or other logic and modules 928, as described herein. The components and functionality of the model AR engine 808 are described in further detail elsewhere herein.

[0158] Various other logic or modules 928 may include an object selection module that provides a user with objects and/or object models to be browsed along with their attributes (e.g., size data, brand, reviews, etc.); computer learning algorithms (e.g., trained using labeled datasets); a storage module for storing and providing access to models; an embeddable module for providing a seamless experience of accessing and managing stored models and metadata via various systems; a UX service 752; an authentication service 754; a data service 756; a web portal for managing models, access, analytics, or other data; a subscription module for managing subscriptions to model AR engine 808 services; an analytics module for monitoring model, module, user, or object statistics, such as models tried on, conversion rates, etc.; or various other modules or logic.

[0159] As depicted, the computing system 900 may include a processor 904, a memory 906, a communication unit 902, an output device 916, an input device 914, and a data storage device 908, which may be communicatively coupled by a communication bus 910. The computing system 900 depicted in Figure 9 is provided by way of example and it should be understood that it may take other forms and include additional or fewer components without departing from the scope of the present disclosure. For instance, various components of the computing devices may be coupled for communication using a variety of communication protocols and/or technologies including, for instance, communication buses, software communication mechanisms, computer networks, etc. While not shown, the computing system 900 may include various operating systems, sensors, additional processors, and other physical configurations. The processor 904, memory 906, communication unit 902, etc., are representative of one or more of these components.

[0160] The processor 904 may execute software instructions by performing various input, logical, and/or mathematical operations. The processor 904 may have various computing architectures to method data signals (e.g., CISC, RISC, etc.). The processor 904 may be physical and/or virtual and may include a single core or plurality of processing units and/or cores. In some implementations, the processor 904 may be coupled to the memory 906 via the bus 910 to access data and instructions therefrom and store data therein. The bus 910 may couple the processor 904 to the other components of the computing system 900 including, for example, the memory 906, the communication unit 902, the input device 914, the output device 916, and the data storage device 908.

[0161] The memory 906 may store and provide access to data to the other components of the computing system 900. The memory 906 may be included in a single computing device or a plurality of computing devices. In some implementations, the memory 906 may store instructions and/or data that may be executed by the processor 904. For example, the memory 906 may store one or more of the web server 824, the model AR engine 808, and their respective components, depending on the configuration. The memory 906 is also capable of storing other instructions and data, including, for example, an operating system, hardware drivers, other software applications, databases, etc. The memory 906 may be coupled to the bus 910 for communication with the processor 904 and the other components of computing system 900.

[0162] The memory 906 may include a non-transitory computer-usable (e.g., readable, writeable, etc.) medium, which can be any non-transitory apparatus or device that can contain, store, communicate, propagate or transport instructions, data, computer programs, software, code, routines, etc., for processing by or in connection with the processor 904. In some implementations, the memory 906 may include one or more of volatile memory and non-volatile memory (e.g., RAM, ROM, hard disk, optical disk, etc.). It should be understood that the memory 906 may be a single device or may include multiple types of devices and configurations.

[0163] The bus 910 can include a communication bus for transferring data between components of a computing device or between computing devices, a network bus system including the network 802 or portions thereof, a processor mesh, a combination thereof, etc. In some implementations, the web server 824, model AR engine 808, and various other components operating on the computing system/ device 900 (operating systems, device drivers, etc.) may cooperate and communicate via a communication mechanism included in or implemented in association with the bus 910. The software communication mechanism can include and/or facilitate, for example, inter-method communication, local function or procedure calls, remote procedure calls, an object broker (e.g., CORBA), direct socket communication (e.g., TCP/IP sockets) among software modules, UDP broadcasts and receipts, HTTP connections, etc. Further, any or all of the communication could be secure (e.g., SSH, HTTPS, etc.).

[0164] The communication unit 902 may include one or more interface devices (I/F) for wired and wireless connectivity among the components of the system 800. For instance, the communication unit 902 may include, but is not limited to, various types known connectivity and interface options. The communication unit 902 may be coupled to the other components of the computing system 900 via the bus 910. The communication unit 902 can provide other connections to the network 802 and to other entities of the system 800 using various communication protocols. In some implementations, the communication unit 902 may include a Wi-Fi, Bluetooth, NFC, RFID, or other radio, and/or may include wired communication units. [0165] The input device 914 may include any device for inputting information into the computing system 900. In some implementations, the input device 914 may include one or more peripheral devices. For example, the input device 914 may include a keyboard, a pointing device, microphone, an image/video capture device (e.g., camera), a sensor 810, a touch-screen display integrated with the output device 916, etc. The output device 916 may be any device capable of outputting information from the computing system 900. The output device 916 may include one or more of a display (LCD, OLED, etc.), a printer, a haptic device, audio reproduction device, touch-screen display, a remote computing device, etc. In some implementations, the output device is a display which may display electronic images and data output by a processor of the computing system 900 for presentation to a user, such as the processor 904 or another dedicated processor.

[0166] The data storage device 908 may include one or more information sources for storing and providing access to data. In some implementations, the data storage device 908 may store data associated with a database management system (DBMS) operable on the computing system 900. For example, the DBMS could include a structured query language (SQL) DBMS, a NoSQL DMBS, various combinations thereof, etc. In some instances, the DBMS may store data in multi-dimensional tables comprised of rows and columns, and manipulate, e.g., insert, query, update and/or delete, rows of data using programmatic operations.

[0167] The data stored by the data storage device 908 may organized and queried using various criteria including any type of data stored by them, such as described herein. For example, the data storage device 908 may store the database 828. The data storage device 908 may include data tables, databases, or other organized collections of data. Examples of the types of data stored by the data storage device 908 may include, but are not limited to, the data described with respect to the figures, such as object data, user data, or models, for example.

[0168] The data storage device 908 may be included in the computing system 900 or in another computing system and/or storage system distinct from but coupled to or accessible by the computing system 900. The data storage device 908 can include one or more non-transitory computer-readable mediums for storing the data. In some implementations, the data storage device 908 may be incorporated with the memory 906 or may be distinct therefrom.

[0169] The components of the computing system 900 may be communicatively coupled by the bus 910 and/or the processor 904 to one another and/or the other components of the computing system 900. In some implementations, the components may include computer logic (e.g., software logic, hardware logic, etc.) executable by the processor 904 to provide their acts and/or functionality. In any of the foregoing implementations, the components may be adapted for cooperation and communication with the processor 904 and the other components of the computing system 900.

[0170] In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be understood that the technology described herein can be practiced without these specific details. Further, various systems, devices, and structures are shown in block diagram form in order to avoid obscuring the description. For instance, various implementations are described as having particular hardware, software, and user interfaces. However, the present disclosure applies to any type of computing device that can receive data and commands, and to any peripheral devices providing services.

[0171] In some instances, various implementations may be presented herein in terms of algorithms and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be a self-consistent set of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. [0172] To ease description, some elements of the system and/or the methods are referred to using the labels first, second, third, etc. These labels are intended to help to distinguish the elements but do not necessarily imply any particular order or ranking unless indicated otherwise. [0173] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout this disclosure, discussions utilizing terms including “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0174] Various implementations described herein may relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, including, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0175] The technology described herein can take the form of an entirely hardware implementation, an entirely software implementation, or implementations containing both hardware and software elements. For instance, the technology may be implemented in software, which includes but is not limited to firmware, resident software, microcode, etc. Furthermore, the technology can take the form of a computer program product accessible from a computer- usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer- usable or computer readable medium can be any non-transitory storage apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0176] A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. Input or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.

[0177] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, storage devices, remote printers, etc., through intervening private and/or public networks. Wireless (e.g., Wi-Fi™) transceivers, Ethernet adapters, and Modems, are just a few examples of network adapters. The private and public networks may have any number of configurations and/or topologies. Data may be transmitted between these devices via the networks using a variety of different communication protocols including, for example, various Internet layer, transport layer, or application layer protocols. For example, data may be transmitted via the networks using transmission control protocol / Internet protocol (TCP/IP), user datagram protocol (UDP), transmission control protocol (TCP), hypertext transfer protocol (HTTP), secure hypertext transfer protocol (HTTPS), dynamic adaptive streaming over HTTP (DASH), real-time streaming protocol (RTSP), realtime transport protocol (RTP) and the real-time transport control protocol (RTCP), voice over Internet protocol (VOIP), file transfer protocol (FTP), WebSocket (WS), wireless access protocol (WAP), various messaging protocols (SMS, MMS, XMS, IMAP, SMTP, POP, WebDAV, etc.), or other known protocols.

[0178] Finally, the structure, algorithms, and/or interfaces presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method blocks. The required structure for a variety of these systems will appear from the description above. In addition, the specification is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the specification as described herein.

[0179] The foregoing description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. As will be understood by those familiar with the art, the specification may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the modules, routines, features, attributes, methodologies, and other aspects are not mandatory or significant, and the mechanisms that implement the specification or its features may have different names, divisions and/or formats.

[0180] Furthermore, the modules, routines, features, attributes, methodologies, and other aspects of the disclosure can be implemented as software, hardware, firmware, or any combination of the foregoing. Also, wherever a component, an example of which is a module, of the specification is implemented as software, the component can be implemented as a standalone program, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future. Additionally, the disclosure is in no way limited to implementation in any specific programming language, or for any specific operating system or environment.

Claims

What is claimed is:

1. A computer-implemented method comprising: determining, by one or more processors, an object model describing an object, the object model including one or more dimensions and one or more visual attributes of the object; determining, by the one or more processors, a user model including a skeletal model and outer dimensions describing a user, the skeletal model including two or more user articulation points of the user; overlaying, by the one or more processors, the object model over the user model using the two or more user articulation points; and providing, by the one or more processors for display on a client computing device, a graphical interface including the object model overlaid over the user model.

2. The computer-implemented method of claim 1, wherein determining the object model includes: receiving a two-dimensional image showing the object; dividing the two-dimensional image of the object into two or more image portions; and generating a three-dimensional model, wherein generating includes piecing together the two or more image portions.

3. The computer-implemented method of claim 2, wherein: determining the object model includes determining an object articulation point of the object; and overlaying the object model over the user model includes: determining an articulation of the user associated with a user articulation point of the two or more user articulation points, associating the object articulation point with the user articulation point, and articulating the object model based on the articulation of the user, the user articulation point, and the object articulation point.

4. The computer-implemented method of claim 1, wherein determining the object model includes: receiving one or more dimensions of the object; and scaling the object model using the one or more dimensions.

5. The computer-implemented method of claim 1, wherein determining the user model includes: receiving an image depicting the user; identifying positions of the two or more user articulation points in the image; and generating the skeletal model using the positions of the two or more user articulation points in the image.

6. The computer-implemented method of claim 5, wherein determining the user model includes: detecting an outline around the skeletal model of the user in the image; determining the outer dimensions of the user using the outline around the skeletal model of the user in the image; and generating the user model including a volumetric three-dimensional model based on the outer dimensions of the user.

7. The computer-implemented method of claim 1, wherein determining the user model includes: receiving an image depicting the user; determining a reference distance in the image, the reference distance including a length of an object visible in the image; and scaling the user model using the reference distance.

8. The computer-implemented method of claim 1, further comprising: receiving, by the one or more processors, a file including a renderable three-dimensional object model describing the object; and transmitting, by the one or more processors, the file including the renderable three- dimensional object model for execution on the client computing device.

9. The computer-implemented method of claim 1, further comprising: receiving, by the one or more processors, a video data stream including a video of the user; detecting a change in position of the user in the video in real time; and overlaying the object over the user model in the video based on the change in position of the user.

10. The computer-implemented method of claim 9, wherein overlaying the object over the user model in the video based on the change in position of the user includes: detecting an articulation of an elbow of the user; and articulating the object model at the elbow of the user.

11. The computer-implemented method of claim 1, further comprising: determining a second object model of the object, the object model and the second object model having different dimensions associated with different object sizes; determining a user dimension based on the user model; automatically determining, by the one or more processors, one of the object model and the second object model based on the user dimension and the one or more dimensions of the object model and the second object model; and displaying, by the one or more processors for display on the client computing device, a recommended size associated with the determined one of the object model and the second object model.

12. A sy stem compri sing : one or more processors; and a non-transitory computer readable memory storing instructions that, when executed by the one or more processors, cause the system to: determine, by the one or more processors, an object model describing an object, the object model including one or more dimensions and one or more visual attributes of the object; determine, by the one or more processors, a user model including a skeletal model and outer dimensions describing a user, the skeletal model including two or more user articulation points of the user; overlay, by the one or more processors, the object model over the user model using the two or more user articulation points; and provide, by the one or more processors for display on a client computing device, a graphical interface including the object model overlaid over the user model.

13. The system of claim 12, wherein determining the object model includes: receiving a two-dimensional image showing the object; dividing the two-dimensional image of the object into two or more image portions; and generating a three-dimensional model, wherein generating includes piecing together the two or more image portions.

14. The system of claim 12, wherein: determining the object model includes determining an object articulation point of the object; and overlaying the object model over the user model includes: determining an articulation of the user associated with a user articulation point of the two or more user articulation points, associating the object articulation point with the user articulation point, and articulating the object model based on the articulation of the user, the user articulation point, and the object articulation point.

15. The system of claim 12, wherein determining the object model includes: receiving one or more dimensions of the object; and scaling the object model using the one or more dimensions.

16. The system of claim 12, wherein determining the user model includes: receiving an image depicting the user; identifying positions of the two or more user articulation points in the image; and generating the skeletal model using the positions of the two or more user articulation points in the image.

17. The system of claim 16, wherein determining the user model includes: detecting an outline around the skeletal model of the user in the image; determining the outer dimensions of the user using the outline around the skeletal model of the user in the image; and generating the user model including a volumetric three-dimensional model based on the outer dimensions of the user.

18. The system of claim 12, wherein determining the user model includes: receiving an image depicting the user; determining a reference distance in the image; and scaling the user model using the reference distance.

19. The system of claim 12, wherein the instructions, when executed by the one or more processors, further cause the system to: receiving, by the one or more processors, a file including a renderable three-dimensional object model describing the object; and transmitting, by the one or more processors, the file including the renderable three- dimensional object model for execution on the client computing device.

20. A system comprising: means for determining an object model describing an object, the object model including one or more dimensions and one or more visual attributes of the object; means for determining a user model including a skeletal model and outer dimensions describing a user, the skeletal model including two or more user articulation points of the user; means for overlaying the object model over the user model using the two or more user articulation points; and means for providing, for display on a computing device, a graphical interface including the object model overlaid over the user model.